infiniband 환경에서 HP-MPI 사용 Solver의 통신방식을 강제로 TCP 로 지정하기
On some Linux machines, Infiniband libraries are installed (for example with OpenMPI) without the corresponding kernel drivers and/or hardware. This could cause a CFD-ACE+ parallel run to stop with error messages related to Infiniband. The error messages maybe any of the following or something similar depending on your configuration:
CFD-ACE-SOLVER-MPM-MPI: Rank 0:0: MPI_Init: didn’t find active interface/port
CFD-ACE-SOLVER-MPM-MPI: Rank 0:0: MPI_Init: Can’t initialize RDMA device
CFD-ACE-SOLVER-MPM-MPI: Rank 0:0: MPI_Init: MPI BUG: Cannot initialize RDMA protoc
In such cases, one has to force HP-MPI to use TCP connections by setting a new environment variable. The MPI_IC_ORDER environment variable can be used to force HP-MPI to ignore all other interconnects except TCP.
Variable name: MPI_IC_ORDER Variable value: TCP
For csh/tcsh: setenv MPI_IC_ORDER “TCP”
This needs to be set only on the master node.
MPI_IC_ORDER is an environment variable whose default contents are:
ibv:vapi:udapl:psm:mx:gm:elan:itapi:TCP”
It instructs HP-MPI to search in a specific order for the presence of an interconnect. Lowercase selections imply ‘use if detected, otherwise keep searching’. An uppercase option demands that the interconnect option be used, and if it cannot be selected the application will terminate with an error. This can be used to set a different interconnect if available.
Your article made me suddenly realize that I am writing a thesis on gate.io. After reading your article, I have a different way of thinking, thank you. However, I still have some doubts, can you help me? Thanks.