MPI - error loading shared libraries

Question

The problem I faced has been solved here: Loading shared library in open-mpi/ mpi-run

I know not how, setting LD_LIBRARY_PATH or specifying -x LD_LIBRARY_PATH fixes the problem, when my installation itself specifies the necessary -L arguments. My installation is in ~/mpi/

I have also included my compile-link configs.

$ mpic++ -showme:version 
mpic++: Open MPI 1.6.3 (Language: C++)

$ mpic++ -showme
g++ -I/home/vigneshwaren/mpi/include -pthread -L/home/vigneshwaren/mpi/lib
-lmpi_cxx -lmpi -ldl -lm -Wl,--export-dynamic -lrt -lnsl -lutil -lm -ldl

$ mpic++ -showme:libdirs
/home/vigneshwaren/mpi/lib

$ mpic++ -showme:libs
mpi_cxx mpi dl m rt nsl util m dl    % Notice mpi_cxx here %

When I compiled with mpic++ <file> and ran with mpirun a.out I got a (shared library) linker error

error while loading shared libraries: libmpi_cxx.so.1: 
cannot open shared object file: No such file or directory

The error has been fixed by setting LD_LIBRARY_PATH. The question is how and why? What am i missing? Why is LD_LIBRARY_PATH required when my installation looks just fine.

Answer 1

libdl, libm, librt, libnsl and libutil are all essential system-wide libraries and they come as part of the very basic OS installation. libmpi and libmpi_cxx are part of the Open MPI installation and in your case are located in a non-standard location that must be explicitly included in the linker search path LD_LIBRARY_PATH.

It is possible to modify the configuration of the Open MPI compiler wrappers and make them pass the -rpath option to the linker. -rpath takes a library path and appends its to a list, stored inside the executable file, which tells the runtime link editor (a.k.a. the dynamic linker) where to search for libraries before it consults the LD_LIBRARY_PATH variable. For example, in your case the following option would suffice:

-Wl,-rpath,/home/vigneshwaren/mpi/lib

This would embed the path to the Open MPI libraries inside the executable and it would not matter if that path is part of LD_LIBRARY_PATH at run time or not.

To make the corresponding wrapper add that option to the list of compiler flags, you would have to modify the mpiXX-wrapper-data.txt file (where XX is cc, c++, CC, f90, etc.), located in mpi/share/openmpi/. For example, to make mpicc pass the option, you would have to modify /home/vigneshwaren/mpi/share/openmpi/mpicc-wrapper-data.txt and add the following to the line that starts with linker_flags=:

linker_flags= ... -Wl,-rpath,${prefix}/lib

${prefix} is automatically expanded by the wrapper to the current Open MPI installation path.