Why I cannot override search path of dynamic libraries with LD_LIBRARY_PATH?

Question

Edit: I resolved this issue, the solution is below.

I am building a code in a shared computing cluster dedicated for scientific computing, thus I can only control files in my home folder. Although I am using fftw as an example, I would like to understand the specific reason, why my attempt to setup LD_LIBRARY_PATH does not work.

I build the fftw and fftw_mpi libraries in my home folder like this

./configure --prefix=$HOME/install/fftw --enable-mpi --enable-shared
make install

It builds fine, but in install/fftw/lib, I find that the freshly built libfftw3_mpi.so links to wrong version of fftw library.

$ ldd libfftw3_mpi.so |grep fftw
  libfftw3.so.3 => /usr/lib64/libfftw3.so.3 (0x00007f7df0979000)

If I now try to set the LD_LIBRARY_PATH correctly pointing to this directory, it still prefers the wrong library:

$ export LD_LIBRARY_PATH=$HOME/install/fftw/lib
$ ldd libfftw3_mpi.so |grep fftw
    libfftw3.so.3 => /usr/lib64/libfftw3.so.3 (0x00007f32b4794000)

Only if I explicitly use LD_PRELOAD, I can override this behavior. I don't think LD_PRELOAD is a proper solution though.

$ export LD_PRELOAD=$HOME/install/fftw/lib/libfftw3.so.3 
$ ldd libfftw3_mpi.so |grep fftw
   $HOME/install/fftw/lib/libfftw3.so.3 (0x00007f5ca3d14000)

Here is what I would have expecting, a small test done in Ubuntu desktop, where I installed fftw to /usr/lib first, and then override this search path with LD_LIBRARY_PATH.

$ export LD_LIBRARY_PATH=
$ ldd q0test_mpi |grep fftw3
    libfftw3.so.3 => /usr/lib/x86_64-linux-gnu/libfftw3.so.3
$ export LD_LIBRARY_PATH=$HOME/install/fftw-3.3.4/lib
$ ldd q0test_mpi |grep fftw3
    libfftw3.so.3 => $HOME/install/fftw-3.3.4/lib/libfftw3.so.3

In short: Why is libfft3_mpi library still finding the wrong dynamic fftw3 library? Where is this searchpath hard coded in a such way that it is prioritized over LD_LIBARY_PATH? Why is this is not the case in another computer?

I am using intel compilers 13.1.2, mkl 11.0.4.183 and openmpi 1.6.2 if this matters.

Edit: Thanks for all the answers. With help of those, we were able to isolate the problem to RPATH, and from there, the cluster support was able to figure out the problem. I accepted the first answer, but both answers were good.

The reason, why this was so hard to figure out, is that we did not know that the compilers were actually wrapper scripts, adding things to compiler command line. Here a part of a reply from the support:

[The] compilation goes through our compiler wrapper. We do RPATH-ing by default as it helps most users in correctly running their jobs without loading LD-LIBRARY_PATH etc. However we exclude certain library paths from default RPATH which includes /lib, /lib64 /proj /home etc. Earlier the /usr/lib64 was not excluded by mistake (mostly). Now we have added that path in the exclusion list.

Answer 1

From http://man7.org/linux/man-pages/man8/ld.so.8.html

When resolving shared object dependencies, the dynamic linker first inspects each dependency string to see if it contains a slash (this can occur if a shared object pathname containing slashes was specified at link time). If a slash is found, then the dependency string is interpreted as a (relative or absolute) pathname, and the shared object is loaded using that pathname.

If a shared object dependency does not contain a slash, then it is searched for in the following order:

o (ELF only) Using the directories specified in the DT_RPATH dynamic section attribute of the binary if present and DT_RUNPATH attribute does not exist. Use of DT_RPATH is deprecated.

o Using the environment variable LD_LIBRARY_PATH. Except if the executable is a set-user-ID/set-group-ID binary, in which case it is ignored.

o (ELF only) Using the directories specified in the DT_RUNPATH dynamic section attribute of the binary if present.

o From the cache file /etc/ld.so.cache, which contains a compiled list of candidate shared objects previously found in the augmented library path. If, however, the binary was linked with the -z nodeflib linker option, shared objects in the default paths are skipped. Shared objects installed in hardware capability directories (see below) are preferred to other shared objects.

o In the default path /lib, and then /usr/lib. (On some 64-bit archiectures, the default paths for 64-bit shared objects are /lib64, and then /usr/lib64.) If the binary was linked with the -z nodeflib linker option, this step is skipped.

• with readelf readelf -d libfftw3_mpi.so you can check if your lib contains such a attribute in the dynamic section.

• with export LD_DEBUG=libs you can debug the search path used to find your libs

• with chrpath -r<new_path> <executable> the rpath can be changed