Executable cannot find dynamically linked mkl library, but ldd does

Question

I have a code which was dynamically linked with the mkl library. when running the code, it reports mkl not found.

./bmdl
/g/software/EMTO/5.7/intel_12.1/ser/bin/bmdl: error while loading shared libraries: libmkl_intel_lp64.so: cannot open shared object file: No such file or directory

But when I use ldd to check the dynamically linked libraries in the executable, it shows the mkl library is found

ldd bmdl
libmkl_intel_lp64.so => /g/software/intelXE/composer_xe_2011_sp1/mkl/lib/intel64/libmkl_intel_lp64.so (0x00002b975d76d000)
libmkl_sequential.so => /g/software/intelXE/composer_xe_2011_sp1/mkl/lib/intel64/libmkl_sequential.so (0x00002b975df53000)
libmkl_core.so => /g/software/intelXE/composer_xe_2011_sp1/mkl/lib/intel64/libmkl_core.so (0x00002b975e631000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x0000003785600000)
libm.so.6 => /lib64/libm.so.6 (0x0000003784e00000)
libc.so.6 => /lib64/libc.so.6 (0x0000003784a00000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x000000378a600000)
libdl.so.2 => /lib64/libdl.so.2 (0x0000003785200000)
/lib64/ld-linux-x86-64.so.2 (0x0000003784600000)

Any idea what could be wrong?

Output from readelf -l ./bmdl

Elf file type is EXEC (Executable file)
Entry point 0x4034b0
There are 8 program headers, starting at offset 64

Program Headers:
  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align
  PHDR           0x0000000000000040 0x0000000000400040 0x0000000000400040
                 0x00000000000001c0 0x00000000000001c0  R E    8
  INTERP         0x0000000000000200 0x0000000000400200 0x0000000000400200
                 0x000000000000001c 0x000000000000001c  R      1
      [Requesting program interpreter: /lib64/ld-linux-x86-64.so.2]
  LOAD           0x0000000000000000 0x0000000000400000 0x0000000000400000
                 0x00000000000e4eb4 0x00000000000e4eb4  R E    200000
  ....

More debugging

$ export LD_DEBUG=libs,files
$ ./bmdl
./bmdl: error while loading shared libraries: libmkl_intel_lp64.so: cannot open shared object file: No such file or directory
$ ldd ./bmdl
     15133: 
     15133: file=libtermcap.so.2 [0];  needed by /bin/sh [0]
     15133: find library=libtermcap.so.2 [0]; searching
     15133:  search path=/g/software/intelXE/composer_xe_2011_sp1/mkl/lib/intel64/tls/x86_64:/g/software/intelXE/composer_xe_2011_sp1/mkl/lib/intel64/tls:/g/software/intelXE/composer_xe_2011_sp1/mkl/lib/intel64/x86_64:/g/software/intelXE/composer_xe_2011_sp1/mkl/lib/intel64:/g/software/intelXE/composer_xe_2011_sp1/lib/intel64/tls/x86_64:/g/software/intelXE/composer_xe_2011_sp1/lib/intel64/tls:/g/software/intelXE/composer_xe_2011_sp1/lib/intel64/x86_64:/g/software/intelXE/composer_xe_2011_sp1/lib/intel64        (LD_LIBRARY_PATH)
     ....      

Seems LD_DEBUG doesn't have effect on running ./bmdl alone.

I just realized the old bmdl has a 'setgid' flag and my new built doesn't have it. Maybe that was the cause?

-rwxr-sr-x 1 root gants 1123992 Jul 23 16:14 /scratch/helpdesk/bmdl

I removed the setgid bit from the old bmdl and running ./bmdl doesn't complain about library not found. Now the question is why setgid can interfere with dynamically linked library?

It happens that setgid with a dynamically linked executable can cause security problems and is greatly limited by GNU glibc. For example, LD_LIBRARY_PATH will be ignored. Maybe the old built never worked before?!

Answer 1

Using MKL reqires env variables including INCLUDE, MKLROOT, LD_LIBRARY_PATH, LIBRARY_PATH, CPATH, FPATH and NLSPATH being properly set.

This can be done by a single script provided by Intel.

If you use intel compiler,

  $ source ${intel_dir}/bin/compilervars.sh intel64

If you use MKL only with gcc compiler,

  $ source ${intel_dir}/mkl/bin/mklvars.sh intel64

You can add this cmd line to your .bashrc so you don't need to run it every time.