Library path order for alternate glibc dynamic linker (ld.so)

linux shared-libraries ld glibc dynamic-linking
Matei David picture Matei David · Dec 14, 2013 · Viewed 10.1k times · Source

I need to use an alternate glibc version, newer than the one installed on my system (2.18 vs 2.15). Several related issues are covered here and here. The specific question I'm asking here is the following:

I set up the library path of the new dynamic linker (ld-2.18.so) so that the new libc (libc-2.18.so) is found ahead of the old libc (libc-2.15.so). However, when I try to run a program with the new ld, the old version of libc is picked up, generating a SEGV. Why is that happening?

Note: I know this can be fixed by using --rpath at compile time or LD_LIBRARY_PATH at run time. However, I would still like to understand why one of these is still needed.

The details follow:

I downloaded glibc-2.18 and built it at /opt/glibc-2.18. By default, the file /opt/glibc-2.18/etc/ld.so.conf is missing. I created it, and updated the library cache of the new glibc as follows. I emphasize that: the new libc is found before the old libc:

$ cat /opt/glibc-2.18/etc/ld.so.conf
/opt/glibc-2.18/lib
/usr/local/lib
/lib/x86_64-linux-gnu
/usr/lib/x86_64-linux-gnu
/usr/lib/x86_64-linux-gnu/mesa
/lib
/usr/lib
$ /opt/glibc-2.18/sbin/ldconfig -v |& grep -E '^[^'$'\t'']|libc\.'
/opt/glibc-2.18/sbin/ldconfig: Path `/opt/glibc-2.18/lib' given more than once
/opt/glibc-2.18/sbin/ldconfig: Can't stat /opt/glibc-2.18/lib64: No such file or directory
/opt/glibc-2.18/sbin/ldconfig: Can't stat /opt/glibc-2.18/libx32: No such file or directory
/opt/glibc-2.18/lib:
        libc.so.6 -> libc-2.18.so
/usr/local/lib:
/lib/x86_64-linux-gnu:
        libc.so.6 -> libc-2.15.so
/usr/lib/x86_64-linux-gnu:
/usr/lib/x86_64-linux-gnu/mesa:
/lib:
/usr/lib:

Then, I created a simple C program:

$ cat <<EOF >a.c
> #include <stdio.h>
> int main()
> {
>     fprintf(stdout, "ok\n");
>     return 0;
> }
> EOF
$ g++ a.c
$ file a.out
a.out: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.24, BuildID[sha1]=0x43b8484e3910072375d68418cb6327478266c0e9, not stripped
$ ldd a.out
    linux-vdso.so.1 =>  (0x00007fffd7ffe000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fa7c47bd000)
    /lib64/ld-linux-x86-64.so.2 (0x00007fa7c4b9b000)
$ readelf -a a.out | grep lib
      [Requesting program interpreter: /lib64/ld-linux-x86-64.so.2]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
000000601000  000100000007 R_X86_64_JUMP_SLO 0000000000000000 __libc_start_main + 0
     1: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND __libc_start_main@GLIBC_2.2.5 (2)
    46: 00000000004005f0     2 FUNC    GLOBAL DEFAULT   13 __libc_csu_fini
    52: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND __libc_start_main@@GLIBC_
    57: 0000000000400560   137 FUNC    GLOBAL DEFAULT   13 __libc_csu_init
  000000: Version: 1  File: libc.so.6  Cnt: 1
$ objdump -x a.out | grep -A3 Version
Version References:
  required from libc.so.6:
    0x09691a75 0x00 02 GLIBC_2.2.5

As seen above, this program has the old ld hard-coded inside. I can forcefully run it with the new ld, and I expect the path of the new ld to be used (you can see the new ld.so.cache being opened). However, for some reason I'm trying to understand, the old libc is found before the new libc, generating a SEGV:

$ /opt/glibc-2.18/lib/ld-2.18.so ./a.out
Segmentation fault (core dumped)
$ strace /opt/glibc-2.18/lib/ld-2.18.so ./a.out |& grep open
open("./a.out", O_RDONLY|O_CLOEXEC)     = 3
open("/opt/glibc-2.18/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
open("/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3

I can also compile with the new library and bake-in the new ld as follows:

$ g++ -L/opt/glibc-2.18/lib -Wl,--dynamic-linker=/opt/glibc-2.18/lib/ld-2.18.so a.c -o a.2.18.out
$ file a.2.18.out
a.2.18.out: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.24, BuildID[sha1]=0x25ab43f3d29b49fa21385a15e43325e9fb904e81, not stripped
$ ldd a.2.18.out
    linux-vdso.so.1 =>  (0x00007fffa68da000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f9df5cbe000)
    /opt/glibc-2.18/lib/ld-2.18.so => /lib64/ld-linux-x86-64.so.2 (0x00007f9df609c000)
$ readelf -a a.2.18.out | grep lib
      [Requesting program interpreter: /opt/glibc-2.18/lib/ld-2.18.so]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
000000601000  000100000007 R_X86_64_JUMP_SLO 0000000000000000 __libc_start_main + 0
     1: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND __libc_start_main@GLIBC_2.2.5 (2)
    54: 0000000000400600     2 FUNC    GLOBAL DEFAULT   13 __libc_csu_fini
    60: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND __libc_start_main@@GLIBC_
    65: 0000000000400570   137 FUNC    GLOBAL DEFAULT   13 __libc_csu_init
  000000: Version: 1  File: libc.so.6  Cnt: 1
$ objdump -x a.2.18.out | grep -A3 Version
Version References:
  required from libc.so.6:
    0x09691a75 0x00 02 GLIBC_2.2.5

Still, if I try to run the new program the same thing happens, the old libc is being used instead of the new libc:

$ ./a.2.18.out
Segmentation fault (core dumped)
$ strace ./a.2.18.out |& grep open
open("/opt/glibc-2.18/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
open("/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3

With either executable, specifying LD_LIBRARY_PATH=/opt/glibc-2.18/lib makes it work. However, my question here is why that is still needed, given that the path of the new ld is configured at the beginning to pick up the new libc ahead of the old libc.

Answer

Matei David picture Matei David · Dec 16, 2013

I got it, the issue was with the OS ABI version. That's the number indicated by file, such as:

$ file /lib/x86_64-linux-gnu/libc-2.15.so | grep -o "for GNU/Linux [0-9.]*"
for GNU/Linux 2.6.24

When glibc is configured with nothing other than --prefix, it builds by default with an ABI version smaller(!!) (in my case, 2.6.16) than the default on the system (2.6.24). So libc-2.18 has ABI version smaller than libc-2.15.

When ldconfig finds 2 versions of libc.so.6 with different ABI numbers, it places them in ld.so.cache in order of descending ABI number, not in order of appearance. This can be checked by swapping their locations, rebuilding the cache (with ldconfig), and listing cache contents (with ldconfig -p). Only when 2 libc.so.6 files have the same ABI version, do they get placed in the cache in order of appearance.

Configuring glibc with --enable-kernel=2.6.24 causes it to use the same ABI version as the system, which in turn fixes the resolution issues in the question statement, without the need for an explicit --rpath or LD_LIBRARY_PATH.

Related questions

What's the difference between .so, .la and .a library files?

I know an .so file is a kind of dynamic library (lots of threads can share such libraries so there is no need to have more than one copy of it in memory). But what is the difference between .a …

Read More
Why do I have to define LD_LIBRARY_PATH with an export every time I run my application?

I have some code that uses some shared libraries (c code on gcc). When compiling I have to explicitly define the include and library directories using -I and -L, since they aren't in the standard places. When I try to …

Read More
Linux error while loading shared libraries: cannot open shared object file: No such file or directory

Program is part of the Xenomai test suite, cross-compiled from Linux PC into Linux+Xenomai ARM toolchain. # echo $LD_LIBRARY_PATH /lib # ls /lib ld-2.3.3.so libdl-2.3.3.so libpthread-0.10.so ld-linux.so.2 libdl.so.2 libpthread.so.0 libc-2.3.3.so libgcc_s.…

Read More