execve file not found when stracing the very same file!

Question

someone i know encountered a problem when running 'lmutil' so i asked them to strace -f lmutil. Why is execve failing with "No such file"!!! It makes no sense, since I am straceing the very same file!! What exactly is going on here???

strace -f /home/tabitha/Starprogram/FLEXlm_11.7/linux-x86_64-2.3.4/bin/lmutil

Output:

execve("/home/tabitha/Starprogram/FLEXlm_11.7/linux-x86_64-2.3.4/bin/lmutil", ["/home/tabitha/Starprogram/FLEXlm"...], [/* 38 vars */]) = -1 ENOENT (No such file or directory)
dup(2)                                  = 3
fcntl(3, F_GETFL)                       = 0x8002 (flags O_RDWR|O_LARGEFILE)
fstat(3, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 1), ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fd7cb8b0000
lseek(3, 0, SEEK_CUR)                   = -1 ESPIPE (Illegal seek)
write(3, "strace: exec: No such file or di"..., 40strace: exec: No such file or directory
) = 40
close(3)                                = 0
munmap(0x7fd7cb8b0000, 4096)            = 0
exit_group(1)                           = ?

ldd output

$ ldd ./lmutil
        linux-vdso.so.1 =>  (0x00007fffcd5ff000)
        libpthread.so.0 => /lib/libpthread.so.0 (0x00007fe40ebbe000)
        libm.so.6 => /lib/libm.so.6 (0x00007fe40e93b000)
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00007fe40e724000)
        libc.so.6 => /lib/libc.so.6 (0x00007fe40e3a1000)
        libdl.so.2 => /lib/libdl.so.2 (0x00007fe40e19d000)
        /lib64/ld-lsb-x86-64.so.3 => /lib64/ld-linux-x86-64.so.2 (0x00007fe40edf5000)

$ find . -name lmutil -exec file {} \;
./bin.linux.x86_64/lmutil: ELF 64-bit LSB executable, AMD x86-64, version 1 (SYSV), for GNU/Linux 2.4.0, dynamically linked (uses shared libs), for GNU/Linux 2.4.0, stripped
./bin.linux.x86/lmutil: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), for GNU/Linux 2.2.5, dynamically linked (uses shared libs), for GNU/Linux 2.2.5, stripped
./lmutil: Bourne shell script text executable

Answer 1

The file you're trying to execute (…/lmutil) exists but its “loader” doesn't exist, where

• the loader of a native executable is its dynamic loader, for example /lib/ld-linux.so.2;
• the loader of a script is the program mentioned on its shebang line, e.g., /bin/sh if the script begins with #!/bin/sh.

From the name of the directory, there's a good chance that lmutil is an amd64 Linux binary, looking for /lib64/ld-linux-x86-64.so.2 as its loader, but you have an amd64 Linux kernel running a 386 (i.e. 32-bit) userland. You need to get suitable binaries for your platform.

I consider this situation to be Unix's most misleading error message. Unfortunately fixing it would be hard: the kernel can only report a numeric error code to the caller of the program, so it only has room for “command not found” (ENOENT) and not for the name of the loader it's looking for. This is one of these rare cases where strace doesn't help.