Using dlopen() on an executable
I need to call a function from another program. If the other program were a library, I could simply use dlopen and dlsym to get a handle to the function. Unfortunately, the other program is a Unix Executable, and building it as a library is not an option. Trying dlopen() on the executable gives this error message:
dlopen([...]/testprogram, 1): no suitable image found. Did find:
[...]/testprogram: can't map
This isn't surprising, as dlopen is meant for use with libraries, not executables. Is there any way to get dlopen and dlsym to work with executables? If not, is there an alternative way of achieving the same thing?
Answer
You can't open executables as libraries. The entry point of an executable will attempt to re-initialize the C library, and take over the brk pointer. This will corrupt your malloc heap. Additionally, the executable is likely to be mapped at a fixed address with no relocations, and if this address overlaps with anything already loaded, it's not possible to map it for that reason as well.
You need to refactor the other program into a library, or add a RPC interface to the other program.
Note that this does not necessarily apply for PIE executables. However, unless the executable is specifically designed for being dlopen()ed, this is unsafe, as main() will not be run, and any initialization done in main() therefore will not occur.