What happens to malloc'ed memory after exec() changes the program image?
I know that when I call one of the exec() system calls in Linux that it will replace the currently running process with a new image. So when I fork a new process and run exec(), the child will be replaced with the new process.
What happens to any memory I've allocated from the heap? Say I want to parse an arbitrary number of commands and send it into exec(). To hold this arbitrary number, I'll likely have to allocate memory at some point since I don't think I can do it correctly with static sized arrays, so I'll likely use malloc() or something equivalent.
I need to keep this memory allocated until after I've called exec(), but exec() never returns.
Does the memory get reclaimed by the operating system?
Answer
When you call fork(), a copy of the calling process is created. This child process is (almost) exactly the same as the parent, i.e. memory allocated by malloc() is preserved and you're free to read or modify it. The modifications will not be visible to the parent process, though, as the parent and child processes are completely separate.
When you call exec() in the child, the child process is replaced by a new process. From execve(2):
execve() does not return on success, and the text, data, bss, and stack
of the calling process are overwritten by that of the program loaded.
By overwriting the data segment, the exec() call effectively reclaims the memory that was allocated before by malloc().
The parent process is unaffected by all this. Assuming that you allocated the memory in the parent process before calling fork(), the memory is still available in the parent process.
EDIT: Modern implementations of malloc() use anonymous memory mappings, see mmap(2). According to execve(2), memory mappings are not preserved over an exec() call, so this memory is also reclaimed.