What Happens When Stack and Heap Collide

Question

I am curious to know what happens when the stack and the heap collide. If anybody has encountered this, please could they explain the scenario.

Answer 1

In a modern languages running on a modern OS, you'll get either a stack overflow (hurray!) or malloc() or sbrk() or mmap() will fail when you try to grow the heap. But not all software is modern, so let's look at the failure modes:

• If the stack grows into the heap, the typically C compiler will silently start to overwrite the heap's data structures. On a modern OS, there will be one or more virtual memory guard pages which prevent the stack from growing indefinitely. As long as the amount of memory in the guard pages is at least as large as the size of the growing procedure's activation record, the OS will guarantee you a segfault. If you're DOS running on a machine with no MMU, you're probably hosed.

• If the heap grows into the stack, the operating system should always be aware of the situation and some sort of system call will fail. The implementation of malloc() almost certainly notices the failure and returns NULL. What happens after that is up to you.

I'm always amazed at the willingness of compiler writers to hope that the OS puts guard pages in place to prevent stack overflow. Of course, this trick works well until you start having thousands of threads, each with its own stack...