I want to comprehend the exact difference between these two types of attack. From what I have read:
Buffer Overflow: It overwrites the ret address on the stack to point to another section of the code where the malicious code is inserted. So effectively - here we need to modify the source code of the program to actually carry out the attack.
Return to Libc- Here instead of modifying the source code, run time function calls provided by the C library are used (to say open up a shell). Here the parameters used for the function call are also passed in the overwriting buffer, ending up after the ret part of the stack.
Is the above an accurate description ?
And another related question - would it be possible to have a buffer overflow attack without actually modifying the source code of the original program? Probably if we write a new program and allow that to modify certain sections of memory (which is the new ret address in the corrupted stack of the original program). Then again, I think this might not be possible due to memory protection offered between processes in the kernel.
In the classic buffer overflow exploit, the stack buffer being overflowed was filled with both the machine code to be executed (called the shellcode, because it typically invoked a shell process) and the new return address. The new return address would be crafted to point back within the overflowed stack buffer itself. Obviously, this requires knowing or guessing the address of that stack buffer in the attacked process.
In those days, the memory layout of processes was typically highly deterministic - the location of the stack buffer could usually be predicted quite well by the attacker (particularly if they knew exactly which version of the target software was being attacked). To improve the chances of success when there was some guesswork involved, the active shellcode would often be preceeded by a large quantity of executable machine code that performed no useful operation - called a "NOP sled" or "NOP slide", where "NOP" is the typical name for a machine code instruction that performs "No Operation". Returning to a point anywhere in the the NOP sled would have the desired effect.
A "return-to-libc" exploit, on the other hand, does not cause the hijacked process to return directly to the shellcode. Instead, it causes the process to return, one by one, to the start of a chain of library functions. These library functions might directly perform the operations the attacker wants, but more commonly they will be used to indirectly execute the attacker's shellcode.