My program is statically compiled with dietlibc. It is compiled on ubuntu x64 (compiled for x86 using the -m32 flag) and is run on a centos x86.
The compiled size is only about 100KB. I compile it with -ggdb3 and no optimization flags.
My program uses signal.h to handle a SIGSEGV signal and then calls abort().
The program runs without problems for days but sometimes segfaults. This is when I get weird backtraces that I do not understand:
username@ubuntu:~/Desktop$ gdb -c core.28569 program-name GNU gdb (GDB) 7.2 Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "--host=x86_64-linux-gnu --target=i386-linux-gnu". For bug reporting instructions, please see: ... Reading symbols from program-name...done. [New Thread 28569] Core was generated by `program-name'. Program terminated with signal 6, Aborted. #0 0x00914410 in __kernel_vsyscall () Setting up the environment for debugging gdb. Function "internal_error" not defined. Make breakpoint pending on future shared library load? (y or [n]) [answered N; input not from terminal] Function "info_command" not defined. Make breakpoint pending on future shared library load? (y or [n]) [answered N; input not from terminal] .gdbinit:8: Error in sourced command file: Argument required (one or more breakpoint numbers). (gdb) bt #0 0x00914410 in __kernel_vsyscall () During symbol reading, incomplete CFI data; unspecified registers (e.g., eax) at 0x914411. #1 0x0804d7f4 in __unified_syscall () #2 0xbf8966c0 in ?? () #3 #4 0x2054454e in ?? () #5 0x20524c43 in ?? () #6 0x2e352e33 in ?? () #7 0x32373033 in ?? () #8 0x2e203b39 in ?? () #9 0x2054454e in ?? () #10 0x20524c43 in ?? () #11 0x2e302e33 in ?? () #12 0x32373033 in ?? () #13 0x4d203b39 in ?? () #14 0x61696465 in ?? () #15 0x6e654320 in ?? () #16 0x20726574 in ?? () #17 0x36204350 in ?? () #18 0x203b302e in ?? () #19 0x54454e2e in ?? () #20 0x43302e34 in ?? () #21 0x00000029 in ?? () #22 0xbf8989a8 in ?? () Backtrace stopped: previous frame inner to this frame (corrupt stack?) (gdb) bt full #0 0x00914410 in __kernel_vsyscall () No symbol table info available. #1 0x0804d7f4 in __unified_syscall () No symbol table info available. #2 0xbf8966c0 in ?? () No symbol table info available. #3 No symbol table info available. #4 0x2054454e in ?? () No symbol table info available. #5 0x20524c43 in ?? () No symbol table info available. #6 0x2e352e33 in ?? () No symbol table info available. #7 0x32373033 in ?? () No symbol table info available. #8 0x2e203b39 in ?? () No symbol table info available. #9 0x2054454e in ?? () No symbol table info available. #10 0x20524c43 in ?? () No symbol table info available. #11 0x2e302e33 in ?? () No symbol table info available. #12 0x32373033 in ?? () No symbol table info available. #13 0x4d203b39 in ?? () No symbol table info available. #14 0x61696465 in ?? () No symbol table info available. #15 0x6e654320 in ?? () No symbol table info available. #16 0x20726574 in ?? () No symbol table info available. #17 0x36204350 in ?? () No symbol table info available. #18 0x203b302e in ?? () No symbol table info available. #19 0x54454e2e in ?? () No symbol table info available. #20 0x43302e34 in ?? () No symbol table info available. #21 0x00000029 in ?? () No symbol table info available. #22 0xbf8989a8 in ?? () No symbol table info available. Backtrace stopped: previous frame inner to this frame (corrupt stack?) (gdb) quit
Answer
It's a stack overrun.
#4 0x2054454e in ?? ()
That looks like text, " TEN" or "NET "
#5 0x20524c43 in ?? ()
" RLC" or "CLR "
And so on.
Treat the addresses as if they were text - see if you can identify where this text overwrites your stack.