"Backtrace stopped: previous frame identical to this frame (corrupt stack?)" on ARM linux
I am trying to debug a segfault while running an application on linux ( ARM architecture). I copied the core dump file and tried to get the backtrace using arm-gdb on x86_64 host. This is the o/p:
$ arm-arago-linux-gnueabi-gdb test_slave6_slave core
GNU gdb (GDB) 7.4
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "--host=i686-oesdk-linux --target=arm-oe-linux-gnueabi".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /home/dvdk/test_slave6_slave...done.
warning: exec file is newer than core file.
[New LWP 6411]
[New LWP 6410]
warning: Could not load shared library symbols for 12 libraries, e.g. /lib/libjson-c.so.2.
Use the "info sharedlibrary" command to see the complete listing.
Do you need "set solib-search-path" or "set sysroot"?
Core was generated by `/usr/bin/test_slave6_slave 5 111.111.111.111 1 2 1 2'.
Program terminated with signal 11, Segmentation fault.
#0 0x47b61dd4 in ?? ()
(gdb) bt
#0 0x47b61dd4 in ?? ()
#1 0x47b2e0fc in ?? ()
#2 0x47b2e0fc in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb)
(The timestamp warning might be because I copied the core file first )
I don't think this issue and my issue is same as I am getting valid address. I think it is still a stack corruption issue. But I am not sure how to debug this further. Should I use GDB or valgrind? Can some one point me towards the correct steps for debugging using these tools. For example, should I use memecheck with valgrind with leak_check?
UPDATE 1: I am using libcurl for https requests. I have noticed that the crash does not happen when another version of libcurl without ssl support is used. (query will fail ofcourse). The ssl enabled libcurl was custom compiled by me.
Answer
For posterity's sake:
The problem here is most likely simply that the debugger is unable to examine the same shared libraries that the program was run using, just as it said in the messages it printed.
Older versions of GDB (e.g. 7.4 as used in the question) need the libraries to be arranged in exactly the same tree structure as they are on the target system. So for it you would need to copy all the libraries the binary used into a directory with a hierarchy exactly as it is on the target host. If you name this directory target then you can tell GDB to find the libraries in it with set sysroot ./target/
It seems as if more modern versions of GDB can only find every library if they are all collected into one directory and then you can tell GDB where you put them using set solib-absolute-prefix and set solib-search-path to both point at this directory. I sometimes still have some trouble getting this to work correctly the first time and I have to retry from the beginning several times to get all the incantations correct and in the correct order.
Here I use a more recent build of GDB with all the libraries in ~/tmp/arm-lib. Note that I did not put the core filename on the command-line!
$ gdb ktest-arm
GNU gdb (GDB) 7.12.0.20161109-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ktest-arm...done.
(gdb) show architecture
The target architecture is set automatically (currently arm)
(gdb) set solib-absolute-prefix ~/tmp/arm-lib/
(gdb) set solib-search-path ~/tmp/arm-lib/
(gdb) core-file ~/tmp/ktest-arm-core
warning: core file may not match specified executable file.
[New LWP 905]
Core was generated by `./ktest'.
Program terminated with signal SIGQUIT, Quit.
#0 0x40134264 in nanosleep () from /home/woods/tmp/arm-lib/libc.so.6
(gdb) bt
#0 0x40134264 in nanosleep () from /home/woods/tmp/arm-lib/libc.so.6
#1 0x00008c9c in main () at /home/woods/tmp/testing/ktest.c:9
(gdb)