The #newinstretch game: dbgsym packages in Debian/stretch
Debug packages include debug symbols and so far were usually named <package>-dbg in Debian. Those packages are essential if you’ve to debug failing (especially: crashing) programs. Since December 2015 Debian has automatic dbgsym packages, being built by default. Those packages are available as <package>-dbgsym, so starting with Debian/stretch you should no longer look for -dbg packages but for -dbgsym instead. Currently there are 13.369 dbgsym packages available for the amd64 architecture of Debian/stretch, comparing this to the 2.250 packages which I counted being available for Debian/jessie this is really a huge improvement. (If you’re interested in the details of dbgsym packages as a package maintainer take a look at the Automatic Debug Packages page in the Debian wiki.)
The dbgsym packages are NOT provided by the usual Debian archive though (which is good thing, since those packages are quite disk space consuming, e.g. just the amd64 stretch mirror of debian-debug consumes 47GB). Instead there’s a new archive called debian-debug. To get access to the dbgsym packages via the debian-debug suite on your Debian/stretch system include the following entry in your apt’s sources.list configuration (replace deb.debian.org with whatever mirror you prefer):
deb http://deb.debian.org/debian-debug/ stretch-debug main
If you’re not yet familiar with usage of such debug packages let me give you a short demo.
Let’s start with sending SIGILL (Illegal Instruction) to a running sha256sum process, causing it to generate a so called core dump file:
% sha256sum /dev/urandom & [1] 1126 % kill -4 1126 % [1]+ Illegal instruction (core dumped) sha256sum /dev/urandom % ls core $ file core core: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, from 'sha256sum /dev/urandom', real uid: 1000, effective uid: 1000, real gid: 1000, effective gid: 1000, execfn: '/usr/bin/sha256sum', platform: 'x86_64'
Now we can run the GNU Debugger (gdb) on this core file, executing:
% gdb sha256sum core [...] Type "apropos word" to search for commands related to "word"... Reading symbols from sha256sum...(no debugging symbols found)...done. [New LWP 1126] Core was generated by `sha256sum /dev/urandom'. Program terminated with signal SIGILL, Illegal instruction. #0 0x000055fe9aab63db in ?? () (gdb) bt #0 0x000055fe9aab63db in ?? () #1 0x000055fe9aab8606 in ?? () #2 0x000055fe9aab4e5b in ?? () #3 0x000055fe9aab42ea in ?? () #4 0x00007faec30872b1 in __libc_start_main (main=0x55fe9aab3ae0, argc=2, argv=0x7ffc512951f8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffc512951e8) at ../csu/libc-start.c:291 #5 0x000055fe9aab4b5a in ?? () (gdb)
As you can see by the several “??” question marks, the “bt” command (short for backtrace) doesn’t provide useful information.
So let’s install the according debug package, which is coreutils-dbgsym in this case (since the sha256sum binary which generated the core file is part of the coreutils package). Then let’s rerun the same gdb steps:
% gdb sha256sum core [...] Type "apropos word" to search for commands related to "word"... Reading symbols from sha256sum...Reading symbols from /usr/lib/debug/.build-id/a4/b946ef7c161f2d215518ca38d3f0300bcbdbb7.debug...done. done. [New LWP 1126] Core was generated by `sha256sum /dev/urandom'. Program terminated with signal SIGILL, Illegal instruction. #0 0x000055fe9aab63db in sha256_process_block (buffer=buffer@entry=0x55fe9be95290, len=len@entry=32768, ctx=ctx@entry=0x7ffc51294eb0) at lib/sha256.c:526 526 lib/sha256.c: No such file or directory. (gdb) bt #0 0x000055fe9aab63db in sha256_process_block (buffer=buffer@entry=0x55fe9be95290, len=len@entry=32768, ctx=ctx@entry=0x7ffc51294eb0) at lib/sha256.c:526 #1 0x000055fe9aab8606 in sha256_stream (stream=0x55fe9be95060, resblock=0x7ffc51295080) at lib/sha256.c:230 #2 0x000055fe9aab4e5b in digest_file (filename=0x7ffc51295f3a "/dev/urandom", bin_result=0x7ffc51295080 "\001", missing=0x7ffc51295078, binary=<optimized out>) at src/md5sum.c:624 #3 0x000055fe9aab42ea in main (argc=<optimized out>, argv=<optimized out>) at src/md5sum.c:1036
As you can see it’s reading the debug symbols from /usr/lib/debug/.build-id/a4/b946ef7c161f2d215518ca38d3f0300bcbdbb7.debug and this is what we were looking for.
gdb now also tells us that we don’t have lib/sha256.c available. For even better debugging it’s useful to have the according source code available. This is also just an `apt-get source coreutils ; cd coreutils-8.26/` away:
~/coreutils-8.26 % gdb sha256sum ~/core [...] Type "apropos word" to search for commands related to "word"... Reading symbols from sha256sum...Reading symbols from /usr/lib/debug/.build-id/a4/b946ef7c161f2d215518ca38d3f0300bcbdbb7.debug...done. done. [New LWP 1126] Core was generated by `sha256sum /dev/urandom'. Program terminated with signal SIGILL, Illegal instruction. #0 0x000055fe9aab63db in sha256_process_block (buffer=buffer@entry=0x55fe9be95290, len=len@entry=32768, ctx=ctx@entry=0x7ffc51294eb0) at lib/sha256.c:526 526 R( h, a, b, c, d, e, f, g, K(25), M(25) ); (gdb) bt #0 0x000055fe9aab63db in sha256_process_block (buffer=buffer@entry=0x55fe9be95290, len=len@entry=32768, ctx=ctx@entry=0x7ffc51294eb0) at lib/sha256.c:526 #1 0x000055fe9aab8606 in sha256_stream (stream=0x55fe9be95060, resblock=0x7ffc51295080) at lib/sha256.c:230 #2 0x000055fe9aab4e5b in digest_file (filename=0x7ffc51295f3a "/dev/urandom", bin_result=0x7ffc51295080 "\001", missing=0x7ffc51295078, binary=<optimized out>) at src/md5sum.c:624 #3 0x000055fe9aab42ea in main (argc=<optimized out>, argv=<optimized out>) at src/md5sum.c:1036 (gdb)
Now we’re ready for all the debugging magic. :)
Thanks to everyone who was involved in getting us the automatic dbgsym package builds in Debian!