Linux kernel live debugging, how it's done and what tools are used?

Question

What are the most common and why not uncommon methods and tools used to do live debugging on the Linux kernel? I know that Linus for eg. is against this kind of debugging for the Linux Kernel or it least was and thus nothing much has been done in that sense in those years, but honestly a lot of time has passed since 2000 and i am interested if that mentality has changed regarding the Linux project and what current methods are used to do live debugging on the Linux kernel at the moment(either local or remote)?

References to walkthroughs and tutorials on mentioned techniques and tools are welcome.

Answer 1

Another option is to use ICE/JTAG controller, and GDB. This 'hardware' solution is especially used with embedded systems,

but for instance Qemu offers similar features:

• start qemu with a gdb 'remote' stub which listens on 'localhost:1234' : qemu -s ...,

• then with GDB you open the kernel file vmlinux compiled with debug information (you can take a look a this mailing list thread where they discuss the unoptimization of the kernel).

• connect GDB and Qemu: target remote localhost:1234

• see your live kernel:

  (gdb) where
  #0  cpu_v7_do_idle () at arch/arm/mm/proc-v7.S:77
  #1  0xc0029728 in arch_idle () atarm/mach-realview/include/mach/system.h:36
  #2  default_idle () at arm/kernel/process.c:166
  #3  0xc00298a8 in cpu_idle () at arch/arm/kernel/process.c:199
  #4  0xc00089c0 in start_kernel () at init/main.c:713
  

unfortunately, user-space debugging is not possible so far with GDB (no task list information, no MMU reprogramming to see different process contexts, ...), but if you stay in kernel-space, that's quite convenient.

• info threads will give you the list and states of the different CPUs

EDIT:

You can get more details about the procedure in this PDF:

Debugging Linux systems using GDB and QEMU.