About Kernel Documentation Linux Kernel Contact Linux Resources Linux Blog

Documentation / gdb-kernel-debugging.txt


Based on kernel version 4.8. Page generated on 2016-10-06 23:16 EST.

1	Debugging kernel and modules via gdb
2	====================================
3	
4	The kernel debugger kgdb, hypervisors like QEMU or JTAG-based hardware
5	interfaces allow to debug the Linux kernel and its modules during runtime
6	using gdb. Gdb comes with a powerful scripting interface for python. The
7	kernel provides a collection of helper scripts that can simplify typical
8	kernel debugging steps. This is a short tutorial about how to enable and use
9	them. It focuses on QEMU/KVM virtual machines as target, but the examples can
10	be transferred to the other gdb stubs as well.
11	
12	
13	Requirements
14	------------
15	
16	 o gdb 7.2+ (recommended: 7.4+) with python support enabled (typically true
17	   for distributions)
18	
19	
20	Setup
21	-----
22	
23	 o Create a virtual Linux machine for QEMU/KVM (see www.linux-kvm.org and
24	   www.qemu.org for more details). For cross-development,
25	   http://landley.net/aboriginal/bin keeps a pool of machine images and
26	   toolchains that can be helpful to start from.
27	
28	 o Build the kernel with CONFIG_GDB_SCRIPTS enabled, but leave
29	   CONFIG_DEBUG_INFO_REDUCED off. If your architecture supports
30	   CONFIG_FRAME_POINTER, keep it enabled.
31	
32	 o Install that kernel on the guest.
33	
34	   Alternatively, QEMU allows to boot the kernel directly using -kernel,
35	   -append, -initrd command line switches. This is generally only useful if
36	   you do not depend on modules. See QEMU documentation for more details on
37	   this mode.
38	
39	 o Enable the gdb stub of QEMU/KVM, either
40	    - at VM startup time by appending "-s" to the QEMU command line
41	   or
42	    - during runtime by issuing "gdbserver" from the QEMU monitor
43	      console
44	
45	 o cd /path/to/linux-build
46	
47	 o Start gdb: gdb vmlinux
48	
49	   Note: Some distros may restrict auto-loading of gdb scripts to known safe
50	   directories. In case gdb reports to refuse loading vmlinux-gdb.py, add
51	
52	    add-auto-load-safe-path /path/to/linux-build
53	
54	   to ~/.gdbinit. See gdb help for more details.
55	
56	 o Attach to the booted guest:
57	    (gdb) target remote :1234
58	
59	
60	Examples of using the Linux-provided gdb helpers
61	------------------------------------------------
62	
63	 o Load module (and main kernel) symbols:
64	    (gdb) lx-symbols
65	    loading vmlinux
66	    scanning for modules in /home/user/linux/build
67	    loading @0xffffffffa0020000: /home/user/linux/build/net/netfilter/xt_tcpudp.ko
68	    loading @0xffffffffa0016000: /home/user/linux/build/net/netfilter/xt_pkttype.ko
69	    loading @0xffffffffa0002000: /home/user/linux/build/net/netfilter/xt_limit.ko
70	    loading @0xffffffffa00ca000: /home/user/linux/build/net/packet/af_packet.ko
71	    loading @0xffffffffa003c000: /home/user/linux/build/fs/fuse/fuse.ko
72	    ...
73	    loading @0xffffffffa0000000: /home/user/linux/build/drivers/ata/ata_generic.ko
74	
75	 o Set a breakpoint on some not yet loaded module function, e.g.:
76	    (gdb) b btrfs_init_sysfs
77	    Function "btrfs_init_sysfs" not defined.
78	    Make breakpoint pending on future shared library load? (y or [n]) y
79	    Breakpoint 1 (btrfs_init_sysfs) pending.
80	
81	 o Continue the target
82	    (gdb) c
83	
84	 o Load the module on the target and watch the symbols being loaded as well as
85	   the breakpoint hit:
86	    loading @0xffffffffa0034000: /home/user/linux/build/lib/libcrc32c.ko
87	    loading @0xffffffffa0050000: /home/user/linux/build/lib/lzo/lzo_compress.ko
88	    loading @0xffffffffa006e000: /home/user/linux/build/lib/zlib_deflate/zlib_deflate.ko
89	    loading @0xffffffffa01b1000: /home/user/linux/build/fs/btrfs/btrfs.ko
90	
91	    Breakpoint 1, btrfs_init_sysfs () at /home/user/linux/fs/btrfs/sysfs.c:36
92	    36              btrfs_kset = kset_create_and_add("btrfs", NULL, fs_kobj);
93	
94	 o Dump the log buffer of the target kernel:
95	    (gdb) lx-dmesg
96	    [     0.000000] Initializing cgroup subsys cpuset
97	    [     0.000000] Initializing cgroup subsys cpu
98	    [     0.000000] Linux version 3.8.0-rc4-dbg+ (...
99	    [     0.000000] Command line: root=/dev/sda2 resume=/dev/sda1 vga=0x314
100	    [     0.000000] e820: BIOS-provided physical RAM map:
101	    [     0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
102	    [     0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
103	    ....
104	
105	 o Examine fields of the current task struct:
106	    (gdb) p $lx_current().pid
107	    $1 = 4998
108	    (gdb) p $lx_current().comm
109	    $2 = "modprobe\000\000\000\000\000\000\000"
110	
111	 o Make use of the per-cpu function for the current or a specified CPU:
112	    (gdb) p $lx_per_cpu("runqueues").nr_running
113	    $3 = 1
114	    (gdb) p $lx_per_cpu("runqueues", 2).nr_running
115	    $4 = 0
116	
117	 o Dig into hrtimers using the container_of helper:
118	    (gdb) set $next = $lx_per_cpu("hrtimer_bases").clock_base[0].active.next
119	    (gdb) p *$container_of($next, "struct hrtimer", "node")
120	    $5 = {
121	      node = {
122	        node = {
123	          __rb_parent_color = 18446612133355256072,
124	          rb_right = 0x0 <irq_stack_union>,
125	          rb_left = 0x0 <irq_stack_union>
126	        },
127	        expires = {
128	          tv64 = 1835268000000
129	        }
130	      },
131	      _softexpires = {
132	        tv64 = 1835268000000
133	      },
134	      function = 0xffffffff81078232 <tick_sched_timer>,
135	      base = 0xffff88003fd0d6f0,
136	      state = 1,
137	      start_pid = 0,
138	      start_site = 0xffffffff81055c1f <hrtimer_start_range_ns+20>,
139	      start_comm = "swapper/2\000\000\000\000\000\000"
140	    }
141	
142	
143	List of commands and functions
144	------------------------------
145	
146	The number of commands and convenience functions may evolve over the time,
147	this is just a snapshot of the initial version:
148	
149	 (gdb) apropos lx
150	 function lx_current -- Return current task
151	 function lx_module -- Find module by name and return the module variable
152	 function lx_per_cpu -- Return per-cpu variable
153	 function lx_task_by_pid -- Find Linux task by PID and return the task_struct variable
154	 function lx_thread_info -- Calculate Linux thread_info from task variable
155	 lx-dmesg -- Print Linux kernel log buffer
156	 lx-lsmod -- List currently loaded modules
157	 lx-symbols -- (Re-)load symbols of Linux kernel and currently loaded modules
158	
159	Detailed help can be obtained via "help <command-name>" for commands and "help
160	function <function-name>" for convenience functions.
Hide Line Numbers


About Kernel Documentation Linux Kernel Contact Linux Resources Linux Blog