This section will cover how to debug a guest application with QEMU and GDB, and will cover different methods of debugging such as:

Intrusive debugging (debugging so that when a breakpoint is hit, the kernel is paused as well)
Non-intrusive debugging (debugging so that when a breakpoint is hit, the kernel is not paused)

Since QEMU emulates the CPU, the breakpoints set by GDB are hardware breakpoints, not software breakpoints.

Differences Between Zynq UltraScale+ MPSoC and Versal Adaptive SoC

The examples on this page are done on the Zynq UltraScale+ MPSoC platform.

On Versal Adaptive SoC, the differences are:

2 ARM-A CPUs instead of 4

Acquiring the Tools

The tools mentioned in this guide are:

aarch64-none-elf-gdb
aarch64-linux-gnu-gdb
arm-none-eabi-gdb
gdb-multiarch
mb-gdb

aarch64-none-elf-gdb, aarch64-linux-gnu-gdb, arm-none-eabi-gdb, and mb-gdb are bundled with PetaLinux, Yocto, and Vitis tools.
gdb-multiarch can be downloaded through your package manager (e.g. apt-get).

Kernel-Intrusive Debugging

Kernel-intrusive debugging allows you to debug the kernel or bare-metal image on your guest, and has the added benefit of being easier to set up.

The main disadvantage of intrusive debugging with QEMU is only being able to capture SIGINT and SIGTRAP signals.
This means if your program has a segmentation fault and a SIGSEGV is emitted, it will not be caught and your program will exit.

Intrusive debugging requires QEMU parameters to connect to its GDB server, and additional GDB commands in order to start a debugging session.

Enabling a GDB connection to QEMU

QEMU contains a GDB server that you can connect to, allowing you to debug your QEMU application.

To enable connection to the GDB server, you need to pass in a parameter to QEMU that specify the hostname and port it should listen on.

QEMU parameter	Info

QEMU parameter	Info
`-gdb tcp:<hostname>:<port>`	Makes QEMU's GDB server listen on host hostname on port port. Generally the hostname is "localhost" and the port can be anything, as long as you can connect to it.
`-gdb tcp:<hostname>:<port> -S`	Makes QEMU's GDB server listen on host hostname on port port and makes emulation start in a paused state. This can allow you to debug the boot sequence of your virtual machine. To un-pause emulation, connect to QEMU using GDB and use the continue command.
`-s`	Shorthand for `-gdb tcp::1234`.

-gdb tcp:localhost:9000

If booting QEMU using PetaLinux, the primary machine will typically listen on localhost:9000.
For example, if booting a ZCU102 machine using PetaLinux, the ARM machine will listen on localhost:9000, while the Microblaze machine will not have remote debugging enabled.

To simultaneously debug both MicroBlaze and ARM machines in a multi-arch environment with PetaLinux, use the --pmu-qemu-args='-gdb tcp:<hostname>:<port>' argument to enable debugging on the MicroBlaze QEMU machine.

Connecting GDB to QEMU

To connect GDB to QEMU, you need to use the GDB that corresponds to your target's architecture.

For example, some of the architectures in the AMD QEMU environments are:

GDB	Architecture

GDB	Architecture
`aarch64-none-elf-gdb` `aarch64-linux-gnu-gdb`	ARM A53, ARM A72
`arm-none-eabi-gdb`	ARM R5 `arm-none-eabi-gdb` may give a 'g' packet error when connecting to QEMU, depending if it is built to handle the `aarch64` architecture or not. This is because QEMU will give register information for the first CPU cluster, which are the ARM A72 or ARM A53 CPUs. If this happens, you can debug the ARM R5s with `gdb-multiarch`.
`gdb-multiarch`	Multiple ARM architectures, and others (excluding Microblaze)
`mb-gdb`	Microblaze

In this guide, we will use gdb-multiarch to debug the ARM CPUs.

To connect to QEMU's GDB server using your host GDB, you need to create a remote connection.

Once you are connected, you can debug your emulated environment like you would debug any other program.

GDB command	Info

GDB command	Info
`target remote <hostname>:<port>` `target remote :<port>`	Attempts to connect to host hostname on port port. If no hostname is specified, GDB will use localhost.
`target extended-remote <hostname>:<port>` `target extended-remote :<port>`	Attempts to connect to host hostname on port port, and will remain connected after the debugged program exits or GDB detaches from it. If no hostname is specified, GDB will use localhost.

$ aarch64-none-elf-gdb hello_a53.elf
GNU gdb (Linaro GDB 2018.04) 8.0.50.20171128-git
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "--host=x86_64-unknown-linux-gnu --target=aarch64-none-elf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /proj/xhdsswstaff/saipava/epdsw1/work/hello_a53.elf...done.
(gdb)
(gdb) target remote:9000
Remote debugging using :9000
_vector_table () at asm_vectors.S:208
208 b _boot
(gdb) b main
Breakpoint 1 at 0x1740: file ../src/main.c, line 75.
(gdb) c
Continuing.

Thread 1 "" hit Breakpoint 1, main () at ../src/main.c:75
75 BootStatus = XPm_GetBootStatus();
(gdb)

Non-Kernel-Intrusive Debugging

Rather than running GDB on the host machine and using the GDB server provided by QEMU, it's possible to run GDB on the guest machine as long as the guest supports it.

This can provide a series of advantages, such as:

Only debugging your program (assuming you're not debugging your kernel)
Being able to catch more signals, such as SIGSEGV
Not losing control of GDB to other signals, such as SIGTRAP

The disadvantages are that it is more work to install GDB on the guest machine, assuming it can be installed at all, and it uses storage space on the guest.

Installing GDB on the Guest

For this example, we will run GDB on a Linux guest, on the 64-bit ARM-A CPUs on a Zynq UltraScale+ MPSoC QEMU machine.
The machine's image will be a default PetaLinux ZCU102 image, and we will install GDB by copying the files from the host using SCP.

Download the ARM64 GDB package from here and save it somewhere.
Unpack the package. This should produce the folders DEBIAN, etc, and usr.
$ sudo dpkg-deb -R gdb_7.12-6_arm64.deb .
Compress the etc and usr folders. Optionally delete the DEBIAN, etc, and usr folders when you are done.
On a default PetaLinux image, zip and tar files are supported.
$ zip -r gdb.zip etc usr
On the guest, copy the compressed file via SCP and extract it.
root@xilinx-zcu102-2020_2:~# scp <host user>@<host IP>:/path/to/gdb.zip . root@xilinx-zcu102-2020_2:~# unzip gdb.zip
On the guest, copy the extracted contents to root.
root@xilinx-zcu102-2020_2:~# cp -r etc usr /

Running GDB on the Guest

GDB can be run from the guest as if you would normally run it from the host.

If your binary and source files are not on the guest machine, copy them from the host before running GDB.

root@xilinx-zcu102-2020_2:~# scp <host user>@<host IP>:/scratch/proj/gdb-test/segfault.c .
<host user>@<host IP>'s password: 
segfault.c                                                  100%  171     0.2KB/s   00:00    
root@xilinx-zcu102-2020_2:~# scp <host user>@<host IP>:/scratch/proj/gdb-test/segfault.elf .
<host user>@<host IP>'s password: 
segfault.elf                                                100%   11KB  10.6KB/s   00:00    
root@xilinx-zcu102-2020_2:~# gdb
gdb: /lib/libncurses.so.5: no version information available (required by gdb)
gdb: /lib/libncurses.so.5: no version information available (required by gdb)
gdb: /lib/libncurses.so.5: no version information available (required by gdb)
gdb: /lib/libtinfo.so.5: no version information available (required by gdb)
GNU gdb (Debian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "aarch64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".
(gdb) file segfault.elf
Reading symbols from segfault.elf...done.
(gdb) r
Starting program: /home/root/segfault.elf 

Program received signal SIGSEGV, Segmentation fault.
0x00000000004005d8 in main () at segfault.c:8
8	    *p_null = 0xAA55AA55;
(gdb)

GDB Commands

This section will cover GDB commands used regardless if GDB is used remotely or locally.

This section covers a small subset of what is available. A full list can be found here.

Loading Symbols

GDB requires symbols from the program being executed, otherwise it won't know anything about the program and won't be able to debug.

You can still debug your application without symbols loaded, however it will be much more difficult.

Command	Info

Command

Info

--args <file.elf>

Loads the symbols in file.elf into GDB
If file.elf does not contain debugging symbols, it must be recompiled with the -g flag passed into gcc.

This option is used when starting GDB, for example: gdb --args file.elf

symbol-file <file.elf>

Loads the symbols in file.elf into GDB.
If file.elf does not contain debugging symbols, it must be recompiled with the -g flag passed into gcc.

symbol-file can only store symbols from one file at a time.

add-symbol-file <file.elf> <addr>

Loads the symbols in file.elf into GDB.
If file.elf does not contain debugging symbols, it must be recompiled with the -g flag passed into gcc.

addr is the start address of the .text section in file.elf. To find the address of .text you can do:

readelf -WS file.elf | grep .text

(gdb) symbol-file test.elf
Reading symbols from test.elf...done.

Controlling Execution

Command	Info

Command	Info
`r` `run`	Starts execution
`CTRL+C`	Pauses execution.
`c` `continue`	Continues execution.
`kill`	Kills the current process.
`q` `quit`	Quits GDB.

komlodi@machine:/scratch/gdb-test$ gdb test.elf
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from test.elf...done.
(gdb) r
Starting program: /scratch/gdb-test/test.elf
^C
Program received signal SIGINT, Interrupt.
main () at test.c:23
23	    while(1);
(gdb) c
Continuing.
^C
Program received signal SIGINT, Interrupt.
main () at test.c:23
23	    while(1);
(gdb) kill
Kill the program being debugged? (y or n) y
(gdb) q
komlodi@machine:/scratch/gdb-test$

Breakpoints and Watchpoints

Breakpoints allow you to pause execution at a specific point in your code. This allows you to observe program state at a specific point in time.

Watchpoints allow you to pause execution when the value of an expression changes. This can include a variable or address being modified, or a more complicated expression such as "var < 10".

Command	Info

Command	Info
`info break`	Gives you information on all breakpoints in your GDB session.
`info watch`	Gives you information on all watchpoints in your GDB session.
`b <file:line-number\|function-name> [if <cond>]`	Adds a breakpoint at line line-number in file or at the start of function function-name. If cond exists, the breakpoint will only trigger when cond is met. The breakpoint number generated here is used when disabling or deleting a breakpoint.
`watch <var>` `watch <addr>` `watch <expr>`	Adds a watchpoint that pauses program when variable var or address addr are modified. Watchpoints can have more complicated expressions with C-like syntax. Watchpoints will trigger after the instruction is executed, meaning GDB will pause on the line after the watchpoint triggers. In a non-local environment, GDB does not support reverse-step to see the exact line where the trigger occurred. When using a watchpoint with an expression, GDB will evaluate the expression and pause when it is non-zero.
`disable <breakpoint-number>`	Disables breakpoint breakpoint-number
`enable <breakpoint-number>`	Enables breakpoint breakpoint-number
`d <breakpoint-number>` `delete <breakpoint-number>`	Deletes breakpoint breakpoint-number

Reading symbols from test.elf...done.
(gdb) b main
Breakpoint 1 at 0x4005de: file test.c, line 20.
(gdb) watch global_var & 0x00000003
Hardware watchpoint 2: global_var & 0x00000003
(gdb) r
Starting program: /scratch/gdb-test/test.elf 

Breakpoint 1, main () at test.c:20
20	{
(gdb) c
Continuing.

Hardware watchpoint 2: global_var & 0x00000003

Old value = 0
New value = 3
foo (s=0x7fffffffe1c0) at test.c:38
38	    if (s->str == NULL) {
(gdb) p/x global_var
$1 = 0xcc33cc33
(gdb) info break
Num     Type           Disp Enb Address            What
1       breakpoint     keep y   0x00000000004005de in main at test.c:20
	breakpoint already hit 1 time
2       hw watchpoint  keep y                      global_var & 0x00000003
	breakpoint already hit 1 time
(gdb) info watch
Num     Type           Disp Enb Address            What
2       hw watchpoint  keep y                      global_var & 0x00000003
	breakpoint already hit 1 time
(gdb) d 1
(gdb) d 2
(gdb) info break
No breakpoints or watchpoints.
(gdb)

Stepping through your program

When paused, GDB allows you to control the program flow in a variety of ways.

Command	Info

Command	Info
`s`	Steps through one line of your program. This will go inside functions, including library functions.
`n`	Steps through one line of your program. This will not go inside functions.
`si`	Steps through one instruction of your program. This will go inside functions, including library functions.
`ni`	Steps through one instruction of your program. This will not go inside functions.
`finish`	Executes until the end of the current function.

Reading symbols from test.elf...done.
(gdb) b main
Breakpoint 1 at 0x4005de: file test.c, line 20.
(gdb) r
Starting program: /scratch/gdb-test/test.elf 

Breakpoint 1, main () at test.c:20
20	{
(gdb) n
23	    foo(&s);
(gdb) s
foo (s=0x7fffffffe1c0) at test.c:30
30	    s->str = "This is a string";
(gdb) n
31	    memset(s->arr, 0xAA, sizeof(s->arr));
(gdb) n
32	    s->var = 1234;
(gdb) c
Continuing.

Stack and frame information

Command	Info

Command	Info
`backtrace`	Prints the call stack, in the order that functions were called, as a list of frames
`frame <frame-number>`	Changes the current frame being observed to frame-number. This does not modify program execution.
`info frame`	Gives you information on the current frame.
`info variables`	Gives you information of all static/global variables and symbols in your program. On some systems, this may print a lot of information.
`info locals`	Gives you information on local variables in the current frame.
`info args`	Gives you information on arguments passed into the current frame.

Breakpoint 1, foo (s=0x7fffffffe1c0) at test.c:30
30	    s->str = "This is a string";
(gdb) backtrace
#0  foo (s=0x7fffffffe1c0) at test.c:30
#1  0x00000000004005f9 in main () at test.c:23
(gdb) frame 0
#0  foo (s=0x7fffffffe1c0) at test.c:30
30	    s->str = "This is a string";
(gdb) info frame
Stack level 0, frame at 0x7fffffffe1c0:
 rip = 0x400620 in foo (test.c:30); saved rip = 0x4005f9
 called by frame at 0x7fffffffe200
 source language c.
 Arglist at 0x7fffffffe1b0, args: s=0x7fffffffe1c0
 Locals at 0x7fffffffe1b0, Previous frame's sp is 0x7fffffffe1c0
 Saved registers:
  rbp at 0x7fffffffe1b0, rip at 0x7fffffffe1b8
(gdb) info locals
No locals.
(gdb) info args
s = 0x7fffffffe1c0
(gdb) frame 1
#1  0x00000000004005f9 in main () at test.c:23
23	    foo(&s);
(gdb) info locals
s = {var = 0, arr = "\000\000\000\000\000\000\000", 
  str = 0x4006b0 <__libc_csu_init> "AWAVA\211\377AUATL\215%N\a ", c = -32 '\340', num = 0}
(gdb) info args
No arguments.
(gdb)

Printing Variables

gdb allows you to print out variable information using expressions. The expressions use C-like syntax.

Command	Info

Command	Info
`print <var>` `p <var>` `p <file>::<var>` `p '<file>'::<file>`	Prints var. By default, GDB will print var based off of what type it is.
`p *<arr>@<len>`	Prints the first len values of arr.
`p/x <var>`	Prints var as a hexadecimal number
`p/d <var>`	Prints var as a signed integer.
`p/t <var>`	Prints var as a binary number.
`p/c <var>`	Prints var as a character.
`ptype <var>` `ptype <type>`	Prints type definition of var, or the type definition of type. This is useful when getting information of a struct.

Reading symbols from test.elf...done.
(gdb) b foo
Breakpoint 1 at 0x400620: file test.c, line 30.
(gdb) r
Starting program: /scratch/gdb-test/test.elf 

Breakpoint 1, foo (s=0x7fffffffe1c0) at test.c:30
(gdb) ptype s
type = volatile struct {
    uint32_t var;
    uint8_t arr[8];
    char *str;
    char c;
} *
30	    s->str = "This is a string";
(gdb) n
31	    memset(s->arr, 0xAA, sizeof(s->arr));
(gdb) n
32	    s->c = 'z';
(gdb) n
33	    s->var = 1;
(gdb) n
34	    global_var = 0xCC33CC33;
(gdb) p s->str
$1 = 0x400734 "This is a string"
(gdb) p *s->str@4
$2 = "This"
(gdb) p/c s->str[0]
$3 = 84 'T'
(gdb) p/x s->arr
$4 = {0xaa, 0xaa, 0xaa, 0xaa, 0xaa, 0xaa, 0xaa, 0xaa}
(gdb) p/t s->arr
$5 = {10101010, 10101010, 10101010, 10101010, 10101010, 10101010, 10101010, 10101010}
(gdb) p/d s->var
$7 = 1

Modifying Variables

Command	Info

Command	Info
`set <var>=<val>`	Sets the variable var to the value val.

foo (s=0x7fffffffe1d0) at test.c:38
38	    if (s->str == NULL) {
(gdb) p s->str
$1 = 0x400734 "This is a string"
(gdb) set s->str=0x00
(gdb) n
39	        puts("s contains a NULL string");

Macros

GDB provides commands to expand, list, and define macros.
In this case, 'macro' refers to any preprocessor definition, not just one that takes arguments. For example, each of the following lines would be considered a macro:

#define REG_FIELD_0_MASK    0x01
#define REG_FIELD_1_MASK    (1 << 1)
#define REG_FIELD_X_MASK(x) (1 << x)

Most compilers don't create macro information by default. For example, on gcc, you would create macro information by passing in the -g3 flag at compile time.

Command	Info

Command	Info
`macro expand <expr>` `macro exp <expr>`	Expands, but does not parse, all preprocessor macros in expr.
`info macro [-a\|-all] <macro>`	Shows the current definition of macro macro, and where it was declared. If the `-a` flag is passed, all definitions of macro will be shown.
`info macros <function>`	Shows all macros that are currently defined in function. This will also print all macro definitions defined in function, including ones from standard libraries.

Breakpoint 1, main () at gdb-macro.c:13
13	    for (i=0; i<sizeof(reg) * 8; ++i) {
(gdb) n
14	        printf("bit %.2d: %d\n", i, REG_BIT_EXTRACT(reg, i));
(gdb) info macro REG_BIT_EXTRACT
Defined at /scratch/test-code/gdb-macro.c:5
#define REG_BIT_EXTRACT(reg, bit) (!!GET_BIT(reg, bit))
(gdb) info macro GET_BIT
Defined at /scratch/test-code/gdb-macro.c:4
#define GET_BIT(val, bit) (val & (1 << bit))
(gdb) macro exp REG_BIT_EXTRACT(reg, i)
expands to: (!!(reg & (1 << i)))
(gdb) macro exp REG_BIT_EXTRACT(0xAA55AA55, 0)
expands to: (!!(0xAA55AA55 & (1 << 0)))
(gdb) p REG_BIT_EXTRACT(0xAA55AA55, 0)
$1 = 1

Signals

When debugging your code it may be useful to manage signals, particularly if they keep interrupting your GDB session.

QEMU GDB server is only capable of handling SIGINT and SIGTRAP signals. Meaning if your program receives a SIGSEGV signal, it will not be passed to GDB.

Command	Info