This section will cover how to debug a guest application with QEMU and GDB, and will cover different methods of debugging such as:

Intrusive debugging (debugging so that when a breakpoint is hit, the kernel is paused as well)
Non-intrusive debugging (debugging so that when a breakpoint is hit, the kernel is not paused)

Since QEMU emulates the CPU, the breakpoints set by GDB are hardware breakpoints, not software breakpoints.

Differences Between Zynq UltraScale+ MPSoC and Versal Adaptive SoC

The examples on this page are done on the Zynq UltraScale+ MPSoC platform.

On Versal Adaptive SoC, the differences are:

2 ARM-A CPUs instead of 4

Acquiring the Tools

The tools mentioned in this guide are:

```
aarch64-none-elf-gdb
```
```
aarch64-linux-gnu-gdb
```
```
arm-none-eabi-gdb
```
```
gdb-multiarch
```
```
mb-gdb
```

aarch64-none-elf-gdb, aarch64-linux-gnu-gdb, arm-none-eabi-gdb, and mb-gdb are bundled with PetaLinux, Yocto, and Vitis tools.
gdb-multiarch can be downloaded through your package manager (e.g. apt-get).

Kernel-Intrusive Debugging

Kernel-intrusive debugging allows you to debug the kernel or bare-metal image on your guest, and has the added benefit of being easier to set up.

The main disadvantage of intrusive debugging with QEMU is only being able to capture SIGINT and SIGTRAP signals.
This means if your program has a segmentation fault and a SIGSEGV is emitted, it will not be caught and your program will exit.

Intrusive debugging requires QEMU parameters to connect to its GDB server, and additional GDB commands in order to start a debugging session.

Enabling a GDB connection to QEMU

QEMU contains a GDB server that you can connect to, allowing you to debug your QEMU application.

To enable connection to the GDB server, you need to pass in a parameter to QEMU that specify the hostname and port it should listen on.

QEMU parameter	Info
`-gdb tcp:<hostname>:<port>`	Makes QEMU's GDB server listen on host hostname on port port. Generally the hostname is "localhost" and the port can be anything, as long as you can connect to it.
`-gdb tcp:<hostname>:<port> -S`	Makes QEMU's GDB server listen on host hostname on port port and makes emulation start in a paused state. This can allow you to debug the boot sequence of your virtual machine. To un-pause emulation, connect to QEMU using GDB and use the continue command.
`-s`	Shorthand for `-gdb tcp::1234`.

-gdb tcp:localhost:9000

If booting QEMU using PetaLinux, the primary machine will typically listen on localhost:9000.
For example, if booting a ZCU102 machine using PetaLinux, the ARM machine will listen on localhost:9000, while the Microblaze machine will not have remote debugging enabled.

To simultaneously debug both MicroBlaze and ARM machines in a multi-arch environment with PetaLinux, use the --pmu-qemu-args='-gdb tcp:<hostname>:<port>' argument to enable debugging on the MicroBlaze QEMU machine.

Connecting GDB to QEMU

To connect GDB to QEMU, you need to use the GDB that corresponds to your target's architecture.

For example, some of the architectures in the Xilinx QEMU environments are:

GDB	Architecture
aarch64-none-elf-gdb aarch64-linux-gnu-gdb	ARM A53, ARM A72
arm-none-eabi-gdb	ARM R5 `arm-none-eabi-gdb` may give a 'g' packet error when connecting to QEMU, depending if it is built to handle the `aarch64` architecture or not. This is because QEMU will give register information for the first CPU cluster, which are the ARM A72 or ARM A53 CPUs. If this happens, you can debug the ARM R5s with `gdb-multiarch`.
gdb-multiarch	Multiple ARM architectures, and others (excluding Microblaze)
mb-gdb	Microblaze

In this guide, we will use gdb-multiarch to debug the ARM CPUs.

To connect to QEMU's GDB server using your host GDB, you need to create a remote connection.

Once you are connected, you can debug your emulated environment like you would debug any other program.

GDB command	Info
target remote <hostname>:<port> target remote :<port>	Attempts to connect to host hostname on port port. If no hostname is specified, GDB will use localhost.
target extended-remote <hostname>:<port> target extended-remote :<port>	Attempts to connect to host hostname on port port, and will remain connected after the debugged program exits or GDB detaches from it. If no hostname is specified, GDB will use localhost.

$ aarch64-none-elf-gdb hello_a53.elf
GNU gdb (Linaro GDB 2018.04) 8.0.50.20171128-git
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "--host=x86_64-unknown-linux-gnu --target=aarch64-none-elf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /proj/xhdsswstaff/saipava/epdsw1/work/hello_a53.elf...done.
(gdb)
(gdb) target remote:9000
Remote debugging using :9000
_vector_table () at asm_vectors.S:208
208 b _boot
(gdb) b main
Breakpoint 1 at 0x1740: file ../src/main.c, line 75.
(gdb) c
Continuing.

Thread 1 "" hit Breakpoint 1, main () at ../src/main.c:75
75 BootStatus = XPm_GetBootStatus();
(gdb)

Non-Kernel-Intrusive Debugging

Rather than running GDB on the host machine and using the GDB server provided by QEMU, it's possible to run GDB on the guest machine as long as the guest supports it.

This can provide a series of advantages, such as:

Only debugging your program (assuming you're not debugging your kernel)
Being able to catch more signals, such as SIGSEGV
Not losing control of GDB to other signals, such as SIGTRAP

The disadvantages are that it is more work to install GDB on the guest machine, assuming it can be installed at all, and it uses storage space on the guest.

Installing GDB on the Guest

For this example, we will run GDB on a Linux guest, on the 64-bit ARM-A CPUs on a Zynq UltraScale+ MPSoC QEMU machine.
The machine's image will be a default PetaLinux ZCU102 image, and we will install GDB by copying the files from the host using SCP.

Download the ARM64 GDB package from here and save it somewhere.
Unpack the package. This should produce the folders DEBIAN, etc, and usr.
```
$ sudo dpkg-deb -R gdb_7.12-6_arm64.deb . 
```
Compress the etc and usr folders. Optionally delete the DEBIAN, etc, and usr folders when you are done.
On a default PetaLinux image, zip and tar files are supported.
```
$ zip -r gdb.zip etc usr
```

On the guest, copy the compressed file via SCP and extract it.

root@xilinx-zcu102-2020_2:~# scp <host user>@<host IP>:/path/to/gdb.zip .
root@xilinx-zcu102-2020_2:~# unzip gdb.zip

On the guest, copy the extracted contents to root.
```
root@xilinx-zcu102-2020_2:~# cp -r etc usr /
```

Running GDB on the Guest

GDB can be run from the guest as if you would normally run it from the host.

If your binary and source files are not on the guest machine, copy them from the host before running GDB.

root@xilinx-zcu102-2020_2:~# scp <host user>@<host IP>:/scratch/proj/gdb-test/segfault.c .
<host user>@<host IP>'s password: 
segfault.c                                                  100%  171     0.2KB/s   00:00    
root@xilinx-zcu102-2020_2:~# scp <host user>@<host IP>:/scratch/proj/gdb-test/segfault.elf .
<host user>@<host IP>'s password: 
segfault.elf                                                100%   11KB  10.6KB/s   00:00    
root@xilinx-zcu102-2020_2:~# gdb
gdb: /lib/libncurses.so.5: no version information available (required by gdb)
gdb: /lib/libncurses.so.5: no version information available (required by gdb)
gdb: /lib/libncurses.so.5: no version information available (required by gdb)
gdb: /lib/libtinfo.so.5: no version information available (required by gdb)
GNU gdb (Debian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "aarch64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".
(gdb) file segfault.elf
Reading symbols from segfault.elf...done.
(gdb) r
Starting program: /home/root/segfault.elf 

Program received signal SIGSEGV, Segmentation fault.
0x00000000004005d8 in main () at segfault.c:8
8	    *p_null = 0xAA55AA55;
(gdb)

GDB Commands

This section will cover GDB commands used regardless if GDB is used remotely or locally.

This section covers a small subset of what is available. A full list can be found here.

Loading Symbols

GDB requires symbols from the program being executed, otherwise it won't know anything about the program and won't be able to debug.

You can still debug your application without symbols loaded, however it will be much more difficult.

Command Info

--args <file.elf>

Loads the symbols in file.elf into GDB
If file.elf does not contain debugging symbols, it must be recompiled with the -g flag passed into gcc.

This option is used when starting GDB, for example: gdb --args file.elf

symbol-file <file.elf>

Loads the symbols in file.elf into GDB.
If file.elf does not contain debugging symbols, it must be recompiled with the -g flag passed into gcc.

symbol-file can only store symbols from one file at a time.

add-symbol-file <file.elf> <addr>

Loads the symbols in file.elf into GDB.
If file.elf does not contain debugging symbols, it must be recompiled with the -g flag passed into gcc.

addr is the start address of the .text section in file.elf. To find the address of .text you can do:

readelf -WS file.elf | grep .text

(gdb) symbol-file test.elf
Reading symbols from test.elf...done.

Controlling Execution

Command	Info
r run	Starts execution
CTRL+C	Pauses execution.
c continue	Continues execution.
kill	Kills the current process.
q quit	Quits GDB.

komlodi@machine:/scratch/gdb-test$ gdb test.elf
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from test.elf...done.
(gdb) r
Starting program: /scratch/gdb-test/test.elf
^C
Program received signal SIGINT, Interrupt.
main () at test.c:23
23	    while(1);
(gdb) c
Continuing.
^C
Program received signal SIGINT, Interrupt.
main () at test.c:23
23	    while(1);
(gdb) kill
Kill the program being debugged? (y or n) y
(gdb) q
komlodi@machine:/scratch/gdb-test$

Breakpoints and Watchpoints

Breakpoints allow you to pause execution at a specific point in your code. This allows you to observe program state at a specific point in time.

Watchpoints allow you to pause execution when the value of an expression changes. This can include a variable or address being modified, or a more complicated expression such as "var < 10".

Command	Info
info break	Gives you information on all breakpoints in your GDB session.
info watch	Gives you information on all watchpoints in your GDB session.
b <file:line-number\|function-name> [if <cond>]	Adds a breakpoint at line line-number in file or at the start of function function-name. If cond exists, the breakpoint will only trigger when cond is met. The breakpoint number generated here is used when disabling or deleting a breakpoint.
watch <var> watch <addr> watch <expr>	Adds a watchpoint that pauses program when variable var or address addr are modified. Watchpoints can have more complicated expressions with C-like syntax. Watchpoints will trigger after the instruction is executed, meaning GDB will pause on the line after the watchpoint triggers. In a non-local environment, GDB does not support reverse-step to see the exact line where the trigger occurred. When using a watchpoint with an expression, GDB will evaluate the expression and pause when it is non-zero.
disable <breakpoint-number>	Disables breakpoint breakpoint-number
enable <breakpoint-number>	Enables breakpoint breakpoint-number
d <breakpoint-number> delete <breakpoint-number>	Deletes breakpoint breakpoint-number

Reading symbols from test.elf...done.
(gdb) b main
Breakpoint 1 at 0x4005de: file test.c, line 20.
(gdb) watch global_var & 0x00000003
Hardware watchpoint 2: global_var & 0x00000003
(gdb) r
Starting program: /scratch/gdb-test/test.elf 

Breakpoint 1, main () at test.c:20
20	{
(gdb) c
Continuing.

Hardware watchpoint 2: global_var & 0x00000003

Old value = 0
New value = 3
foo (s=0x7fffffffe1c0) at test.c:38
38	    if (s->str == NULL) {
(gdb) p/x global_var
$1 = 0xcc33cc33
(gdb) info break
Num     Type           Disp Enb Address            What
1       breakpoint     keep y   0x00000000004005de in main at test.c:20
	breakpoint already hit 1 time
2       hw watchpoint  keep y                      global_var & 0x00000003
	breakpoint already hit 1 time
(gdb) info watch
Num     Type           Disp Enb Address            What
2       hw watchpoint  keep y                      global_var & 0x00000003
	breakpoint already hit 1 time
(gdb) d 1
(gdb) d 2
(gdb) info break
No breakpoints or watchpoints.
(gdb)

Stepping through your program

When paused, GDB allows you to control the program flow in a variety of ways.

Command	Info
s	Steps through one line of your program. This will go inside functions, including library functions.
n	Steps through one line of your program. This will not go inside functions.
si	Steps through one instruction of your program. This will go inside functions, including library functions.
ni	Steps through one instruction of your program. This will not go inside functions.
finish	Executes until the end of the current function.

Reading symbols from test.elf...done.
(gdb) b main
Breakpoint 1 at 0x4005de: file test.c, line 20.
(gdb) r
Starting program: /scratch/gdb-test/test.elf 

Breakpoint 1, main () at test.c:20
20	{
(gdb) n
23	    foo(&s);
(gdb) s
foo (s=0x7fffffffe1c0) at test.c:30
30	    s->str = "This is a string";
(gdb) n
31	    memset(s->arr, 0xAA, sizeof(s->arr));
(gdb) n
32	    s->var = 1234;
(gdb) c
Continuing.

Stack and frame information

Command	Info
backtrace	Prints the call stack, in the order that functions were called, as a list of frames
frame <frame-number>	Changes the current frame being observed to frame-number. This does not modify program execution.
info frame	Gives you information on the current frame.
info variables	Gives you information of all static/global variables and symbols in your program. On some systems, this may print a lot of information.
info locals	Gives you information on local variables in the current frame.
info args	Gives you information on arguments passed into the current frame.

Breakpoint 1, foo (s=0x7fffffffe1c0) at test.c:30
30	    s->str = "This is a string";
(gdb) backtrace
#0  foo (s=0x7fffffffe1c0) at test.c:30
#1  0x00000000004005f9 in main () at test.c:23
(gdb) frame 0
#0  foo (s=0x7fffffffe1c0) at test.c:30
30	    s->str = "This is a string";
(gdb) info frame
Stack level 0, frame at 0x7fffffffe1c0:
 rip = 0x400620 in foo (test.c:30); saved rip = 0x4005f9
 called by frame at 0x7fffffffe200
 source language c.
 Arglist at 0x7fffffffe1b0, args: s=0x7fffffffe1c0
 Locals at 0x7fffffffe1b0, Previous frame's sp is 0x7fffffffe1c0
 Saved registers:
  rbp at 0x7fffffffe1b0, rip at 0x7fffffffe1b8
(gdb) info locals
No locals.
(gdb) info args
s = 0x7fffffffe1c0
(gdb) frame 1
#1  0x00000000004005f9 in main () at test.c:23
23	    foo(&s);
(gdb) info locals
s = {var = 0, arr = "\000\000\000\000\000\000\000", 
  str = 0x4006b0 <__libc_csu_init> "AWAVA\211\377AUATL\215%N\a ", c = -32 '\340', num = 0}
(gdb) info args
No arguments.
(gdb)

Printing Variables

gdb allows you to print out variable information using expressions. The expressions use C-like syntax.

Command	Info
print <var> p <var> p <file>::<var> p '<file>'::<file>	Prints var. By default, GDB will print var based off of what type it is.
p *<arr>@<len>	Prints the first len values of arr.
p/x <var>	Prints var as a hexadecimal number
p/d <var>	Prints var as a signed integer.
p/t <var>	Prints var as a binary number.
p/c <var>	Prints var as a character.
ptype <var> ptype <type>	Prints type definition of var, or the type definition of type. This is useful when getting information of a struct.

Reading symbols from test.elf...done.
(gdb) b foo
Breakpoint 1 at 0x400620: file test.c, line 30.
(gdb) r
Starting program: /scratch/gdb-test/test.elf 

Breakpoint 1, foo (s=0x7fffffffe1c0) at test.c:30
(gdb) ptype s
type = volatile struct {
    uint32_t var;
    uint8_t arr[8];
    char *str;
    char c;
} *
30	    s->str = "This is a string";
(gdb) n
31	    memset(s->arr, 0xAA, sizeof(s->arr));
(gdb) n
32	    s->c = 'z';
(gdb) n
33	    s->var = 1;
(gdb) n
34	    global_var = 0xCC33CC33;
(gdb) p s->str
$1 = 0x400734 "This is a string"
(gdb) p *s->str@4
$2 = "This"
(gdb) p/c s->str[0]
$3 = 84 'T'
(gdb) p/x s->arr
$4 = {0xaa, 0xaa, 0xaa, 0xaa, 0xaa, 0xaa, 0xaa, 0xaa}
(gdb) p/t s->arr
$5 = {10101010, 10101010, 10101010, 10101010, 10101010, 10101010, 10101010, 10101010}
(gdb) p/d s->var
$7 = 1

Modifying Variables

Command	Info
set <var>=<val>	Sets the variable var to the value val.

foo (s=0x7fffffffe1d0) at test.c:38
38	    if (s->str == NULL) {
(gdb) p s->str
$1 = 0x400734 "This is a string"
(gdb) set s->str=0x00
(gdb) n
39	        puts("s contains a NULL string");

Macros

GDB provides commands to expand, list, and define macros.
In this case, 'macro' refers to any preprocessor definition, not just one that takes arguments. For example, each of the following lines would be considered a macro:

#define REG_FIELD_0_MASK    0x01
#define REG_FIELD_1_MASK    (1 << 1)
#define REG_FIELD_X_MASK(x) (1 << x)

Most compilers don't create macro information by default. For example, on gcc, you would create macro information by passing in the -g3 flag at compile time.

Command	Info
macro expand <expr> macro exp <expr>	Expands, but does not parse, all preprocessor macros in expr.
info macro [-a\|-all] <macro>	Shows the current definition of macro macro, and where it was declared. If the `-a` flag is passed, all definitions of macro will be shown.
info macros <function>	Shows all macros that are currently defined in function. This will also print all macro definitions defined in function, including ones from standard libraries.

Breakpoint 1, main () at gdb-macro.c:13
13	    for (i=0; i<sizeof(reg) * 8; ++i) {
(gdb) n
14	        printf("bit %.2d: %d\n", i, REG_BIT_EXTRACT(reg, i));
(gdb) info macro REG_BIT_EXTRACT
Defined at /scratch/test-code/gdb-macro.c:5
#define REG_BIT_EXTRACT(reg, bit) (!!GET_BIT(reg, bit))
(gdb) info macro GET_BIT
Defined at /scratch/test-code/gdb-macro.c:4
#define GET_BIT(val, bit) (val & (1 << bit))
(gdb) macro exp REG_BIT_EXTRACT(reg, i)
expands to: (!!(reg & (1 << i)))
(gdb) macro exp REG_BIT_EXTRACT(0xAA55AA55, 0)
expands to: (!!(0xAA55AA55 & (1 << 0)))
(gdb) p REG_BIT_EXTRACT(0xAA55AA55, 0)
$1 = 1

Signals

When debugging your code it may be useful to manage signals, particularly if they keep interrupting your GDB session.

QEMU GDB server is only capable of handling SIGINT and SIGTRAP signals. Meaning if your program receives a SIGSEGV signal, it will not be passed to GDB.

Command

Info

info signals [sig]

Prints information on all symbols, or only signal sig if specified

catch signal [sig1 sig2 | all ]

Sets a catchpoint at signals sig1 and sig2, or all signals if all is specified.

handle sig [nostop stop print noprint noignore ignore]

Handles signal sig depending on the following keywords:

nostop
GDB should not stop your program when this signal happens
stop
GDB should stop your program when this signal happens
print
GDB should print when this signal happens
noprint
GDB should not print when this signal happens
noignore
GDB should allow your program to see this signal
ignore
GDB should not allow your program to see this signal

(gdb) symbol-file myapp
Reading symbols from myapp...done.
(gdb) b main
Breakpoint 6 at 0x4008a4: file myapp.c, line 85.
(gdb) c
Continuing.
[Switching to Thread 1.1]

Thread 1 hit Breakpoint 6, main () at myapp.c:85
85 puts("Program start");
(gdb) n

Thread 1 received signal SIGTRAP, Trace/breakpoint trap.
0xffffff800810f814 in ?? ()
(gdb)
Cannot find bounds of current function
(gdb)
Cannot find bounds of current function
(gdb) c
Continuing.
^C
Thread 1 received signal SIGINT, Interrupt.
0xffffff800809c088 in ?? ()
(gdb) info signals SIGTRAP
Signal Stop Print Pass to program Description
SIGTRAP Yes Yes Yes Trace/breakpoint trap
(gdb) handle SIGTRAP nostop print
SIGTRAP is used by the debugger.
Are you sure you want to change it? (y or n) y
Signal Stop Print Pass to program Description
SIGTRAP No Yes Yes Trace/breakpoint trap
(gdb) c
Continuing.
[Switching to Thread 1.2]

Thread 2 hit Breakpoint 6, main () at myapp.c:85
85 puts("Program start");
(gdb) n

Thread 2 received signal SIGTRAP, Trace/breakpoint trap.
# Execution resumes

Threads

Command	Info
info threads	Gives you information on all current threads in the program. The current thread of execution will have an asterisk (*) by it.
thread <thread-ID>	Changes the current thread to thread-ID, and prints out the current frame in thread-ID
set print thread-events <on\|off>	Prints when a thread is created or deleted.
show print thread-events	Shows if thread events are printed or not
thread apply [thread-ids \| all] <cmd>	Applies cmd to threads thread-ids or to all threads if all is specified.

(gdb) info threads
  Id   Target Id         Frame 
* 1    Thread 1.1 (Cortex-A72 #0 [running]) 0xffffff800809c088 in ?? ()
  2    Thread 1.2 (Cortex-A72 #1 [running]) 0xffffff800809c088 in ?? ()
(gdb) thread 2
[Switching to thread 2 (Thread 1.2)]
#0  0xffffff800809c088 in ?? ()
(gdb) thread 1
[Switching to thread 1 (Thread 1.1)]
#0  0xffffff800809c088 in ?? ()
(gdb) set print thread-events
(gdb) show print thread-events
Printing of thread events is on.
(gdb) thread apply all catch signal all

Thread 2 (Thread 1.2):
Catchpoint 1 (any signal)

Thread 1 (Thread 1.1):
Catchpoint 2 (any signal)
(gdb)

Debugging Multiple Processes

In some situations you may want to debug multiple processes simultaneously, or a process that already exists but GDB does not have immediate knowledge of.
On platforms with multiple ARM architectures, these same commands can be used to switch between the different ARM processors.

Command	Info
attach <id>	Attaches to a running process with ID id. When remote debugging, you must target gdbserver using the `extended-remote` command in order to attach to a new process.
add-inferior	Adds an executable to the current debug session
inferior <inf>	Makes inferior inf the current inferior to be debugged by GDB
info inferior	Prints a list of all inferiors currently managed by GDB.
set print inferior-events <on\|off>	Enables or disables printing messages when GDB notices new inferiors have started or stopped. By default inferior events are not printed.
show print inferior-events	Show if inferior event messages are printed or not.

(gdb) target extended-remote :9000
Remote debugging using :9000
0xffffff8008112eb4 in ?? ()
(gdb) info thread
  Id   Target Id         Frame 
* 1    Thread 1.1 (Cortex-A53 #0 [running]) 0xffffff8008112eb4 in ?? ()
  2    Thread 1.2 (Cortex-A53 #1 [running]) 0xffffff8008112eb4 in ?? ()
  3    Thread 1.3 (Cortex-A53 #2 [running]) 0xffffff8008112eb4 in ?? ()
  4    Thread 1.4 (Cortex-A53 #3 [running]) 0xffffff8008113440 in ?? ()
(gdb) add-inferior
Added inferior 2
(gdb) inferior 2
[Switching to inferior 2 [<null>] (<noexec>)]
(gdb) attach 2
Attaching to process 2
[New Thread 2.6]
0xffff0000 in ?? ()
(gdb) info thread
  Id   Target Id         Frame 
  1.1  Thread 1.1 (Cortex-A53 #0 [running]) 0xffffff8008112eb4 in ?? ()
  1.2  Thread 1.2 (Cortex-A53 #1 [running]) 0xffffff8008112eb4 in ?? ()
  1.3  Thread 1.3 (Cortex-A53 #2 [running]) 0xffffff8008112eb4 in ?? ()
  1.4  Thread 1.4 (Cortex-A53 #3 [running]) 0xffffff8008113440 in ?? ()
* 2.1  Thread 2.5 (Cortex-R5 #0 [halted ]) 0xffff0000 in ?? ()
  2.2  Thread 2.6 (Cortex-R5 #1 [halted ]) 0xffff0000 in ?? ()
(gdb) info inferior
  Num  Description       Executable        
  1    process 1                           
* 2    process 2                           
(gdb) inferior 1
[Switching to inferior 1 [process 1] (<noexec>)]
[Switching to thread 1.1 (Thread 1.1)] 
#0  0xffffff8008112eb4 in ?? ()
(gdb) set print inferior-events on
(gdb) show print inferior-events
Printing of inferior events is on.
(gdb) c
Continuing.

To simplify this process you can add a function to GDB. To do this edit your .gdbinit file (located in your home directory) and add the following lines:

define rdo_arm
target extended-remote :9000
add-inferior
inferior 2
attach 2
info threads
end

This will allow you to use the command rdo_arm to connect to QEMU's GDB server and automatically discover the RPU.

Lower Level Examining

Command	Info
info registers	Prints out your CPU's registers.
x/x <addr>	Prints out the value at addr interpreted as hex.
x/4x <addr>	Prints out the first 4 values at addr interpreted as hex.
x/s <addr>	Prints out the value at addr interpreted as a string.
disassemble <addr> disassemble <function-name>	Prints out the disassembly at address addr or at function function-name.

Reading symbols from test.elf...done.
(gdb) b foo
Breakpoint 1 at 0x400620: file test.c, line 32.
(gdb) r
Starting program: /scratch/gdb-test/test.elf 

Breakpoint 1, foo (s=0x7fffffffe1c0) at test.c:32
32	    s->str = "This is a string";
(gdb) n
33	    memset(s->arr, 0xAA, sizeof(s->arr));
(gdb) n
34	    s->var = 1234;
(gdb) n
36	    global_var = 0xCC33CC33;
(gdb) n
38	    if (s->str == NULL) {
(gdb) x/x &global_var
0x601054 <global_var>:	0xcc33cc33
(gdb) x/4x &global_var
0x601054 <global_var>:	0xcc33cc33	0x00000000	0x00000000	0x00000000
(gdb) x/s s->str
0x400724:	"This is a string"
(gdb) info registers
rax            0x7fffffffe1c0	140737488347584
rbx            0x0	0
rcx            0x0	0
rdx            0x8	8
rsi            0x7fffffffe1cc	140737488347596
rdi            0x7fffffffe1c4	140737488347588
rbp            0x7fffffffe1b0	0x7fffffffe1b0
rsp            0x7fffffffe1a0	0x7fffffffe1a0
r8             0x400710	4196112
r9             0x7ffff7de7ab0	140737351940784
r10            0x34e	846
r11            0x7ffff7b7f970	140737349417328
r12            0x4004e0	4195552
r13            0x7fffffffe2d0	140737488347856
r14            0x0	0
r15            0x0	0
rip            0x40065a	0x40065a <foo+70>
eflags         0x246	[ PF ZF IF ]
cs             0x33	51
ss             0x2b	43
ds             0x0	0
es             0x0	0
fs             0x0	0
gs             0x0	0
k0             0x0	0
k1             0x0	0
k2             0x0	0
k3             0x0	0
k4             0x0	0
k5             0x0	0
k6             0x0	0
k7             0x0	0
(gdb) n
42	    disassemble_me();
(gdb) disassemble disassemble_me
Dump of assembler code for function disassemble_me:
   0x0000000000400679 <+0>:	push   %rbp
   0x000000000040067a <+1>:	mov    %rsp,%rbp
   0x000000000040067d <+4>:	movq   $0x0,-0x8(%rbp)
   0x0000000000400685 <+12>:	mov    -0x8(%rbp),%rax
   0x0000000000400689 <+16>:	movzbl (%rax),%eax
   0x000000000040068c <+19>:	mov    %al,-0x9(%rbp)
   0x000000000040068f <+22>:	nop
   0x0000000000400690 <+23>:	pop    %rbp
   0x0000000000400691 <+24>:	retq   
End of assembler dump.

Debugging Examples

Zynq UltraScale+ MPSoC and Versal Adaptive SoC PS + PMU simultaneous debugging

To debug The PS and PMU simultaneously you need to boot up the Microblaze and ARM64 QEMU instances with the -gdb flag.
Once the virtual machines are started, you can remotely target them with two separate instances of GDB.

Make sure the ports you pass in to each QEMU instance are different from each other.

Most QEMU arguments are omitted for brevity. A list of example arguments used when booting a Zynq UltraScale+ MPSoC or Versal Adaptive SoC machine can be found here.

# Most QEMU arguments omitted for brevity
qemu-system-microblazeel \
-M microblaze-fdt \
# 8<-------------------------------------
-gdb tcp:127.0.0.1:9090 -S

qemu-system-aarch64 \
-M arm-generic-fdt \
# 8<-------------------------------------
-gdb tcp:127.0.0.1:9000

Connect mb-gdb to Microblaze QEMU

$ mb-gdb
GNU gdb (crosstool-NG 1.20.0) 8.2.1.20190121-git
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "--host=x86_64-build_unknown-linux-gnu --target=microblaze-xilinx-elf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word".
(gdb) target remote :9090
Remote debugging using :9090
warning: No executable has been specified and target does not support
determining executable automatically.  Try using the "file" command.
0x0000d0ff in ?? ()
(gdb) c
Continuing.

Connect gdb-multiarch to AArch64 QEMU and discover the ARM-R processors.

 gdb-multiarch
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".
(gdb) target extended-remote :9000
Remote debugging using :9000
0xffffff8008112eb4 in ?? ()
(gdb) info thread
  Id   Target Id         Frame 
* 1    Thread 1.1 (Cortex-A53 #0 [running]) 0xffffff8008112eb4 in ?? ()
  2    Thread 1.2 (Cortex-A53 #1 [running]) 0xffffff8008112eb4 in ?? ()
  3    Thread 1.3 (Cortex-A53 #2 [running]) 0xffffff8008112eb4 in ?? ()
  4    Thread 1.4 (Cortex-A53 #3 [running]) 0xffffff8008113440 in ?? ()
(gdb) add-inferior
Added inferior 2
(gdb) inferior 2
[Switching to inferior 2 [<null>] (<noexec>)]
(gdb) attach 2
Attaching to process 2
[New Thread 2.6]
0xffff0000 in ?? ()
(gdb) info thread
  Id   Target Id         Frame 
  1.1  Thread 1.1 (Cortex-A53 #0 [running]) 0xffffff8008112eb4 in ?? ()
  1.2  Thread 1.2 (Cortex-A53 #1 [running]) 0xffffff8008112eb4 in ?? ()
  1.3  Thread 1.3 (Cortex-A53 #2 [running]) 0xffffff8008112eb4 in ?? ()
  1.4  Thread 1.4 (Cortex-A53 #3 [running]) 0xffffff8008113440 in ?? ()
* 2.1  Thread 2.5 (Cortex-R5 #0 [halted ]) 0xffff0000 in ?? ()
  2.2  Thread 2.6 (Cortex-R5 #1 [halted ]) 0xffff0000 in ?? ()
(gdb) info inferior
  Num  Description       Executable        
  1    process 1                           
* 2    process 2

Now you are able to debug the Microblaze, ARM-A, and ARM-R CPUs simultaneously.

Xilinx Wiki

Debugging Guest Applications with QEMU and GDB