/
Linux Debug infrastructure (KProbe/UProbe/LTTng)

Linux Debug infrastructure (KProbe/UProbe/LTTng)

Apr 18, 2019

Pre-requisites and modify Linux image

Pre-requisites

Modifying Linux-kernel

For any tracer to work, there are some dependent kernel modules which needs to be enabled in kernel. Following are the modules required by LTTng and Kprobes and steps to enable them (a module can be enabled by selecting that module and pressing ‘y’)

 

Including packages in root-fs (Required only for LTTng)

  1.  

    1.  

      1. Include packages in Petalinux build. Include IMAGE_INSTALL_append = " lttng-tools lttng-modules lttng-ust" in the file project-spec/meta-user/recipes-core/images/petalinux-image.bbappend

      2. Enable packages in rootfs. Use Petalinux config, this will open rootfs menuconfig.

        $ petalinux-config -c rootfs

      3. Enable “lttng-tools”, “lttng-modules”. Goto user packages and select

        lttng-modules, lltng-tools

      4. Enable “lttng-ust”. Goto Filesystem Packages -> misc and select lttng-ust

      5. Enable “babeltrace”. Goto Filesystem Packages -> misc and select babeltrace

      6. Building project for linux-image. 
        $ petalinux-build -x build

Kprobes

Linux image for Kprobes

Follow the steps mentioned in Pre-requisites and Modifying Linux-kernel found under "Pre-requisites and modify Linux image"

Sample kprobe module

kprobes provide "pre" handlers that run before the specific instruction, and "post" handlers that run afterwards. Generally, these are useful in resolving kernel-panics. The following steps will illustrate how to assign the handlers and build the module.

debugging with kprobes

Uprobes

Uprobes provides a way to insert a probe point in user-code at runtime, i.e there is no need to recompilation of source code every time the user wants to add/change a probe-point. A probe point is a debug statement that helps explore execution (execution flow), like printf.

Inserting uprobe probe-point

Viewing trace output

traces will be logged into a file, open it to view the logs.

# cat /sys/kernel/debug/tracing/trace

Sample output:

         tracer: nop

                              _-----=> irqs-off

                             / _----=> need-resched

                            | / _---=> hardirq/softirq

                            || / _--=> preempt-depth

                            ||| /     delay

            TASK-PID   CPU#  ||||   TIMESTAMP  FUNCTION

              |         |          |                 |                        |

           myapp-2432  [001] ....   442.382453: func_1_entry: (0x400634)

           myapp-2432  [001] ....   443.382597: func_1_entry: (0x400634)

           myapp-2432  [001] ....   444.382714: func_1_entry: (0x400634)

           myapp-2432  [001] ....   445.382832: func_1_entry: (0x400634)

           myapp-2432  [001] ....   446.382950: func_1_entry: (0x400634)

           myapp-2432  [001] ....   447.383018: func_1_entry: (0x400634)

           myapp-2432  [001] ....   448.383136: func_1_entry: (0x400634)

           myapp-2432  [001] ....   449.383255: func_1_entry: (0x400634)

           myapp-2432  [001] ....   450.383372: func_1_entry: (0x400634)

           myapp-2432  [001] ....   451.383490: func_1_entry: (0x400634)

           myapp-2432  [001] ....   452.383609: func_1_entry: (0x400634)

           myapp-2432  [001] ....   453.383727: func_2_entry: (0x40066c)

           myapp-2432  [001] ....   454.383951: func_2_entry: (0x40066c)

           myapp-2432  [001] ....   455.384031: func_2_entry: (0x40066c)

           myapp-2432  [001] ....   456.384160: func_2_entry: (0x40066c)

LTTng

Linux image for LTTng

Follow the steps mentioned in Pre-requisites, Modifying Linux-kernel and Including packages in root-fs  found under "Pre-requisites and modify Linux image"

Kernel-space tracing using LTTng

Viewing kernel-trace-logs using Babeltrace

To view the trace, a third-party tool Babeltrace is required. Since LTTng generates the traces as CTF(Common Trace Format) which is a binary format, it is required to convert them using a tool which translates it(CTF) to normal text log. Generate trace-log using Babeltrace providing the trace directory input (same directory path given as input while creating LTTNG SESSION)

# babeltrace /tmp/kernel > kernelTrace.txt

 

User-space tracing using LTTng (standard libc events)

Tracing with standard libc gives the advantage of run-time decision of event to be traced, i.e. there is no need of recompiling the source-code.

User-space tracing using LTTng (user tracepoints)

Tracing with user-tracepoints gives the advantage of having tracepoint wherever user wants, but this has limitation of recompiling everytime there is a change of tracepoint location. The steps will be same but the source code must be changed to accommodate tracepoints. Details can be found here