Linux Debug infrastructure (KProbe/UProbe/LTTng)

Pre-requisites and modify Linux image

Pre-requisites

1. 1. Petalinux environment (set the required environment set for petalinux)
  2. Petalinux (source from Petalinux daily)
  3. Petalinux project (create). petalinux-create -t project -s <hardware_specific.bsp>
    $ petalinux-create -t project -s /proj/petalinux/petalinux-v2018.1_bsps_daily_latest/xilinx-zcu102-v2018.1-final.bsp
  4. Petalinux project (configure basic settings). petalinux-config --get-hw-description=<hardware_specific>
    $ petalinux-config --get-hw-description=hardware/xilinx-zcu102-2018.1/xilinx-zcu102-2018.1.sdk/.

Modifying Linux-kernel

For any tracer to work, there are some dependent kernel modules which needs to be enabled in kernel. Following are the modules required by LTTng and Kprobes and steps to enable them (a module can be enabled by selecting that module and pressing ‘y’)

1. 1. To configure kernel use Petalinux-config. This will open kernel menu-config.
    $ petalinux-config -c kernel
  2. Enable “Kprobes”. Goto General setup and select Kprobes
  3. Enable “ksymoops”. Goto General setup -> Configure standard kernel features (expert users) and select Load all symbols for debugging/ksymoops.
  4. Enable “kallsyms”. Goto General setup -> Configure standard kernel features (expert users) and select Include all symbols in kallsyms.
  5. Enable “Kernel Function tracer”, “Interrupts-off Latency tracer”, “Premption-off Latency tracer”, “Trace syscalls”. Goto Kernel hacking -> Tracers and select

- - Kernel Function tracer
  - Interrupts-off Latency tracer
  - Scheduling Latency tracer
  - Trace syscalls

Including packages in root-fs (Required only for LTTng)

Include packages in Petalinux build. Include IMAGE_INSTALL_append = " lttng-tools lttng-modules lttng-ust" in the file project-spec/meta-user/recipes-core/images/petalinux-image.bbappend
Enable packages in rootfs. Use Petalinux config, this will open rootfs menuconfig.
$ petalinux-config -c rootfs
Enable “lttng-tools”, “lttng-modules”. Goto user packages and select
lttng-modules, lltng-tools
Enable “lttng-ust”. Goto Filesystem Packages -> misc and select lttng-ust
Enable “babeltrace”. Goto Filesystem Packages -> misc and select babeltrace
Building project for linux-image.
$ petalinux-build -x build

Kprobes

Linux image for Kprobes

Follow the steps mentioned in Pre-requisites and Modifying Linux-kernel found under "Pre-requisites and modify Linux image"

Sample kprobe module

kprobes provide "pre" handlers that run before the specific instruction, and "post" handlers that run afterwards. Generally, these are useful in resolving kernel-panics. The following steps will illustrate how to assign the handlers and build the module.

1. 1. create a kernel module. Run the Petalinux create command from project-root.
    $ petalinux-create -t modules --name mymodule –enable
  2. create an instance of kprobe structure and populate it with handlers and probing address. sample source can be found here mymodule.c
    /*kprobe pre_handler: called just before the probed instruction is executed*/
    int handler_pre(struct kprobe *p, struct pt_regs *regs)
    {
            printk("pre_handler: p->addr=0x%p\n", p->addr);
            printk("please add your post-handler here.\n");
            return 0;
    }
    /*kprobe post_handler: called after the probed instruction is executed*/
    void handler_post(struct kprobe *p, struct pt_regs *regs, unsigned long flags)
    {
            printk("post_handler: p->addr=0x%p\n", p->addr);
            printk("please add your post-handler here.\n");
            return;
    }
    static struct kprobe kp;
    kp.pre_handler = handler_pre;
    kp.post_handler = handler_post;
    kp.fault_handler = handler_fault;
    kp.addr = (kprobe_opcode_t*) probe_addr;
  3. build the module. Run Petalinux command to build.
    $ petalinux-builld

debugging with kprobes

1. 1. load the module. While loading the module provide the function which has to be traced, in this case its _do_fork
    # insmod mymodule.ko kmodToProbe="_do_fork"
  2. check if pre-handlers and post-handlers are getting invoked.
    # ls
  3. unload the module.
    # rmmod mymodule

Uprobes

Uprobes provides a way to insert a probe point in user-code at runtime, i.e there is no need to recompilation of source code every time the user wants to add/change a probe-point. A probe point is a debug statement that helps explore execution (execution flow), like printf.

Inserting uprobe probe-point

1. 1. get the address of the instruction to be probed. Do a objdump of executable (if not available, cross-compile the user-app). And Locate the function (point of interest) address. sample code myapp.c
    $ arm-linux-gnueabihf-objdump -d myapp
    0000000000400634 <func_1>:
      400634:       a9bf7bfd        stp     x29, x30, [sp, #-16]!
      400638:       910003fd        mov     x29, sp
    000000000040066c <func_2>:
      40066c:       a9bf7bfd        stp     x29, x30, [sp, #-16]!
      400670:       910003fd        mov     x29, sp
  2. insert the probe-point.
    # echo 'p:func_2_entry myapp:0x634' > /sys/kernel/debug/tracing/uprobe_events
    # echo 'p:func_1_entry myapp:0x66c' >> /sys/kernel/debug/tracing/uprobe_events
  3. enabling tracing.
    # echo 1 > /sys/kernel/debug/tracing/events/uprobes/enable
  4. run application. ./myapp <total_iterations> <function1_iterations>
    # ./myapp 15 11

Viewing trace output

traces will be logged into a file, open it to view the logs.

# cat /sys/kernel/debug/tracing/trace

Sample output:

tracer: nop

_-----=> irqs-off

/ _----=> need-resched

| / _---=> hardirq/softirq

|| / _--=> preempt-depth

||| / delay

TASK-PID CPU# |||| TIMESTAMP FUNCTION

| | | | |

myapp-2432 [001] .... 442.382453: func_1_entry: (0x400634)

myapp-2432 [001] .... 443.382597: func_1_entry: (0x400634)

myapp-2432 [001] .... 444.382714: func_1_entry: (0x400634)

myapp-2432 [001] .... 445.382832: func_1_entry: (0x400634)

myapp-2432 [001] .... 446.382950: func_1_entry: (0x400634)

myapp-2432 [001] .... 447.383018: func_1_entry: (0x400634)

myapp-2432 [001] .... 448.383136: func_1_entry: (0x400634)

myapp-2432 [001] .... 449.383255: func_1_entry: (0x400634)

myapp-2432 [001] .... 450.383372: func_1_entry: (0x400634)

myapp-2432 [001] .... 451.383490: func_1_entry: (0x400634)

myapp-2432 [001] .... 452.383609: func_1_entry: (0x400634)

myapp-2432 [001] .... 453.383727: func_2_entry: (0x40066c)

myapp-2432 [001] .... 454.383951: func_2_entry: (0x40066c)

myapp-2432 [001] .... 455.384031: func_2_entry: (0x40066c)

myapp-2432 [001] .... 456.384160: func_2_entry: (0x40066c)

LTTng

Linux image for LTTng

Follow the steps mentioned in Pre-requisites, Modifying Linux-kernel and Including packages in root-fs found under "Pre-requisites and modify Linux image"

Kernel-space tracing using LTTng

1. 1. create a lttng session. lttng create <session name> --output=<dest log dir. optional>
    # lttng create mySession --output=/tmp/
  2. enable events for a lttng session. lttng enable-event --kernel <list of events to be traced comma seperated>
    # lttng enable-event --kernel sched_switch, sched_process_fork
  3. start lttng session.
    # lttng start
  4. stop and destroy the session. By destroying session, traces won’t be lost.
    # lttng stop
    # lttng destroy

Viewing kernel-trace-logs using Babeltrace

To view the trace, a third-party tool Babeltrace is required. Since LTTng generates the traces as CTF(Common Trace Format) which is a binary format, it is required to convert them using a tool which translates it(CTF) to normal text log. Generate trace-log using Babeltrace providing the trace directory input (same directory path given as input while creating LTTNG SESSION)

# babeltrace /tmp/kernel > kernelTrace.txt

User-space tracing using LTTng (standard libc events)

Tracing with standard libc gives the advantage of run-time decision of event to be traced, i.e. there is no need of recompiling the source-code.

1. 1. create a lttng session. lttng create <session name> --output=<dest log dir. optional>
    # lttng create mySession --output=/tmp/
  2. enable events for a lttng session. lttng enable-event --userspace <list of events to be traced comma seperated>
    # lttng enable-event --userspace lttng_ust_libc:malloc
  3. start lttng session.
    # lttng start
  4. run application with lttng-ust library preloaded.
    # LD_PRELOAD=liblttng-ust-libc-wrapper.so ./myapp
  5. stop and destroy the session. By destroying session, traces won’t be lost.
    # lttng stop
    # lttng destroy

User-space tracing using LTTng (user tracepoints)

Tracing with user-tracepoints gives the advantage of having tracepoint wherever user wants, but this has limitation of recompiling everytime there is a change of tracepoint location. The steps will be same but the source code must be changed to accommodate tracepoints. Details can be found here