Versal Libmetal Demo

This page describes how to set up and run the libmetal demo on the Versal Platform, in order to demonstrate the communication between a user-space application running on A72 and bare-metal application running on R5.

Table of Contents

Introduction

This wiki provides the steps to use Vitis to build libmetal applications targeting the R5 and A72 processors. It also details how to set up a device-tree and build a sysroot with library support to build the Linux libmetal application. The application executables built in Vitis are installed in the PetaLinux project RootFS. The example demo shows the communication between a Linux user-space program on A72 and a bare-metal application running on the R5.

Libmetal provides a common user API across various OS environments for device access, interrupt handling and memory access. Using the standardized API, the same application can be run in different environments, such as Linux userspace, FreeRTOS or baremetal. Please refer to our user guide Access-Devices-with-Libmetal to understand how the Libmetal environment is set up, the API’s are used to access the devices, specify shared memory regions and interprocessor interrupts (IPI) .

PetaLinux

In this section, a Petalinux project will be modified to enable libmetal libraries; define shared memory, IPI and remoteproc nodes; and then export the resulting sysroot to Vitis for application development.

The PetaLinux project for this demo is built using a VCK190 BSP. The BSP can be downloaded here https://www.xilinx.com/support/download/index.html/content/xilinx/en/downloadNav/embedded-design-tools.html. You can also create a project using the template flow and your custom XSA.

  1. Create the PetaLinux project using the VCK190 BSP:

    $ petalinux-create -t project -n vck190-libmetal-demo -s xilinx-vck190-v2022.1-final.bsp
  2. Configure the project:

    $ cd vck190-libmetal-demo/ $ petalinux-config
  3. Modify the system-user.dtsi

    1. Device-tree modifications for version 2022.x:
      Copy the content below into the system-user.dtsi. The reserved memory denotes the section of the memory being reserved in the DDR for running the RPU application. This region will also be reflected in the lscript.ld of the RPU application being built in Vitis. In the 2022.x releases and later, TCM and RPU nodes require the power-domain property as follows:

      <&versal_firmware power-domain-node>

      The power node numbers for Versal Adaptive SoC devices can be found here.

      $ vim project-spec/meta-user/recipes-bsp/device-tree/files/system-user.dtsi /include/ "system-conf.dtsi" / { }; / { reserved-memory { #address-cells = <2>; #size-cells = <2>; ranges; rproc_0_reserved: rproc@3ed00000 { no-map; reg = <0x0 0x3ed00000 0x0 0x40000>; }; }; tcm_0a@ffe00000 { no-map; reg = <0x0 0xffe00000 0x0 0x10000>; phandle = <0x40>; status = "okay"; compatible = "mmio-sram"; power-domain = <&versal_firmware 0x1831800b>; }; tcm_0b@ffe20000 { no-map; reg = <0x0 0xffe20000 0x0 0x10000>; phandle = <0x41>; status = "okay"; compatible = "mmio-sram"; power-domain = <&versal_firmware 0x1831800c>; }; rf5ss@ff9a0000 { compatible = "xlnx,zynqmp-r5-remoteproc"; #address-cells = <0x2>; #size-cells = <0x2>; ranges; xlnx,cluster-mode = <1>; reg = <0x0 0xff9a0000 0x0 0x10000>; r5f_0 { compatible = "xilinx,r5f"; #address-cells = <0x2>; #size-cells = <0x2>; ranges; sram = <0x40 0x41>; memory-region = <&rproc_0_reserved>; power-domain = <&versal_firmware 0x18110005>; }; }; /* Shared memory */ shm0: shm@0 { compatible = "shm_uio"; reg = <0x0 0x3ed80000 0x0 0x1000000>; }; /* IPI device */ ipi_amp: ipi@ff360000 { compatible = "ipi_uio"; reg = <0x0 0xff360000 0x0 0x1000>; interrupt-parent = <&gic>; interrupts = <0 33 4>; }; /* Triple Timer Counter */ ttc0 { compatible = "ttc-uio"; reg = <0x0 0xFF0E0000 0x0 0x1000>; }; };
    2. Device-tree modifications in 2021.2 .

      Add the content below to the system-user.dtsi for the reserved memory node, TCM memories of the RPU and the remoteproc node. Nodes are also added for IPI and shared memory (used as UIO devices).

      $ vim project-spec/meta-user/recipes-bsp/device-tree/files/system-user.dtsi /include/ "system-conf.dtsi" / { }; &amba { zyxclmm_drm { compatible = "xlnx,zocl-versal"; }; }; / { reserved-memory { #address-cells = <2>; #size-cells = <2>; ranges; rproc_0_reserved: rproc@3ed00000 { no-map; reg = <0x0 0x3ed00000 0x0 0x40000>; }; }; tcm_0a@ffe00000 { no-map; reg = <0x0 0xffe00000 0x0 0x10000>; phandle = <0x40>; status = "okay"; compatible = "mmio-sram"; }; tcm_0b@ffe20000 { no-map; reg = <0x0 0xffe20000 0x0 0x10000>; phandle = <0x41>; status = "okay"; compatible = "mmio-sram"; }; rf5ss@ff9a0000 { compatible = "xlnx,zynqmp-r5-remoteproc"; #address-cells = <0x2>; #size-cells = <0x2>; ranges; xlnx,cluster-mode = <1>; reg = <0x0 0xff9a0000 0x0 0x10000>; r5f_0 { compatible = "xilinx,r5f"; #address-cells = <0x2>; #size-cells = <0x2>; ranges; sram = <0x40 0x41>; memory-region = <&rproc_0_reserved>; power-domain = <0x18110005>; }; }; /* Shared memory */ shm0: shm@0 { compatible = "shm_uio"; reg = <0x0 0x3ed80000 0x0 0x1000000>; }; /* IPI device */ ipi_amp: ipi@ff360000 { compatible = "ipi_uio"; reg = <0x0 0xff360000 0x0 0x1000>; interrupt-parent = <&gic>; interrupts = <0 33 4>; }; /* Triple Timer Counter */ ttc0 { compatible = "ttc-uio"; reg = <0x0 0xFF0E0000 0x0 0x1000>; }; };
  4. RootFS modifications:

    $ petalinux-config -c rootfs Filesystem Packages---> misc ---> sysfsutils ---> [*] libsysfs libs ---> libmetal---> [*] libmetal [*] libmetal-demos
  5. Build and package sysroot:

    $ petalinux-build --sdk $ petalinux-package --sysroot
  6. Install the libmetal application in the RootFS. Create a template app to install the Libmetal R5 firmware and Linux applications built in Vitis to the RootFS. Copy the application ELFs built in Vitis to the meta-user/recipes-apps/libmetal-apps/files/ directory.

    $ petalinux-create -t apps --template install -n libmetal-apps --enable $ cp <vitis-proj-libmetal-r5>/Debug/libmetal-r50.elf project-spec/meta-user/recipes-apps/libmetal-apps/files/ $ cp <vitis-proj-libmetal-linux>/Debug/libmetal-linux.elf project-spec/meta-user/recipes-apps/libmetal-apps/files/
  7. Modify the libmetal-apps.bb recipes to include the application elf build in Vitis to the RootFS

    1. For the 2022.x versions and later please note the syntax changes in the Yocto recipe. The [:]colon character replaces the use of [ _ ] underscore when referring to an override.

      # # This file is the libmetal-apps recipe. # SUMMARY = "Simple libmetal-apps application" SECTION = "PETALINUX/apps" LICENSE = "MIT" LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302" SRC_URI = "file://libmetal-linux.elf \ file://libmetal-r50.elf \ " S = "${WORKDIR}" INSANE_SKIP:${PN} = "arch" RDEPENDS:${PN} += " \ libmetal \ open-amp \ " do_install() { install -d ${D}/lib/firmware install -m 0644 ${S}/libmetal-r50.elf ${D}/lib/firmware/libmetal-r50.elf install -d ${D}/${bindir} install -m 0755 ${S}/libmetal-linux.elf ${D}/${bindir}/libmetal-linux.elf } FILES:${PN} = "/lib/firmware/libmetal-r50.elf ${bindir}/libmetal-linux.elf"
    2. For the 2021.x versions and earlier, the recipe is as shown below:

      $ vim project-spec/meta-user/recipes-apps/libmetal-apps/libmetal-apps.bb # This file is the libmetal-apps recipe. # SUMMARY = "Simple libmetal-apps application" SECTION = "PETALINUX/apps" LICENSE = "MIT" LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302" SRC_URI = "file://libmetal-linux.elf \ file://libmetal-r50.elf \ " S = "${WORKDIR}" INSANE_SKIP_${PN} = "arch" RDEPENDS_${PN}_append = " \ libmetal \ open-amp \ " do_install() { install -d ${D}/lib/firmware install -m 0644 ${S}/libmetal-r50.elf ${D}/lib/firmware/libmetal-r50.elf install -d ${D}/${bindir} install -m 0755 ${S}/libmetal-linux.elf ${D}/${bindir}/libmetal-linux.elf } FILES_${PN} = "/lib/firmware/libmetal-r50.elf ${bindir}/libmetal-linux.elf"
  8. Build the PetaLinux project and package the images.

    $ petalinux-build $ petalinux-package --boot --plm --psmfw --u-boot --dtb #QEMU $ petalinux-package --boot --u-boot --qemu-rootfs images/linux/rootfs.cpio.gz.u-boot --force

Vitis

Vitis is used to build the libmetal application examples for the R5 and A72 processors. The executables built in Vitis are then included in the PetaLinux recipes-apps generated above and installed to your RootFS.

Libmetal R5 App

  1. Create a standalone Libmetal demo application targeting the R5-0 processor. Click browse and import your custom XSA, or you can select the vck190.xsa from fixed platforms.

  2. Select the target processor versal_cips_0_pspmc_0_psv_cortexr5_0

  3. Select Libmetal AMP Demo from the available templates.

  4. Build the application. The generated .elf will be in the <vitis-app>/Debug/ directory. This .elf will need to be copied to the PetaLinux meta-user/recipes-apps folder as described in Step 6 of the PetaLinux project setup instructions.

Libmetal Linux App

  1. Create a new application project for the Libmetal demo on Linux. You can use the same VCK190 platform created for your standalone application on R5

  2. Check the box to show all processors in the hardware configuration. Select the target processor psv_cortexa72 SMP

  3. Create a new domain and provide the sysroot path. The sysroot path will be <plnx-proj>/images/linux/sdk/sysroots/cortexa72-cortexa53-xilinx-linux/

  4. Select Linux empty application from the available templates and click finish.

  5. Clone the Libmetal repo. Copy the src files below to your Vitis libmetal-linux application's src folder

    $ git clone https://github.com/Xilinx/libmetal.git $ cd libmetal $ git checkout xlnx_rel_v2022.1 $ cd examples/system/linux/zynqmp/zynqmp_amp_demo/ $ cp common.h ipi_latency_demo.c ipi_shmem_demo.c ipi-uio.c shmem_atomic_demo.c shmem_demo.c shmem_latency_demo.c shmem_throughput_demo.c sys_init.c sys_init.h libmetal_amp_demo.c <vitis-proj-libmetal-linux>/src/
  6. Add the metal library under ARM v8 Linux gcc linker Libraries.  

  7. Add the compiler flag CONFIG_IPI_MASK=0x08 under the ARM v8 Linux gcc compiler symbols 

  8. In common.h, make the following changes 

    #define IPI_DEV_NAME "ff360000.ipi" #define TTC_DEV_NAME "ff0e0000.ttc0"
  9. Build the Libmetal Linux project.
    The generated .elf will be in the <vitis-app>/Debug/ directory. This .elf will need to be copied to the PetaLinux meta-user/recipes-apps folder as described in Step 6 of the PetaLinux project setup instructions.

Testing on Hardware/QEMU

Note: Autologin has been disabled from 2022.x onwards. The default user will be petalinux and the password should be set on first boot. In order to run the libmetal demos, you will need to be the root user. Please login as root before the running the demos.

Known issue: The TTC is put in reset by the PM framework. As a workaround, you will need to request for the TTC node (PM_DEV_TTC_0 (0x18224024U)) using the SysFS interface.
The API is documented here and the device node numbers to request for the node can be found here

$ petalinux-boot --qemu --kernel root@xilinx-vck190-20221:~# cd /lib/firmware/ root@xilinx-vck190-20221:~# echo pm_request_node 0x18224024 0x7 0x64 0 > /sys/kernel/debug/zynqmp-firmware/pm root@xilinx-vck190-20221:/lib/firmware# echo libmetal-r50.elf > /sys/class/remoteproc/remoteproc0/firmware root@xilinx-vck190-20221:/lib/firmware# echo start > /sys/class/remoteproc/remoteproc0/state [ 58.166144] remoteproc remoteproc0: powering up ff9a0000.rf5ss:r5f_0 [ 58.172381] remoteproc remoteproc0: Booting fw image libmetal-r50.elf, size 776192 [ 58.304269] remoteproc remoteproc0: no resource table found. [ 58.531414] remoteproc remoteproc0: remote processor ff9a0000.rf5ss:r5f_0 is now up root@xilinx-vck190-20221:/lib/firmware# registering: 0, name=ff340000.ipi registering: 1, name=3ed80000.shm registering: 2, name=ff0E0000.ttc SERVER> ====== libmetal demo: shared memory ====== SERVER> Wait for shared memory demo to start. root@xilinx-vck190-20221:/lib/firmware# libmetal-linux.elf CLIENT> ****** libmetal demo: shared memory ****** metal: info: metal_linux_dev_open: checking driver vfio-platform,3ed80000.shm,(null) metal: info: metal_uio_dev_open: No IRQ for device 3ed80000.shm. CLIENT> Setting up shared memory demo. CLIENT> Starting shared memory demo. SERVER> Demo has started. CLIENT> Sending message: Hello World - libmetal shared memory demo CLIENT> Message Received: Hello World - libmetal shared memory demo CLIENT SER> SVER> Shharedare memory ded memory test mo: Pfinished assed. SERVER> ====== libmetal demo: atomic operation over shared memory ====== SERVER> Starting atomic add on shared memory demo. CLIENT> ****** libmetal demo: atomic operation over shared memory ****** metal: info: metal_linux_dev_open: checking driver vfio-platform,3ed80000.shm,(null) metal: info: metal_uio_dev_open: No IRQ for device 3ed80000.shm. metal: info: metal_linux_dev_open: checking driver vfio-platform,ff360000.ipi,(null) CLIENT> Starting atomic shared memory task. SERVER> Shared memory with atomics test finished SERVER> ====== libmetal demo: IPI and shared memory ====== SERVER> Wait for echo test to start. CLIENT> shm atomic demo PASSED! CLIENT> ****** libmetal demo: IPI and shared memory ****** metal: info: metal_linux_dev_open: checking driver vfio-platform,3ed80000.shm,(null) metal: info: metal_uio_dev_open: No IRQ for device 3ed80000.shm. metal: info: metal_linux_dev_open: checking driver vfio-platform,ff360000.ipi,(null) CLIENT> Start echo flood testing.... CLIENT> Sending msgs to the remote. CLIENT> Waiting for messages to echo back and verify. CLIENT> Kick remote to notify shutdown message sent... SERVER> Received shutdown message SERVER> IPI with shared memory demCLIENT> Total packages: 1024, time_avg = 0s, 12780nso finished with exit code: 0 . SERVER> ====== libmetal demo: IPI latency ====== SERVER> Starting IPI latency demo CLIENT> ****** libmetal demo: IPI latency ****** metal: info: metal_linux_dev_open: checking driver vfio-platform,3ed80000.shm,(null) metal: info: metal_uio_dev_open: No IRQ for device 3ed80000.shm. metal: info: metal_linux_dev_open: checking driver vfio-platform,ff0e0000.ttc0,(null) metal: info: metal_uio_dev_open: No IRQ for device ff0e0000.ttc0. metal: info: metal_linux_dev_open: checking driver vfio-platform,ff360000.ipi,(null) CLIENT> Starting IPI latency task CLIENT> sleep(1) check: TTC= 1331148510 / CLOCK_MONOTONIC= 1001169920 = 1.33 CLIENT> IPI latency: 1000 iterations took 1848847250 n SERVER> ===s (CLOCK_MONOTONIC)=== libmetal demo: shared memory latency ====== CLIENT> TTC [min,max] are in TTC ticks: 10 ns per tick CLIENT SERVER> Starting IPI latency demo > APU to RPU: [3278, 168236] avg: 61759 ns CLIENT> RPU to APU: [13876, 2672383] avg: 1898774 ns CLIENT> Finished IPI latency task CLIENT> ****** libmetal demo: shared memory latency ****** metal: info: metal_linux_dev_open: checking driver vfio-platform,3ed80000.shm,(null) metal: info: metal_uio_dev_open: No IRQ for device 3ed80000.shm. metal: info: metal_linux_dev_open: checking driver vfio-platform,ff0e0000.ttc0,(null) metal: info: metal_uio_dev_open: No IRQ for device ff0e0000.ttc0. metal: info: metal_linux_dev_open: checking driver vfio-platform,ff360000.ipi,(null) CLIENT> Starting shared memory latency task TTC [min,max] are in TTC ticks: 10 ns per tick CLIENT> package size 16 latency: CLIENT> APU to RPU: [6322, 95942] avg: 85698 ns CLIENT> RPU to APU: [38280, 2378071] avg: 3995714 ns CLIENT> package size 32 latency: CLIENT> APU to RPU: [6100, 84200] avg: 86960 ns CLIENT> RPU to APU: [33206, 2343168] avg: 3896139 ns CLIENT> package size 64 latency: CLIENT> APU to RPU: [5869, 751036] avg: 94125 ns CLIENT> RPU to APU: [30962, 2555077] avg: 3709274 ns CLIENT> package size 128 latency: CLIENT> APU to RPU: [6425, 88837] avg: 88917 ns CLIENT> RPU to APU: [39736, 1838721] avg: 3822752 ns CLIENT> package size 256 latency: CLIENT> APU to RPU: [6325, 2319090] avg: 116439 ns CLIENT> RPU to APU: [30589, 2576644] avg: 3873588 ns CLIENT> package size 512 latency: CLIENT> APU to RPU: [6672, 67747] avg: 95702 ns CLIENT> RPU to APU: [40773, 1882366] avg: 3608078 ns CLIENT> package size 1024 latency: CLIENT> APU to RPU: [6633, 38775] avg: 91844 ns CLIENT> RPU to APU: [34869, 1855085] avg: 2662481 ns CLIENT> Finished shared memory latency task SERVER> ====== libmetal demo: shared memory throughput ====== SERVER> Starting shared mem throughput demo CLIENT> ****** libmetal demo: shared memory throughput ****** metal: info: metal_linux_dev_open: checking driver vfio-platform,3ed80000.shm,(null) metal: info: metal_uio_dev_open: No IRQ for device 3ed80000.shm. metal: info: metal_linux_dev_open: checking driver vfio-platform,ff0e0000.ttc0,(null) metal: info: metal_uio_dev_open: No IRQ for device ff0e0000.ttc0. metal: info: metal_linux_dev_open: checking driver vfio-platform,ff360000.ipi,(null) CLIENT> Starting shared mem throughput demo [ 103.882296] hrtimer: interrupt took 10665070 ns [ 104.545676] sched: RT throttling activated CLIENT> Shared memory throughput of pkg size 16 : CLIENT> APU send: 268015043, 1.5 MB/s CLIENT> RPU receive: 280875726, 1.4 MB/s CLIENT> RPU send: 241277296, 1.7 MB/s CLIENT> APU receive: 241518127, 1.7 MB/s CLIENT> Shared memory throughput of pkg size 32 : CLIENT> APU send: 139217799, 2.9 MB/s CLIENT> RPU receive: 143476969, 2.8 MB/s CLIENT> RPU send: 119158394, 3.4 MB/s CLIENT> APU receive: 113943809, 3.5 MB/s CLIENT> Shared memory throughput of pkg size 64 : CLIENT> APU send: 69706256, 5.7 MB/s CLIENT> RPU receive: 72084178, 5.5 MB/s CLIENT> RPU send: 60930571, 6.6 MB/s CLIENT> APU receive: 57316156, 7.0 MB/s CLIENT> Shared memory throughput of pkg size 128 : CLIENT> APU send: 32433331, 12.3 MB/s CLIENT> RPU receive: 34144369, 11.7 MB/s CLIENT> RPU send: 33306619, 12.0 MB/s CLIENT> APU receive: 21222799, 18.8 MB/s CLIENT> Shared memory throughput of pkg size 256 : CLIENT> APU send: 16963667, 23.6 MB/s CLIENT> RPU receive: 18042092, 22.2 MB/s CLIENT> RPU send: 16303446, 24.5 MB/s CLIENT> APU receive: 16862468, 23.7 MB/s CLIENT> Shared memory throughput of pkg size 512 : CLIENT> APU send: 9366064, 42.7 MB/s CLIENT> RPU receive: 10539373, 38.0 MB/s CLIENT> RPU send: 8365674, 47.8 MB/s CLIENT> APU receive: 1772022, 225.7 MB/s CLIENT> Shared memory throughput of pkg size 1024 : CLIENT> APU send: 5731912, 69.8 MB/s CLIENT> RPU receive: 6596383, 60.6 MB/s CLIENT> RPU send: 5519762, 72.5 MB/s CLIENT> APU receive: 4745741, 84.3 MB/s CLIENT> Finished shared memory throughput

Related Links



© Copyright 2019 - 2022 Xilinx Inc. Privacy Policy