This design note is to document a set of steps in Vitis and PetaLinux that will create a system with Linux on the APU and bare metal on each of the R5s in the RPU of a Versal device (in this example a VCK190). Libmetal will be used to communicate between the APU and RPU.

In this system, the APU will boot first and bring up and initialize each of the R5s. Example applications for the R5s and Linux on the APU will be created to use many features of Libmetal.

The VCK190 and 2023.2 tools are used in this demonstration.
- Vitis Classic is used, not Vitis Unified
The system boots from SD.

Introduction

This demonstration leverages the Versal Libmetal Demo and the Libmetal AMP Demo in (UG1186). In each of those references there is only a single R5 instance running with a single Linux application interacting with it. For this exercise the R5s will operate in split mode with each having its own application. There will be two Linux applications, each paired with an R5 instance.

General Steps

It is assumed that you have an XSA exported from a Vivado project for the VCK190 which incorporates the requirements noted in the Appendix
Create bare metal applications for each of the R5s which use Libmetal and have been configured to run on the respective core.
Create a PetaLinux project which includes Libmetal with a properly configured device tree and make the resulting sysroot available for the creation of the Linux applications.
Create Linux applications which will use Libmetal to communicate with each of the respective R5s.
Integrate the R5 bare metal applications as well as the Linux applications into the Linux image and package the project to be deployed on the target board.
Deploy the packaged project on the VCK190 and exercise the various included Libmetal functions.

Workflow

Vitis Pass 1 : Create a Vitis project for the VCK190 which incorporates Libmetal AMP Demo applications for R5-0 and R5-1.

PetaLinux Pass 1 : Create a PetaLinux project for the VCK190 using its BSP, configure the project, and create a sysroot which can be used in Vitis for the creation of the Linux applications.

Vitis Pass 2 : Create the Libmetal AMP demo Linux applications, configuring them to use the sysroot from the PetaLinux project.

PetaLinux Pass 2 : Integrate the R5 and Linux apps into the Linux image, configure the PetaLinux project, build, and package.

Deploy and Test : Provision the VCK190 to boot into the demo and run the demo.

Vitis Pass 1 and 2 can be merged if PetaLinux Pass 1 is completed first.

Vitis Pass 1

Create the platform for bare metal applications

Launch Vitis in classic mode (vitis --classic on Linux)

Create a Vitis project in a workspace named vitis_prj.

Close the welcome tab.

Select File->New->Platform Project…

Platform Project Name: VCK190_hw
Click Next
Under the tab "Create a new platform from hardware (XSA)' use the
drop down menu to select vck190
Set the Processor to versal_cips_0_pspmc_0_psv_cortexr5_0
Click Finish

Modify the versal_cips_0_pspmc_0_psv_cortexr5_0 Domain.

Use Modify BSP Settings… to add the LibMetal library

Build the platform.

Create bare metal application for R5-0

Select File->New->Application Project…

Click Next
Click Next (use the newly created platform and domain)
Application project name: my-R5-0-app
Click Next
Select Libmetal AMP Demo
Click Finish

Modify the my-R5-0-app application. The changes are to define the shared memory region between R5-0 and the APU (adjusted because another region for the R5-1 and APU will be needed) and to make the print statements easily identifiable to R5-0.

In common.h change

#define SHM_DEV_NAME "3ed80000.shm'
#define SHM_BASE_ADDR 0x3ED800000
xil_printf("\r\nSERVER> " format, ##__V_ARGS__)

To

#define SHM_DEV_NAME "3d000000.shm"
#define SHM_BASE_ADDR 0x3D0000000
xil_printf("\r\nSERVER0> " format, ##__VA_ARGS__)

Build the application by right clicking on it and selecting Build Project.

Create the bare metal application for R5-1

Create a Domain for R5-1.

Select platform.spr in VCK190_hw.

Use the green '+' to add a domain.

Name : R5-1-standalone

Processor : versal_cips_0_pspmc_0_psv_cortexr5_1

OK

Modify the versal_cips_0_pspmc_0_psv_cortexr5_1 domain.

Use Modify BSP Settings… to add the LibMetal library

Build the platform.
Select File->New->Application Project . . .

Click Next

Click Next (use the VCK190_hw platform)

Application project name : my-R5-1-app

Choose the R5-1 processor

Click Next

Select Libmetal AMP Demo

Click Finish

Modify the my-R5-1-app application. The changes are to define the shared memory region between R5-1 and the APU (accounting for the region allocated to R5-0 and the APU) and to make the print statements easily identifiable to R5-1. Additionally, a different IPI channel is being used for the R5-1 communications with the APU than the one for the R5-0 so its corresponding base address, interrupt vector ID, and mask are adjusted accordingly. Finally, a different TTC is used for R5-1 so its corresponding values are modified.

In common.h change

#define SHM_DEV_NAME "3ed80000.shm'
#define SHM_BASE_ADDR 0x3ED800000
xil_printf("\r\nSERVER> " format, ##__V_ARGS__)

#define IPI_DEV_NAME "ff340000.ipi"

#define IPI_BASE_ADDR 0xFF340000

#define IPI_IRQ_VECT_ID 63

#define IPI_MASK 0x20

#define TTC0_BASE_ADDR 0xFF0E0000

#define TTC_DEV_NAME "ff0E0000.ttc"

To

#define SHM_DEV_NAME "3e000000.shm"
#define SHM_BASE_ADDR 0x3E0000000
xil_printf("\r\nSERVER1> " format, ##__VA_ARGS__)

#define IPI_DEV_NAME "ff350000.ipi"

#define IPI_BASE_ADDR 0xFF350000

#define IPI_IRQ_VECT_ID 64

#define IPI_MASK 0x40

#define TTC0_BASE_ADDR 0xFF0F0000

#define TTC_DEV_NAME "ff0F0000.ttc"

Build the application by right clicking on it and selecting Build Project.

PetaLinux Pass 1

Create and configure the PetaLinux project

Create the project using the VCK190 BSP.

petalinux-create -t project -s <>/xilinx-vck190-v2023.2-10140544.bsp -n plnx_prj
cd plnx_prj
petalinux-config -c rootfs

Filesystem Packages->libs->libmetal->[*]libmetal

Filesystem Packages->libs->libmetal->[*]libmetal-demos

Filesystem Packages->misc->libudev->[*]libudev

Filesystem Packages->misc->sysfsutils->[*]libsysfs

Exit all the way out, saving the configuration changes when prompted.

Replace the contents of plnx_prj /project-spec/meta-user/recipes-bsp /device-tree/files/system-user.dtsi with the code shown below.

/include/ "system-conf.dtsi"
/ {
};

/{
        reserved-memory {
                #address-cells = <2>;
                #size-cells = <2>;
                ranges;
                rproc_0_reserved: rproc@3cd00000 {
                        no-map;
                        reg = <0x0 0x3cd00000 0x0 0x80000>;
                };
                rproc_1_reserved: rproc@3cd80000 {
                        no-map;
                        reg = <0x0 0x3cd80000 0x0 0x80000>;
                };
        };

        tcm_0a@ffe00000 {
                no-map;
                reg = <0x0 0xffe00000 0x0 0x10000>;
                phandle = <0x40>;
                status = "okay";
                compatible = "mmio-sram";
                power-domain = <&versal_firmware 0x1831800b>;
        };

        tcm_0b@ffe20000 {
                no-map;
                reg = <0x0 0xffe20000 0x0 0x10000>;
                phandle = <0x41>;
                status = "okay";
                compatible = "mmio-sram";
                power-domain = <&versal_firmware 0x1831800c>;
        };

        tcm_1a@ffe90000 {
                no-map;
                reg = <0x0 0xffe90000 0x0 0x10000>;
                phandle = <0x42>;
                status = "okay";
                compatible = "mmio-sram";
                power-domain = <&versal_firmware 0x1831800d>;
        };

        tcm_1b@ffeb0000 {
                no-map;
                reg = <0x0 0xffeb0000 0x0 0x10000>;
                phandle = <0x43>;
                status = "okay";
                compatible = "mmio-sram";
                power-domain = <&versal_firmware 0x1831800e>;
        };

        rf5ss@ff9a0000 {
                compatible = "xlnx,versal-r5-remoteproc";
                xlnx,cluster-mode = <1>;
                ranges;
                reg = <0x0 0xFF9A0000 0x0 0x10000>;
                #address-cells = <0x2>;
                #size-cells = <0x2>;

                r5f_0 {
                        compatible = "xilinx,r5f";
                        #address-cells = <2>;
                        #size-cells = <2>;
                        ranges;
                        sram = <0x40 0x41>;
                        memory-region = <&rproc_0_reserved>;
                        power-domain = <&versal_firmware 0x18110005>;
                };

                r5f_1 {
                        compatible = "xilinx,r5f";
                        #address-cells = <2>;
                        #size-cells = <2>;
                        ranges;
                        sram = <0x42 0x43>;
                        memory-region = <&rproc_1_reserved>;
                        power-domain = <&versal_firmware 0x18110006>;
                };

        };
	/* Shared memory */
	shm0: shm@0 {
	         compatible = "shm_uio";
	         reg = <0x0 0x3d000000 0x0 0x1000000>;
	};
	shm1: shm@1 {
	         compatible = "shm_uio";
	         reg = <0x0 0x3e000000 0x0 0x1000000>;
	};
	/* IPI device */
	/* IPI 3 */
	ipi_amp3: ipi@ff360000 {
	        compatible = "ipi_uio";
	        reg = <0x0 0xff360000 0x0 0x1000>;
	        interrupt-parent = <&gic>;
	        interrupts = <0 33 4>;
	};
	/* IPI 4 */
	ipi_amp4: ipi@ff370000 {
	        compatible = "ipi_uio";
	        reg = <0x0 0xff370000 0x0 0x1000>;
	        interrupt-parent = <&gic>;
	        interrupts = <0 34 4>;
	};
	ttc0 {
        	compatible = "ttc-uio";
		reg = <0x0 0xFF0E0000 0x0 0x1000>;
	};
	ttc1 {
        	compatible = "ttc-uio";
		reg = <0x0 0xFF0F0000 0x0 0x1000>;
	};
};

Incorporate the Vivado Output.

petalinux-config --get-hw-description=./vivado_prj/HSI

Build and package the PetaLinux project

Build and package the sysroot.

Vitis Pass 2

Create the platform for Linux applications

Create a new platform project (for Linux) in the original workspace.

Select File->New->Platform Project…

Platform Project Name : libmetaldemo-plat

XSA File : vck190 (from drop down list)

Operating system : linux

Processor : psv_cortexa72

Click Finish

In the platform.spr for libmetaldemo-plat select the 'linux on psv_cortexa72’ setting group.

Sysroot directory : Directory : <>/plnx_prj/images/linux/sdk/sysroots/cortexa72-cortexa53-xilinx-linux

Build Project on libmetaldemo-plat.

Create the Linux application paired to R5-0

Create a new application project (Libmetal).

Select File->New->Application Project…

Use the libmetaldemo-plat platform, Next

Application project name : my-lm-linux-app0

sysroot path : should be populated using the one that was assigned to the platform

Root FS : <>/plnx_prj/images/linux/rootfs.ext4

Kernel image : <>/plnx_prj/images/linux/image.ub

Template : Linux Empty Application

Click Finish

Open C/C++ Build Settings on the app.

C/C++ Settings->ARM v8 Linux gcc linker->Libraries

add metal as a library

Click Apply and Close

Clone the Libmetal repository to get the example code.

Import the following source files from <>/libmetal/examples/system/linux/zynqmp/zynqmp_amp_demo by right clicking on the app, selecting Import Sources… and navigating to the directory just noted.

common.h	ipi_latency_demo.c	ipi_shmem_demo.c

common.h	ipi_latency_demo.c	ipi_shmem_demo.c
ipi-uio.c	shmem_atomic_demo.c	shmem_demo.c
shmem_latency_demo.c	shmem_throughput_demo.c	sys_init.c
sys_init.h	libmetal_amp_demo.c

Add the compiler flag CONFIG_IPI_MASK=0x08 under ARM v8 Linux gcc compiler symbols. This will select which IPI channel is used.

In common.h change

#define IPI_DEV_NAME CONFIG_IPI_DEV_NAME
#define SHM_DEV_NAME "3ed80000.shm"

#define TTC_DEV_NAME CONFIG_TTC_DEV_NAME
xil_printf("CLIENT> " format, ##__V_ARGS__)

To

#define IPI_DEV_NAME "ff360000.ipi"

#define SHM_DEV_NAME "3d000000.shm"
#define TTC_DEV_NAME "ff0e0000.ttc0"
xil_printf("CLIENT0> " format, ##__VA_ARGS__)

Build the application by right clicking on it and selecting Build Project.

Create Linux application paired to R5-1

Create a new application project (Libmetal).

Select File->New->Application Project…

Use the libmetaldemo-plat platform, click Next

Application project name : my-lm-linux-app1

sysroot path : should be populated using the one that was assigned to the platform

Root FS : <>/plnx_prj/images/linux/rootfs.ext4

Kernel image : <>/plnx_prj/images/linux/image.ub

Template : Linux Empty Application

Click Finish

Open C/C++ Build Settings on the app.

C/C++ Settings->ARM v8 Linux gcc linker->Libraries

add metal as a library

Click Apply and Close

Import the same set of source files from <>/libmetal/examples/system/linux/zynqmp/zynqmp_amp_demo as was done for my-lm-linux-app0.
Add the compiler flag CONFIG_IPI_MASK=0x10 under ARM v8 Linux gcc compiler symbols. This will select a different IPI channel than the one used for my-lm-linux-app0.

In common.h change

#define IPI_DEV_NAME CONFIG_IPI_DEV_NAME
#define SHM_DEV_NAME "3ed80000.shm"

#define TTC_DEV_NAME CONFIG_TTC_DEV_NAME
xil_printf("CLIENT> " format, ##__V_ARGS__)

To

#define IPI_DEV_NAME "ff370000.ipi"

#define SHM_DEV_NAME "3e000000.shm"
#define TTC_DEV_NAME "ff0f0000.ttc1"
xil_printf("CLIENT1> " format, ##__VA_ARGS__)

Build the application by right clicking on it and selecting Build Project.

PetaLinux Pass 2

Incorporate applications for R5-0 into rootfs

Create an application in the PetaLinux project.

In plnx_prj execute petalinux-create -t apps --template install -n my-lm-linux-app0 --enable.

Copy in pre-built apps for APU and RPU.

Replace the contents of <>/plnx_prj/project-spec/meta-user/recipes-apps/my-lm-linux-app0/my-lm-linux-app0.bb with the code below. This adds references to the apps for R5-0 and APU (app0), notes dependencies for the Linux app, and instructs where to locate these apps in the root file system.

Incorporate applications for R5-1 into rootfs

Create an application in the PetaLinux project.

In plnx_prj execute petalinux-create -t apps --template install -n my-lm-linux-app1 --enable.

Copy in pre-built apps for APU and RPU.

Replace the contents of <>/plnx_prj/project-spec/meta-user/recipes-apps/my-lm-linux-app1/my-lm-linux-app1.bb with the code below. This adds references to the apps for R5-1 and APU (app1), notes dependencies for the Linux app, and instructs you where to locate these apps in the root file system.

Configure rootfs

Change the rootfs type (this is done to avoid using an ext4 partition on the SD card. You could instead install the rootfs to an ext4 partition).

cd to <>/plnx_prj

petalinux-config

Image Packaging Configuration->INITRAMFS/INITRD Image name

From

To

Exit all the way out, saving the configuration changes when prompted.

Build and package PetaLinux project

Rebuild and package the PetaLinux project.

Deploy and Test

Prepare test platform

Copy BOOT.BIN, boot.scr, and image.ub onto SD card.
Power up the VCK190 with the SD inserted and boot mode set to SD.

Login is petalinux, you will be prompted to set a password.

Check that the files are in place.

ls /lib/firmware, should see my-R5-0-app.elf, my-R5-1-app.elf

Execute tests

At the Linux prompt, execute the following.

An example session log is attached as LM_DualR5_Log.txt.

Appendix

In the Vivado project it is necessary to ensure that the Inter-Processor Interrupts (IPI) and the Triple Timer Counters (TTCs) are properly enabled / configured.

The IPIs are used for communications between the APU and R5-0 / R5-1. The TTCs are used to measure time intervals in the demo applications.

See below for where this is done in the CIPS wizard.

Xilinx Wiki

Libmetal AMP Demo - Dual R5s on VCK190

Table of Contents

Introduction

General Steps

Workflow

Vitis Pass 1

Create the platform for bare metal applications

Create bare metal application for R5-0

Create the bare metal application for R5-1

PetaLinux Pass 1

Create and configure the PetaLinux project

Build and package the PetaLinux project

Vitis Pass 2

Create the platform for Linux applications

Create the Linux application paired to R5-0

Create Linux application paired to R5-1

PetaLinux Pass 2

Incorporate applications for R5-0 into rootfs

Incorporate applications for R5-1 into rootfs

Configure rootfs

Build and package PetaLinux project

Deploy and Test

Prepare test platform

Execute tests

Appendix

Related Links