RPMsg in Linux and OpenAMP Project

This document covers how to configure RPMsg in Kernelspace or Userspace.

Table of Contents

Introduction

The OpenAMP RPMsg API allows inter-process communications (IPC) between software running on independent cores in an AMP system. This is also compliant with the RPMsg BUS infrastructure present in the Linux Kernel version 3.18 and later.

There are different implementations of RPMsg,

  1. Linux kernel rpmsg (primary) and RPU OpenAMP remote processor (secondary).

  2. Linux userspace OpenAMP application (primary) and RPU OpenAMP application (secondary).

Some important configuration elements to take care of when dealing with RPMsg are the resource table (for firmware), shared memory with shared buffers and vring buffers. There is also another configuration element which is the Inter-Processor Interrupt (IPI) which is used by one core to notify the other.

  • Resource table: Indicates the location of vrings (only for remote processor)

  • Shared memory buffers

RPMsg in Kernelspace

Figure: RPMsg Implementation in Kernel Space

The Linux rpmsg driver parses the RPU firmware ELF and updates the resource table in the ELF with vring address locations (based on what is provided in the kernel device tree). The shared memory address in the RPU firmware should match the shared memory address in the kernel device tree.

Modifications for RPMsg in kernel space

Refer to the OpenAMP Application Build Process page for Building Linux Distribution using PetaLinux, for modifying Demo Tests Firmware, and for booting up and running demo tests in kernelspace.

OpenAMP version specific modifications are covered below; for example, changes in the device tree, RPU firmware, Linux application, etc.

2022.1 Example

Device Tree

Device tree file location:

<plnx-proj-root>/project-spec/meta-user/recipes-bsp/device-tree/files/system-user.dtsi

Start with the sample DTSI files published at Xilinx/meta-openamp/zynqmp-openamp.dtsi and then refer to the following sample device tree examples for further changes.

/ { reserved-memory { #address-cells = <2>; #size-cells = <2>; ranges; rpu0vdev0vring0: rpu0vdev0vring0@3ed40000 { no-map; reg = <0x0 0x3ed40000 0x0 0x4000>; }; rpu0vdev0vring1: rpu0vdev0vring1@3ed44000 { no-map; reg = <0x0 0x3ed44000 0x0 0x4000>; }; rpu0vdev0buffer: rpu0vdev0buffer@3ed48000 { no-map; reg = <0x0 0x3ed48000 0x0 0x100000>; }; rproc_0_reserved: rproc@3ed00000 { no-map; reg = <0x0 0x3ed00000 0x0 0x40000>; }; }; tcm_0a@ffe00000 { no-map; reg = <0x0 0xffe00000 0x0 0x10000>; phandle = <0x40>; status = "okay"; compatible = "mmio-sram"; power-domain = <&zynqmp_firmware 15>; }; tcm_0b@ffe20000 { no-map; reg = <0x0 0xffe20000 0x0 0x10000>; phandle = <0x41>; status = "okay"; compatible = "mmio-sram"; power-domain = <&zynqmp_firmware 16>; }; rf5ss@ff9a0000 { compatible = "xlnx,zynqmp-r5-remoteproc"; xlnx,cluster-mode = <0>; /*Lock-step mode*/ ranges; reg = <0x0 0xFF9A0000 0x0 0x10000>; #address-cells = <0x2>; #size-cells = <0x2>; r5f_0 { compatible = "xilinx,r5f"; #address-cells = <2>; #size-cells = <2>; ranges; sram = <0x40 0x41>; memory-region = <&rproc_0_reserved>, <&rpu0vdev0buffer>, <&rpu0vdev0vring0>, <&rpu0vdev0vring1>; power-domain = <&zynqmp_firmware 7>; mboxes = <&ipi_mailbox_rpu0 0>, <&ipi_mailbox_rpu0 1>; mbox-names = "tx", "rx"; }; }; zynqmp_ipi1 { compatible = "xlnx,zynqmp-ipi-mailbox"; interrupt-parent = <&gic>; interrupts = <0 29 4>; xlnx,ipi-id = <7>; #address-cells = <1>; #size-cells = <1>; ranges; /* APU<->RPU0 IPI mailbox controller */ ipi_mailbox_rpu0: mailbox@ff990600 { reg = <0xff990600 0x20>, <0xff990620 0x20>, <0xff9900c0 0x20>, <0xff9900e0 0x20>; reg-names = "local_request_region", "local_response_region", "remote_request_region", "remote_response_region"; #mbox-cells = <1>; xlnx,ipi-id = <1>; }; }; };
/ { reserved-memory { #address-cells = <2>; #size-cells = <2>; ranges; rpu0vdev0vring0: rpu0vdev0vring0@3ed40000 { no-map; reg = <0x0 0x3ed40000 0x0 0x4000>; }; rpu0vdev0vring1: rpu0vdev0vring1@3ed44000 { no-map; reg = <0x0 0x3ed44000 0x0 0x4000>; }; rpu0vdev0buffer: rpu0vdev0buffer@3ed48000 { no-map; reg = <0x0 0x3ed48000 0x0 0x100000>; }; rproc_0_reserved: rproc@3ed00000 { no-map; reg = <0x0 0x3ed00000 0x0 0x40000>; }; rproc_1_reserved: rproc@3ef00000 { no-map; reg = <0x0 0x3ef00000 0x0 0x40000>; }; rpu1vdev0vring0: rpu1vdev0vring0@3ef40000 { no-map; reg = <0x0 0x3ef40000 0x0 0x4000>; }; rpu1vdev0vring1: rpu1vdev0vring1@3ef44000 { no-map; reg = <0x0 0x3ef44000 0x0 0x4000>; }; rpu1vdev0buffer: rpu1vdev0buffer@3ef48000 { no-map; compatible = "shared-dma-pool"; reg = <0x0 0x3ef48000 0x0 0x100000>; }; }; tcm_0a@ffe00000 { no-map; reg = <0x0 0xffe00000 0x0 0x10000>; phandle = <0x40>; status = "okay"; compatible = "mmio-sram"; power-domain = <&zynqmp_firmware 15>; }; tcm_0b@ffe20000 { no-map; reg = <0x0 0xffe20000 0x0 0x10000>; phandle = <0x41>; status = "okay"; compatible = "mmio-sram"; power-domain = <&zynqmp_firmware 16>; }; tcm_1a@ffe90000 { no-map; reg = <0x0 0xffe90000 0x0 0x10000>; phandle = <0x42>; status = "okay"; compatible = "mmio-sram"; power-domain = <&zynqmp_firmware 17>; }; tcm_1b@ffeb0000 { no-map; reg = <0x0 0xffeb0000 0x0 0x10000>; phandle = <0x43>; status = "okay"; compatible = "mmio-sram"; power-domain = <&zynqmp_firmware 18>; }; rf5ss@ff9a0000 { compatible = "xlnx,zynqmp-r5-remoteproc"; xlnx,cluster-mode = <1>; /*Split mode*/ ranges; reg = <0x0 0xFF9A0000 0x0 0x10000>; #address-cells = <0x2>; #size-cells = <0x2>; r5f_0 { compatible = "xilinx,r5f"; #address-cells = <2>; #size-cells = <2>; ranges; sram = <0x40 0x41>; memory-region = <&rproc_0_reserved>, <&rpu0vdev0buffer>, <&rpu0vdev0vring0>, <&rpu0vdev0vring1>; power-domain = <&zynqmp_firmware 7>; mboxes = <&ipi_mailbox_rpu0 0>, <&ipi_mailbox_rpu0 1>; mbox-names = "tx", "rx"; }; r5f_1 { compatible = "xilinx,r5f"; #address-cells = <2>; #size-cells = <2>; ranges; sram = <0x42 0x43>; memory-region = <&rproc_1_reserved>, <&rpu1vdev0buffer>, <&rpu1vdev0vring0>, <&rpu1vdev0vring1>; power-domain = <&zynqmp_firmware 8>; mboxes = <&ipi_mailbox_rpu1 0>, <&ipi_mailbox_rpu1 1>; mbox-names = "tx", "rx"; }; }; zynqmp_ipi1 { compatible = "xlnx,zynqmp-ipi-mailbox"; interrupt-parent = <&gic>; interrupts = <0 29 4>; xlnx,ipi-id = <7>; #address-cells = <1>; #size-cells = <1>; ranges; /* APU<->RPU0 IPI mailbox controller */ ipi_mailbox_rpu0: mailbox@ff990600 { reg = <0xff990600 0x20>, <0xff990620 0x20>, <0xff9900c0 0x20>, <0xff9900e0 0x20>; reg-names = "local_request_region", "local_response_region", "remote_request_region", "remote_response_region"; #mbox-cells = <1>; xlnx,ipi-id = <1>; }; }; zynqmp_ipi2 { compatible = "xlnx,zynqmp-ipi-mailbox"; interrupt-parent = <&gic>; interrupts = <0 30 4>; xlnx,ipi-id = <8>; #address-cells = <1>; #size-cells = <1>; ranges; /* APU<->RPU1 IPI mailbox controller */ ipi_mailbox_rpu1: mailbox@ff990640 { reg = <0xff3f0b00 0x20>, <0xff3f0b20 0x20>, <0xff3f0940 0x20>, <0xff3f0960 0x20>; reg-names = "local_request_region", "local_response_region", "remote_request_region", "remote_response_region"; #mbox-cells = <1>; xlnx,ipi-id = <2>; }; }; };
/ { reserved-memory { #address-cells = <2>; #size-cells = <2>; ranges; rproc_0_reserved: rproc@3ed00000 { no-map; reg = <0x0 0x3ed00000 0x0 0x40000>; }; rpu0vdev0vring0: rpu0vdev0vring0@3ed40000 { no-map; reg = <0x0 0x3ed40000 0x0 0x4000>; }; rpu0vdev0vring1: rpu0vdev0vring1@3ed44000 { no-map; reg = <0x0 0x3ed44000 0x0 0x4000>; }; rpu0vdev0buffer: rpu0vdev0buffer@3ed48000 { no-map; reg = <0x0 0x3ed48000 0x0 0x100000>; }; }; tcm_0a@ffe00000 { no-map; reg = <0x0 0xffe00000 0x0 0x10000>; phandle = <0x40>; status = "okay"; compatible = "mmio-sram"; power-domain = <&versal_firmware 0x1831800b>; }; tcm_0b@ffe20000 { no-map; reg = <0x0 0xffe20000 0x0 0x10000>; phandle = <0x41>; status = "okay"; compatible = "mmio-sram"; power-domain = <&versal_firmware 0x1831800c>; }; rf5ss@ff9a0000 { compatible = "xlnx,zynqmp-r5-remoteproc"; #address-cells = <0x2>; #size-cells = <0x2>; ranges; xlnx,cluster-mode = <0>; /*Lock-step mode*/ reg = <0x0 0xff9a0000 0x0 0x10000>; r5f_0 { compatible = "xilinx,r5f"; #address-cells = <0x2>; #size-cells = <0x2>; ranges; sram = <0x40 0x41>; memory-region = <&rproc_0_reserved>, <&rpu0vdev0buffer>, <&rpu0vdev0vring0>, <&rpu0vdev0vring1>; power-domain = <&versal_firmware 0x18110005>; mboxes = <&ipi_mailbox_rpu0 0>, <&ipi_mailbox_rpu0 1>; mbox-names = "tx", "rx"; }; }; zynqmp_ipi1 { compatible = "xlnx,zynqmp-ipi-mailbox"; interrupt-parent = <&gic>; interrupts = <0 33 4>; xlnx,ipi-id = <5>; #address-cells = <1>; #size-cells = <1>; ranges; /* APU<->RPU0 IPI mailbox controller */ ipi_mailbox_rpu0: mailbox@ff990600 { reg = <0xff3f0ac0 0x20>, <0xff3f0ae0 0x20>, <0xff3f0740 0x20>, <0xff3f0760 0x20>; reg-names = "local_request_region", "local_response_region", "remote_request_region", "remote_response_region"; #mbox-cells = <1>; xlnx,ipi-id = <3>; }; }; };
/ { reserved-memory { #address-cells = <2>; #size-cells = <2>; ranges; rproc_0_reserved: rproc@3ed00000 { no-map; reg = <0x0 0x3ed00000 0x0 0x40000>; }; rpu0vdev0vring0: rpu0vdev0vring0@3ed40000 { no-map; reg = <0x0 0x3ed40000 0x0 0x4000>; }; rpu0vdev0vring1: rpu0vdev0vring1@3ed44000 { no-map; reg = <0x0 0x3ed44000 0x0 0x4000>; }; rpu0vdev0buffer: rpu0vdev0buffer@3ed48000 { no-map; reg = <0x0 0x3ed48000 0x0 0x100000>; }; rproc_1_reserved: rproc@3ef00000 { no-map; reg = <0x0 0x3ef00000 0x0 0x40000>; }; rpu1vdev0vring0: rpu1vdev0vring0@3ef40000 { no-map; reg = <0x0 0x3ef40000 0x0 0x4000>; }; rpu1vdev0vring1: rpu1vdev0vring1@3ef44000 { no-map; reg = <0x0 0x3ef44000 0x0 0x4000>; }; rpu1vdev0buffer: rpu1vdev0buffer@3ef48000 { no-map; compatible = "shared-dma-pool"; reg = <0x0 0x3ef48000 0x0 0x100000>; }; }; tcm_0a@ffe00000 { no-map; reg = <0x0 0xffe00000 0x0 0x10000>; phandle = <0x40>; status = "okay"; compatible = "mmio-sram"; power-domain = <&versal_firmware 0x1831800b>; }; tcm_0b@ffe20000 { no-map; reg = <0x0 0xffe20000 0x0 0x10000>; phandle = <0x41>; status = "okay"; compatible = "mmio-sram"; power-domain = <&versal_firmware 0x1831800c>; }; tcm_1a@ffe90000 { no-map; reg = <0x0 0xffe90000 0x0 0x10000>; phandle = <0x42>; status = "okay"; compatible = "mmio-sram"; power-domain = <&versal_firmware 0x1831800d>; }; tcm_1b@ffeb0000 { no-map; reg = <0x0 0xffeb0000 0x0 0x10000>; phandle = <0x43>; status = "okay"; compatible = "mmio-sram"; power-domain = <&versal_firmware 0x1831800e>; }; rf5ss@ff9a0000 { compatible = "xlnx,zynqmp-r5-remoteproc"; #address-cells = <0x2>; #size-cells = <0x2>; ranges; xlnx,cluster-mode = <1>; /*Split mode*/ reg = <0x0 0xff9a0000 0x0 0x10000>; r5f_0 { compatible = "xilinx,r5f"; #address-cells = <0x2>; #size-cells = <0x2>; ranges; sram = <0x40 0x41>; memory-region = <&rproc_0_reserved>, <&rpu0vdev0buffer>, <&rpu0vdev0vring0>, <&rpu0vdev0vring1>; power-domain = <&versal_firmware 0x18110005>; mboxes = <&ipi_mailbox_rpu0 0>, <&ipi_mailbox_rpu0 1>; mbox-names = "tx", "rx"; }; r5f_1 { compatible = "xilinx,r5f"; #address-cells = <0x2>; #size-cells = <0x2>; ranges; sram = <0x42 0x43>; memory-region = <&rproc_1_reserved>, <&rpu1vdev0buffer>, <&rpu1vdev0vring0>, <&rpu1vdev0vring1>; power-domain = <&versal_firmware 0x18110006>; mboxes = <&ipi_mailbox_rpu1 0>, <&ipi_mailbox_rpu1 1>; mbox-names = "tx", "rx"; }; }; zynqmp_ipi1 { compatible = "xlnx,zynqmp-ipi-mailbox"; interrupt-parent = <&gic>; interrupts = <0 33 4>; xlnx,ipi-id = <5>; #address-cells = <1>; #size-cells = <1>; ranges; /* APU<->RPU0 IPI mailbox controller */ ipi_mailbox_rpu0: mailbox@ff990600 { reg = <0xff3f0ac0 0x20>, <0xff3f0ae0 0x20>, <0xff3f0740 0x20>, <0xff3f0760 0x20>; reg-names = "local_request_region", "local_response_region", "remote_request_region", "remote_response_region"; #mbox-cells = <1>; xlnx,ipi-id = <3>; }; /* APU<->RPU1 IPI mailbox controller */ ipi_mailbox_rpu1: mailbox@ff990640 { reg = <0xff3f0b00 0x20>, <0xff3f0b20 0x20>, <0xff3f0940 0x20>, <0xff3f0960 0x20>; reg-names = "local_request_region", "local_response_region", "remote_request_region", "remote_response_region"; #mbox-cells = <1>; xlnx,ipi-id = <4>; }; }; };

RPU Firmware

  • In rsc_table.c, ensure RING_TX and RING_RX are set as FW_RSC_U32_ADDR_ANY, where FW_RSC_U32_ADDR_ANY = (-1). The kernel driver relies on this value to update the resource table with vring addresses.

  • In platform_info.c you can find the below address, size and offset assignment which should match the device tree description of shared buffers.

#ifndef SHARED_MEM_PA #if XPAR_CPU_ID == 0 #define SHARED_MEM_PA 0x3ED40000UL #else #define SHARED_MEM_PA 0x3EF40000UL #endif /* XPAR_CPU_ID */ #endif /* !SHARED_MEM_PA */ #ifndef SHARED_MEM_SIZE #define SHARED_MEM_SIZE 0x100000UL #endif /* !SHARED_MEM_SIZE */ #ifndef SHARED_BUF_OFFSET #define SHARED_BUF_OFFSET 0x8000UL #endif /* !SHARED_BUF_OFFSET */

RPMsg in Userspace

Figure: OpenAMP RPMsg Implementation in Linux Userspace

The OpenAMP library can also be used in Linux userspace. However, in this scenario, its important to match up the Linux userspace application, kernel device tree and RPU firmware with respect to the vring address locations and shared memory address.

Modifications for RPMsg in user space

Refer to the OpenAMP Application Build Process page for Building Linux Distribution using PetaLinux, for modifying Demo Tests Firmware, and for booting up and running demo tests in userspace. OpenAMP version specific modifications are covered below; for example, changes in the device tree, RPU firmware, Linux application, etc.

2022.1 Example

Device Tree

Device tree file location: <plnx-proj-root>/project-spec/meta-user/recipes-bsp/device-tree/files/system-user.dtsi

/include/ "system-conf.dtsi" / { reserved-memory { #address-cells = <2>; #size-cells = <2>; ranges; rproc_0_reserved: rproc@3ed000000 { no-map; reg = <0x0 0x3ed00000 0x0 0x1000000>; }; }; amba { compatible = "simple-bus"; u-boot,dm-pre-reloc; #address-cells = <0x2>; #size-cells = <0x2>; ranges; shm@3ed20000 { compatible = "shm_uio"; reg = <0x0 0x3ed20000 0x0 0x100000>; }; ipi@ff340000 { compatible = "ipi_uio"; reg = <0x0 0xff340000 0x0 0x1000>; interrupt-parent = <0x4>; interrupts = <0x0 0x1d 0x4>; }; }; };
/include/ "system-conf.dtsi" / { chosen { stdout-path = "serial0:115200"; }; }; / { reserved-memory { #address-cells = <2>; #size-cells = <2>; ranges; rproc_0_reserved: rproc@3ed000000 { no-map; reg = <0x0 0x3ed00000 0x0 0x1000000>; }; }; amba { compatible = "simple-bus"; u-boot,dm-pre-reloc; #address-cells = <0x2>; #size-cells = <0x2>; ranges; shm@3ed20000 { compatible = "shm_uio"; reg = <0x0 0x3ed20000 0x0 0x0100000>; }; ipi@ff360000 { compatible = "ipi_uio"; reg = <0x0 0xff360000 0x0 0x1000>; interrupt-parent = <&gic>; interrupts = <0 33 4>; }; }; };

Linux application

  • In platform_info.c you can find the below addresses, sizes and offset assignment which should match the device tree description of shared memory.

#ifndef SHM_DEV_NAME #define SHM_DEV_NAME "3ed20000.shm" /* shared device name */ #endif /* SHM_DEV_NAME */ #ifndef RSC_MEM_PA #define RSC_MEM_PA 0x3ED20000UL /* Resource table location */ #endif /* !RSC_MEM_PA */ #ifndef RSC_MEM_SIZE #define RSC_MEM_SIZE 0x2000UL /* Resource table size */ #endif /* !RSC_MEM_SIZE */ #ifndef VRING_MEM_PA #define VRING_MEM_PA 0x3ED40000UL /* Vring memory location */ #endif /* !VRING_MEM_PA */ #ifndef VRING_MEM_SIZE #define VRING_MEM_SIZE 0x8000UL /* Vring size */ #endif /* !VRING_MEM_SIZE */ #ifndef SHARED_BUF_PA #define SHARED_BUF_PA 0x3ED48000UL /* shared buffer location */ #endif /* !SHARED_BUF_PA */ #ifndef SHARED_BUF_SIZE #define SHARED_BUF_SIZE 0x40000UL /* shared buffer size */ #endif /* !SHARED_BUF_SIZE */

RPU Firmware

OpenAMP's implementation of RPMsg in userspace only allows for static vring entries so make sure to change it to the address as detailed below.

  • Update RING_TX from FW_RSC_U32_ADDR_ANY to 0x3ed40000 and RING_RX from FW_RSC_U32_ADDR_ANY to 0x3ed44000 in rsc_table.c

- #define RING_TX FW_RSC_U32_ADDR_ANY - #define RING_RX FW_RSC_U32_ADDR_ANY + #define RING_TX 0x3ed40000 + #define RING_RX 0x3ed44000
  • In platform_info.c you can find the below address, size and offset assignment which should match the device tree description of shared buffers.

#ifndef SHARED_MEM_PA #if XPAR_CPU_ID == 0 #define SHARED_MEM_PA 0x3ED40000UL #else #define SHARED_MEM_PA 0x3EF40000UL #endif /* XPAR_CPU_ID */ #endif /* !SHARED_MEM_PA */ #ifndef SHARED_MEM_SIZE #define SHARED_MEM_SIZE 0x100000UL #endif /* !SHARED_MEM_SIZE */ #ifndef SHARED_BUF_OFFSET #define SHARED_BUF_OFFSET 0x8000UL #endif /* !SHARED_BUF_OFFSET */

RPU firmware resource table location

  • In the linker script file lscript.ld at <plnx-proj-root>/components/yocto/workspace/sources/<recipe-name>/<recipe-name>/, the resource table location can be found at 0x3ED20000 which should align with RSC_MEM_PA from platform_info.c and also the kernel device tree shared memory location.

.resource_table 0x3ed20000 : { . = ALIGN(4); *(.resource_table) } > psu_ddr_S_AXI_BASEADDR

Related Links

© Copyright 2019 - 2022 Xilinx Inc. Privacy Policy