This document describes the base set of hardware required for OpenAMP to operate successfully as presented in Xilinx Vitis OpenAMP and libmetal template examples.

Table of Contents

Introduction

This document describes the base set of hardware required for OpenAMP to operate successfully as presented in Xilinx Vitis OpenAMP and libmetal template examples. It covers configurations for the RPU memory, shared memory for both the APU and RPU, generic interrupt controllers (GIC) and the inter-processor interconnect (IPI) interrupts. There are several examples of using Vitis, PetaLinux and OpenAMP, however this is not a tutorial for these tools. The reader is encouraged to refer to Xilinx user guides listed in the Related Links.

OpenAMP on Xilinx SoCs

Xilinx SoC products combine a set of heterogeneous hardware designs into one powerful and flexible platform that includes Arm Cortex-Ax, Cortex-R5 and Xilinx MicroBlaze processors. The OpenAMP project enables a distributed software architecture across this asymmetric multiprocessing platform (AMP).

There are two components of the OpenAMP framework:

1. Life Cycle Management (LCM) : enables a master processor to load, start and stop a firmware on a remote processor.

2. Inter-Processor Communication (IPC): interrupts, shared memory and message passing between the master and the remote processor.

Both the LCM and the IPC in the demos are using the remoteproc and the rpmsg drivers in the Linux kernel.  Xilinx SoCs based systems should provide access to a specific set of hardware resources in order to use the OpenAMP framework. Depending on the OpenAMP or libmetal demo this includes:

1. Processing cores; need at least two:

   1. application processing unit (APU)

   2. real-time processing unit (RPU) for Versal and ZynqMP

2. Memory for the processing system (PS): APU (master), RPU (remote) and shared memory:

   1. DDR

   2. Tightly-coupled memory (TCM)

   3. On-chip memory (OCM)

3. Interrupts:

   1. APU GIC interrupts

   2. RPU GIC interrupts

   3. IPI interrupts

The next section tries to generalize each hardware component as it is used in the OpenAMP, followed by specific example of configuring some demos with the default values and another section showing how to change the defaults.

Zynq UltraScale+ MPSoC and Versal

LCM: Memory for RPU Firmware

Most demos use the DDR or the Tightly Coupled Memory (TCM) for RPU firmware text and data depending on the linker script (e.g.: linker_remote.ld) while the stack, heap and the interrupt vector table are in the TCM. The RPU is configured to enable the TCM interfaces from reset with base address 0 for TCMA. The Linux remoteproc driver on the APU will preload the TCMA with the boot code while the RPU is halted. When the halt is removed the RPU starts fetching instructions from the reset vector address in the normal way. The RPU's firmware .vectors section is loaded into TCMA at address 0 (see linker script). It starts with a set of branch instructions called _vector_table and a few code blocks labeled _boot, init and a function Init_MPU(). When the remoteproc driver requests RPU start t he the execution begins at address 0 in TCMA with a jump to _boot which calls Init_MPU(). All this is fetched from and executed in the TCMA. At the end the _boot jumps to _startup which could be in the DDR. The _startup prepares the "C runtime" and calls main().

IPC: Shared Memory

The OpenAMP demos have a .resource_table ELF section as in this example linker_remote.ld. It has list of system resources required by the Linux remoteproc, e.g.: virtIO device data used by rpmsg, see rsc_table.h and rsc_table.c.

The default location for shared meory memory is the DDR.

SHM_DEV_NAME "3ed80000.shm"
IPI_DEV_NAME="ff340000.ipi"         (if Versal=)  "ff360000.ipi"
TTC_DEV_NAME="ff110000.timer"       (if Versal=)  "ff0e0000.ttc0"

/sys/bus/rpmsg/devices/virtio0.rpmsg-openamp-demo-channel.-1.0

IPC: Interrupts

The OpenAMP and Libmetal demos use the IPI interrupts to enable one processor (source agent) to interrupt another processor (destination agent). The communications process uses the IPI interrupt register structure, the system interrupt structure, and the IPI message buffers. On the RPU’s firmware side these are represented by the struct metal_device ipi_device and IPI_CHN_BITMASK in platform_info.c.

The demos need two IPI channels (and IPI masks) one for the APU and one for the RPU. These are selected from the set of seven reprogrammable IPIs. Other system users might be requesting IPI channels. Both the firmware and the Linux device tree should be checked to detect and avoid conflicts.

zynqmp_ipi1 {
    compatible = "xlnx,zynqmp-ipi-mailbox";
    interrupt-parent = <&gic>;
    interrupts = <0 29 4>;
    xlnx,ipi-id = <7>;
    #address-cells = <1>;
    #size-cells = <1>;
    ranges;

    /* APU<->RPU0 IPI mailbox controller */
    ipi_mailbox_rpu0: mailbox@ff90000 {
        reg = <0xff990600 0x20>,
              <0xff990620 0x20>,
              <0xff9900c0 0x20>,
              <0xff9900e0 0x20>;
        reg-names = "local_request_region",
                "local_response_region",
                "remote_request_region",
                "remote_response_region";
        #mbox-cells = <1>;
        xlnx,ipi-id = <1>;
    };
};

In Versal the channel names and numbers are different, see Versal-ACAP-TRM

zynqmp_ipi1 {
    compatible = "xlnx,zynqmp-ipi-mailbox";
    interrupt-parent = <&gic>;
    interrupts = <0 33 4>;
    xlnx,ipi-id = <5>;
    #address-cells = <1>;
    #size-cells = <1>;
    ranges;

    /* APU<->RPU0 IPI mailbox controller */
    ipi_mailbox_rpu0: mailbox@ff990600 {
        reg = <0xff3f0ac0 0x20>,
              <0xff3f0ae0 0x20>,
              <0xff3f0740 0x20>,
              <0xff3f0760 0x20>;
        reg-names = "local_request_region",
                "local_response_region",
                "remote_request_region",
                "remote_response_region";
        #mbox-cells = <1>;
        xlnx,ipi-id = <3>;
    };
};

Build and Run Demos

Build RPU firmware

Step 1. Launch Vitis IDE and select a workspace directory with enough free space and fast access. Local disk is prefered as some NFS can be slow.

Step 2. Create a new application project using "File > New Application Project".  This will start a wizard that will guide you through the process.

• Create a new platform from hardware description XSA file. Select from the included XSA’s or use browse and import your own XSA.

• Specify the application project name. E.g.: lm_amp and select psu_cortexr5_0 as the target processor from the list.

• Select template for your project, e.g. "Libmetal AMP Application" and click "Finish":

Processor for the RPU firmware project	Domain: FreeRTOS or generic	Libmetal AMP Demo, OpenAMP echo-test,..

Configuration of some hardware blocks used in this demo must match between the RPU’s firmware and APU’s Linux kernel, device tree and the user application. Please, see the "Hardware specification" tab to examine the address maps for APU and RPU processors:

1. Memory: DDR and Tightly-coupled memory (TCM)

2. Interrupts: PS-to-PS interrupts and Inter-processor interrupts (IPI)

3. Timers: Triple Timer Counters (TTC)

Hardware specification: Memory Maps	IPI:  psu_ipi_1 example	Linker Script: lscript.ld

Firmware memory map: Select lscript.ld file in the source Explorer. The summary view shows the memory regions for this application:

1. The processor is configured to enable the TCM interfaces from reset with base address 0x0 for TCMA and 0x20000 for TCMB. This demo does not use TCMB.

2. The stack, heap and the interrupt vector table are in psu_r5_atcm_MEM_0, TCMA.

3. The text, read-only, initialized, uninitialized data sections, etc. are in psu_r5_ddr_0_MEM_0 starting at 0x3ED00000 with LENGTH=0x80000.

4. OpenAMP demos have .resource_table ELF section. It has list of system resources required by the Linux remoteproc, e.g.: virtIO device data.

5. The Linux remoteproc driver on the APU will preload the TCMA with the boot code while halting the RPU via nCPUHALTm pin. When the nCPUHALTm pin is deasserted, the processor starts fetching instructions from the reset vector address in the normal way. The FreeRTOS _vector_table is loaded into TCMA at address 0x0 followed by _boot and Init_MPU().

6. The execution starts at address 0x0 in TCMA via a jump to _boot which calls Init_MPU(). All this is fetched from and executed in the TCMA. At the end the _boot jumps to _startup in the DDR. The _startup prepares the "C runtime" and calls main(). Starting from _startup the RPU fetches its instructions from the DDR while its stack and heap are in the TCMA.

Step 3. Build the application project: click lm_amp or lm_amp_system in the source Explorer and select "Project > Build Project". The build produces an ARM CortexR5 binary in the Debug (or Release) directory called lm_amp.elf. In this example this binary will be loaded to execute on the RPU (r5_0) via Xilinx remoteproc Linux driver.

Build PetaLinux

Create a PetaLinux project using the BSP where we took the XSA file for the RPU firmware:

1. Code Block
   source <path_to_petalinux_settings> petalinux-create -t project -s <path_to_petalinux_project_bsp>

2. Depending on the selected demo you have to enable the following kernel config options (see: project-spec/meta-user/recipes-kernel/linux/linux-xlnx_%.bbappend and the kernel config file (e.g.: defconfig) it includes):

   Code Block
   CONFIG_UIO_PDRV_GENIRQ=m CONFIG_RPMSG_CHAR=m CONFIG_RPMSG_VIRTIO=m CONFIG_RPMSG=m CONFIG_VIRTIO=m CONFIG_REMOTEPROC=y CONFIG_ZYNQMP_R5_REMOTEPROC=m CONFIG_SPARSEMEM_VMEMMAP=y

3. Copy one of the following device tree overlay files into <plnx-proj>/project-spec/meta-user/recipes-bsp/device-tree/files/system-user.dtsi depending on the demo selected for the RPU firmware:

• The <reg> property of the rproc_0_reserved node has the same memory region as the one selected in the lscript.ld in the Vitis application. This allows Xilinx Linux remoteproc driver to take our lm_amp CortexR5 binary (from /lib/firmware), parse its ELF headers and set up the segments in the DDR where the RPU can read and execute the text and access the shared data.

• The <reg> property of shm0 node is included for Linux UIO to let user space demo to talk to our firmware application lm_amp on the RPU.

• Cross-reference for Shared memory: the Linux device tree (system-user.dtsi), RPU linker script (lscript.ld), metal_dev_table from freertos/zynqmp_r5/zynqmp_amp_demo/sys_init.c

• Cross-reference for the IPI. The ipi0 node for Linux UIO uses IPI Channel 7 with base address 0xFF34_0000, while the RPU uses its default IPI Channel-1 with BASE_ADDRESS 0xFF31_0000.

• The RPU code uses metal_dev_table from sys_init.c, common.h, CMakeLists.txt

• Cross-reference for the TTCs, mapped via UIO for the Linux demo app

Build the PetaLinux project and package the BOOT.bin for Versal

      petalinux-build
      petalinux-package --boot --plm --psmfw --u-boot --dtb

For other architectures please see UG1144 PetaLinux Tools.

Changing the Defaults

LCM: TCM for RPU Firmware

Some OpenAMP demos are configured to have their text segment in the DDR memory. This allows larger firmware to run on RPU, but it might be slower compared to running from the TCM. According to Cortex-R5 Technical Reference Manual: “A ATCM typically holds interrupt or exception code that must be accessed at high speed, without any potential delay resulting from a cache miss.  A BTCM typically holds a block of data for intensive processing, such as audio or video processing.” Another reason to run from the TCM is power usage, i.e. if the APU decides  to suspend the DDR can be powered down or put into retention mode to save power.

The linker script controls the location of the ELF sections of the RPU firmware. For example, to run with .vectors, .text and .note.gnu.build-id  in psu_r5_atcm_MEM_0, and  with everything else in psu_r5_btcm_MEM_0, use the following linker script:

_STACK_SIZE = DEFINED(_STACK_SIZE) ? _STACK_SIZE : 0x2000;
_HEAP_SIZE = DEFINED(_HEAP_SIZE) ? _HEAP_SIZE : 0x4000;

_ABORT_STACK_SIZE = DEFINED(_ABORT_STACK_SIZE) ? _ABORT_STACK_SIZE : 1024;
_SUPERVISOR_STACK_SIZE = DEFINED(_SUPERVISOR_STACK_SIZE) ? _SUPERVISOR_STACK_SIZE : 2048;
_IRQ_STACK_SIZE = DEFINED(_IRQ_STACK_SIZE) ? _IRQ_STACK_SIZE : 1024;
_FIQ_STACK_SIZE = DEFINED(_FIQ_STACK_SIZE) ? _FIQ_STACK_SIZE : 1024;
_UNDEF_STACK_SIZE = DEFINED(_UNDEF_STACK_SIZE) ? _UNDEF_STACK_SIZE : 1024;

/* Define Memories in the system */

MEMORY
{
   psu_r5_atcm_MEM_0 : ORIGIN = 0x0, LENGTH = 0x10000
   psu_r5_btcm_MEM_0 : ORIGIN = 0x20000, LENGTH = 0x10000
   psu_r5_ddr_0_MEM_0 : ORIGIN = 0x3ed00000, LENGTH = 0x80000
}

/* Specify the default entry point to the program */

ENTRY(_boot)

/* Define the sections, and where they are mapped in memory */

SECTIONS
{
.vectors : {
   KEEP (*(.vectors))
   *(.boot)
} > psu_r5_atcm_MEM_0

.text : {
   *(.text)
   *(.text.*)
   *(.gnu.linkonce.t.*)
   *(.plt)
   *(.gnu_warning)
   *(.gcc_execpt_table)
   *(.glue_7)
   *(.glue_7t)
   *(.vfp11_veneer)
   *(.ARM.extab)
   *(.gnu.linkonce.armextab.*)
   *(.note.gnu.build-id)
} > psu_r5_atcm_MEM_0

.note.gnu.build-id : {
   KEEP (*(.note.gnu.build-id))
} > psu_r5_atcm_MEM_0

.init : {
   KEEP (*(.init))
} > psu_r5_btcm_MEM_0

.fini : {
   KEEP (*(.fini))
} > psu_r5_btcm_MEM_0

.interp : {
   KEEP (*(.interp))
} > psu_r5_btcm_MEM_0

.note-ABI-tag : {
   KEEP (*(.note-ABI-tag))
} > psu_r5_btcm_MEM_0

.rodata : {
   __rodata_start = .;
   *(.rodata)
   *(.rodata.*)
   *(.gnu.linkonce.r.*)
   __rodata_end = .;
} > psu_r5_btcm_MEM_0

.rodata1 : {
   __rodata1_start = .;
   *(.rodata1)
   *(.rodata1.*)
   __rodata1_end = .;
} > psu_r5_btcm_MEM_0

.sdata2 : {
   __sdata2_start = .;
   *(.sdata2)
   *(.sdata2.*)
   *(.gnu.linkonce.s2.*)
   __sdata2_end = .;
} > psu_r5_btcm_MEM_0

.sbss2 : {
   __sbss2_start = .;
   *(.sbss2)
   *(.sbss2.*)
   *(.gnu.linkonce.sb2.*)
   __sbss2_end = .;
} > psu_r5_btcm_MEM_0

.data : {
   __data_start = .;
   *(.data)
   *(.data.*)
   *(.gnu.linkonce.d.*)
   *(.jcr)
   *(.got)
   *(.got.plt)
   __data_end = .;
} > psu_r5_btcm_MEM_0

.data1 : {
   __data1_start = .;
   *(.data1)
   *(.data1.*)
   __data1_end = .;
} > psu_r5_btcm_MEM_0

.got : {
   *(.got)
} > psu_r5_btcm_MEM_0

.ctors : {
   __CTOR_LIST__ = .;
   ___CTORS_LIST___ = .;
   KEEP (*crtbegin.o(.ctors))
   KEEP (*(EXCLUDE_FILE(*crtend.o) .ctors))
   KEEP (*(SORT(.ctors.*)))
   KEEP (*(.ctors))
   __CTOR_END__ = .;
   ___CTORS_END___ = .;
} > psu_r5_btcm_MEM_0

.dtors : {
   __DTOR_LIST__ = .;
   ___DTORS_LIST___ = .;
   KEEP (*crtbegin.o(.dtors))
   KEEP (*(EXCLUDE_FILE(*crtend.o) .dtors))
   KEEP (*(SORT(.dtors.*)))
   KEEP (*(.dtors))
   __DTOR_END__ = .;
   ___DTORS_END___ = .;
} > psu_r5_btcm_MEM_0

.fixup : {
   __fixup_start = .;
   *(.fixup)
   __fixup_end = .;
} > psu_r5_btcm_MEM_0

.eh_frame : {
   *(.eh_frame)
} > psu_r5_btcm_MEM_0

.eh_framehdr : {
   __eh_framehdr_start = .;
   *(.eh_framehdr)
   __eh_framehdr_end = .;
} > psu_r5_btcm_MEM_0

.gcc_except_table : {
   *(.gcc_except_table)
} > psu_r5_btcm_MEM_0

.mmu_tbl (ALIGN(16384)) : {
   __mmu_tbl_start = .;
   *(.mmu_tbl)
   __mmu_tbl_end = .;
} > psu_r5_btcm_MEM_0

.ARM.exidx : {
   __exidx_start = .;
   *(.ARM.exidx*)
   *(.gnu.linkonce.armexidix.*.*)
   __exidx_end = .;
} > psu_r5_btcm_MEM_0

.preinit_array : {
   __preinit_array_start = .;
   KEEP (*(SORT(.preinit_array.*)))
   KEEP (*(.preinit_array))
   __preinit_array_end = .;
} > psu_r5_btcm_MEM_0

.init_array : {
   __init_array_start = .;
   KEEP (*(SORT(.init_array.*)))
   KEEP (*(.init_array))
   __init_array_end = .;
} > psu_r5_btcm_MEM_0

.fini_array : {
   __fini_array_start = .;
   KEEP (*(SORT(.fini_array.*)))
   KEEP (*(.fini_array))
   __fini_array_end = .;
} > psu_r5_btcm_MEM_0

.ARM.attributes : {
   __ARM.attributes_start = .;
   *(.ARM.attributes)
   __ARM.attributes_end = .;
} > psu_r5_btcm_MEM_0

.sdata : {
   __sdata_start = .;
   *(.sdata)
   *(.sdata.*)
   *(.gnu.linkonce.s.*)
   __sdata_end = .;
} > psu_r5_btcm_MEM_0

.sbss (NOLOAD) : {
   __sbss_start = .;
   *(.sbss)
   *(.sbss.*)
   *(.gnu.linkonce.sb.*)
   __sbss_end = .;
} > psu_r5_btcm_MEM_0

.tdata : {
   __tdata_start = .;
   *(.tdata)
   *(.tdata.*)
   *(.gnu.linkonce.td.*)
   __tdata_end = .;
} > psu_r5_btcm_MEM_0

.tbss : {
   __tbss_start = .;
   *(.tbss)
   *(.tbss.*)
   *(.gnu.linkonce.tb.*)
   __tbss_end = .;
} > psu_r5_btcm_MEM_0

.bss (NOLOAD) : {
   . = ALIGN(4);
   __bss_start__ = .;
   *(.bss)
   *(.bss.*)
   *(.gnu.linkonce.b.*)
   *(COMMON)
   . = ALIGN(4);
   __bss_end__ = .;
} > psu_r5_btcm_MEM_0

_SDA_BASE_ = __sdata_start + ((__sbss_end - __sdata_start) / 2 );

_SDA2_BASE_ = __sdata2_start + ((__sbss2_end - __sdata2_start) / 2 );

/* Generate Stack and Heap definitions */

.heap (NOLOAD) : {
   . = ALIGN(16);
   _heap = .;
   HeapBase = .;
   _heap_start = .;
   . += _HEAP_SIZE;
   _heap_end = .;
   HeapLimit = .;
} > psu_r5_btcm_MEM_0

.stack (NOLOAD) : {
   . = ALIGN(16);
   _stack_end = .;
   . += _STACK_SIZE;
   _stack = .;
   __stack = _stack;
   . = ALIGN(16);
   _irq_stack_end = .;
   . += _IRQ_STACK_SIZE;
   __irq_stack = .;
   _supervisor_stack_end = .;
   . += _SUPERVISOR_STACK_SIZE;
   . = ALIGN(16);
   __supervisor_stack = .;
   _abort_stack_end = .;
   . += _ABORT_STACK_SIZE;
   . = ALIGN(16);
   __abort_stack = .;
   _fiq_stack_end = .;
   . += _FIQ_STACK_SIZE;
   . = ALIGN(16);
   __fiq_stack = .;
   _undef_stack_end = .;
   . += _UNDEF_STACK_SIZE;
   . = ALIGN(16);
   __undef_stack = .;
} > psu_r5_btcm_MEM_0

_end = .;
}

IPC: Shared Memory

Shared memory parameter changes need to be coordinated between the RPU firmware code and the APU's Linux device tree as shown in this table:

IPC: IPI Channels

Selecting a different IPI channel also requires coordinating changes in the RPU firmware and the APU's Linux device tree:

Please, see the IPI Interrupt Channel Architecture:

Additional Information

Timers for the Libmetal AMP Demo

The timers are used in the "libmetal AMP demo" and are not needed for any of the OpenAMP demos. ZynqMP and Versal have four Triple Timer Counters (TTC[0-3]). The code in the demo uses TTC0 by default. However any of the four TTCs can be used.

For example, what code changes are needed to use a different TTC  in zynqmp_amp_demo? Find  the base address of TTC1 in ug1085 zu+ TRM (it's 0xFF12_0000). Unfortunately,  some some values for TTC0 are hard coded and not easy to changerequire source code changes.

Zynq-7000

Zynq-7000 has two Arm Cortex-A9 MPCore CPUs. Our demos use one core A9_0 to run Linux master and A9_1 to run a baremetal or FreeRTOS remote firmware.

Related Links

Anchor
Related-Links Related-Links

1. UG1186 - Libmetal and OpenAMP User Guide (v2020.2)

2. UG1416 - Vitis Unified Software Development Platform (v2020.2)

3. UG1085 - Zynq UltraScale+ Device Technical Reference Manual (v2.2)

4. UG1087 - Zynq UltraScale+ MPSoC Register Reference

5. UG1304 - Versal ACAP System Software Developers Guide (v2020.2)

6. Versal ACAP Technical Reference Manual

7. Versal ACAP Register Reference