The reference design for Zynq MPSoC isolation mechanisms is XAPP1320, where all of the different features are discussed extensively using a simple use case where the APU and RPU are running a baremetal application.

The following collection of example designs demonstrate how to use the isolation configuration wizard to support more complex use cases where the APU is running a PetaLinux image while RPU is either not used, used in lockstep, or used in split mode running a baremetal application.

Introduction

The isolation configuration wizard in Vivado is the configuration tool provided to partition the SoC device into different subsystem. The defined configuration is used to generate the relevant software code that will be in charge of configuring some of protection units present in the system (see System Protection Units in UG1085) as well as the configuration object provided to the PMU firmware (see Configuration object in UG11347).

The following example designs are based on the default ZCU102 board configuration settings in Vivado, where only some of the high speed peripherals have been disabled to simplify the configuration settings.

Subsystems

A subsystem is a set of master devices (CPU or DMA capable peripherals, i.e. QSPI) and the associate resources that they will make use of (either a memory range, a peripheral, or a control register). A slave device might be defined in multiple subsystems when it is meant to be accessed by different masters (i.e. both the APU and the RPU).

Any slave on a subsystem without a defined master can be access by ALL masters

Isolation Configuration GUI

Default policy settings

The isolation configuration menu for Zynq MPSoC devices is only available when the Advanced Mode is enabled in the PCW window and is disabled by default. Once enabled, there are two additional checkboxes at the top of the window that are crucially important in the isolation flow that determines the default access policy in the SoC.

Enable Secure Debug: Enables access to any slave defined in the subsystems by the DAP master. This means that the debugger can be used to load the software in the device or to debug it once running in the target.
Lock Unused Memory: Disables access to any slave (memory/peripheral/register) not defined within the subsystems. Once enabled, the following rule will be applied to the slaves:
1. Slave not defined in any subsystem: Cannot be accessed by any master
2. Slave defined in a single subsystem: Cannot be accessed by masters on any other subsystem
3. Slave defined in multiple subsystems: Can be accessed by masters that have the slave present on their subsystems

For a basic isolation system, the "Lock Unused Memory" option should deselected, such that the configuration menu would be used to focus on those resources that require protection to be enabled.

Base isolation configuration

There are different ways to configure the isolation configuration, mostly driven by the application requirements of the customer.

Vivado provides a basic configuration that is intended to protect the access to several configuration registers that can be accessed only by the PMU Firmware (i..e XMPU/XPPU) or any secure master.

Known issues and limitations

The configuration menu does not allow selection of a different TZ or Access Setting for a specific slave. That means that a slave node (memory/peripheral/register) cannot be accessed with different access permissions (i.e. APU R and RPU R/W) or TZ permissions (i.e. APU non-secure and PMU secure).

Example Subsystems

APU Linux only

The following subsystem example is an implementation of an APU cluster running a PetaLinux image, where the isolation features are used to protect the memory area used by the secure software, FSBL and ARM Trusted Firmware (TF-A). This way, neither the Linux kernel or the application software running on top would be able to modify any of the secure software at runtime.

Required changes

Subsystem	Description

Subsystem	Description
APU Secure Subsystem	Add OCM memory: Required to execute FSBL (see here) and TF-A (see here).
PMU Firmware	Add OCM memory: Required to load FSBL on warm restart (see here) and to get PM configuration object from FSBL (see here)

Example configuration

APU Linux and RPU split non-secure

The following subsystem example is an implementation of an APU cluster running a PetaLinux image and the RPU cluster running a baremetal application in split mode with non-secure configuration. The isolation features are used to ensure that secure software memory areas are protected, and that memory areas assigned to each CPU are exclusively accessed by the specific master of each subsystem.

A small area of DDR memory is assigned to each RPU in order to provide an extended memory area beside the TCM memory assigned to each core.

(0x00000000 - 0x7FCFFFFF): 2045MB for APU to run the Linux image
(0x7FD00000 - 0x7FDFFFFF): 1MB for RPU0 as a buffer
(0x7FE00000 - 0x7FEFFFFF): 1MB for RPU1 as a buffer
(0x7FF00000 - 0x7FFFFFFF): 1MB for PMU firmware and secure masters to store a copy of the FSBL

Required changes

Subsystem	Description

Subsystem	Description
APU Subsystem	Add masters: Peripherals used by the APU that include the master interface Add peripherals: Peripherals used by the APU Add DDR memory: Used by the Linux image and U-Boot Add RPU control register: Used by the FSBL to release RPU reset during handoff process
PMU Firmware	Add CSU master: Used by the PMU to move data around Add DDR memory: Used by the Linux image to share data with the PMU Firmware Modify RPU control register: Required NonSecure attribute due to non-secure RPU master access requirement
RPU0 Subsystem	Add TCM memory: RPU0 ATCM and BTCM Add DDR memory: 1MB DDR buffer Add RPU control register: Used by the boot code to determine if running split or lockstep
RPU1 Subsystem	Add TCM memory: RPU1 ATCM and BTCM Add DDR memory: 1MB DDR buffer Add RPU control register: Used by the boot code to determine if running split or lockstep
Secure Subsystem	Add OCM memory: Used by secure APU and PMU Firmware Add DDR memory: 1MB DDR for storing a copy of the FSBL Remove RPU control register: Configured on each specific subsystem as a non-secure slave

Example configuration

APU Linux and RPU lockstep secure

The following subsystem example is an implementation of an APU cluster running a PetaLinux image and the RPU cluster running a baremetal application in lockstep mode as a secure master. Additionally, the FSBL is executed in the RPU, reflecting that it is the main processor of the system in charge of the whole boot and configuration steps. The isolation features are used to ensure that secure software memory areas are protected, and that memory areas assigned to each CPU are exclusively accessed by the specific master of each subsystem.

A small area of DDR memory is assigned to the RPU in order to provide an extended memory area beside the TCM memory.

(0x00000000 - 0x7FDFFFFF): 2046MB for APU to run the Linux image
(0x7FE00000 - 0x7FEFFFFF): 1MB for RPU as a buffer
(0x7FF00000 - 0x7FFFFFFF): 1MB for PMU firmware and secure masters to store a copy of the FSBL

Required changes

Subsystem	Description

Subsystem	Description
APU Subsystem	Add masters: Peripherals used by the APU that include master interface Add peripherals: Periphreal used by the APU Add DDR memory: Used by the Linux image and U-Boot
PMU Firmware	Add CSU master: Used by the PMU to move data around Add DDR memory: Used by the Linux image to share data with the PMU Firmware
RPU Subsystem	Add TCM memory: Full TCM memory range Add DDR memory: 1MB DDR buffer Add peripherals: Peripherals used by the RPU application and FSBL Add RPU control register: Used by the boot code to determine if running split or lockstep
Secure Subsystem	Add OCM: Used by secure RPU and PMU Firmware Add DDR memory: Both RPU reserved range and the PMU reserved range Remove RPU control register: Configured on the RPU subsystem

Example configuration

Software

FSBL

The FSBL is responsible for configuring the isolation engines (XMPU/XPPU) after loading the boot image partitions and before releasing the required processors or performing the handoff to the application code. It does not require any specific change, but FSBL debug verbosity is increased using the FSBL_DEBUG_INFO symbol at compilation time (due to size constraints, some other features are required to be excluded, i.e. FSBL_SECURE_EXCLUDE).

The following code shows how to patch the FSBL in a PetaLinux project as described in this wiki page.

$ cat project-spec/meta-user/recipes-bsp/embeddedsw/fsbl-firmware_%.bbappend 
YAML_COMPILER_FLAGS:append = " -DFSBL_DEBUG_INFO"
YAML_COMPILER_FLAGS:append = " -DFSBL_SECURE_EXCLUDE"

The same thing can be accomplished modifying the FSBL source code files (xfsbl_config.h) if the FSBL is built in Vitis.

#ifndef FSBL_DEBUG_INFO_VAL
#define FSBL_DEBUG_INFO_VAL         (1U)
#endif

...

#ifndef FSBL_SECURE_EXCLUDE_VAL
#define FSBL_SECURE_EXCLUDE_VAL			(1U)
#endif

PMU Firmware

The PMU firmware is responsible for monitoring system errors and of these three examples, it is the only one with access permission to access the isolation engines (XMPU/XPPU) after the initial configuration. The PMU firmware does not require any specific change to make the isolation work, but verbosity has been enabled specifically for XMPU/XPPU using the XPU_INTR_DEBUG_PRINT_ENABLE symbol at compilation time (this option has some other dependencies, XPFW_DEBUG_DETAILED and ENABLE_EM).

See this Wiki page for further information about PMU Firmware build flags.

The following code shows how to patch the PMU Firmware in a PetaLinux project as described in the following Wiki page.

$ cat project-spec/meta-user/recipes-bsp/pmu-firmware/pmu-firmware_%.bbappend 
YAML_COMPILER_FLAGS:append = " -DENABLE_EM"
YAML_COMPILER_FLAGS:append = " -DXPFW_DEBUG_DETAILED"
YAML_COMPILER_FLAGS:append = " -DXPU_INTR_DEBUG_PRINT_ENABLE"

The same thing can be accomplished by modifying the PMU Firmware source code files (xpfw_config.h) if the PMU Firmware is built in Vitis.

#ifndef XPFW_DEBUG_DETAILED_VAL
#define XPFW_DEBUG_DETAILED_VAL				(1U)
#endif

...

#ifndef ENABLE_EM_VAL
#define	ENABLE_EM_VAL						(1U)
#endif

...

#ifndef XPU_INTR_DEBUG_PRINT_ENABLE_VAL
#define	XPU_INTR_DEBUG_PRINT_ENABLE_VAL		(1U)
#endif

U-Boot

U-Boot is the software responsible for loading the Linux image and is by default compiled to be placed and executed from DDR memory. Although the example boot sequence might not be using OCM and TCM memories explicitly, the default configuration maps both memory regions in the MMU table as normal memory (see here). This can lead the processor to perform speculative accesses to the mentioned memory even when there is no explicit access in the code or from the U-Boot prompt. In order to prevent faulty transactions from happening in the boot process, the option has to be disabled in the U-Boot configuration.

The following line can be appended to the bsp.cfg file in the u-boot folder of the meta-user layer (created by the PetaLinux BSP).

$ tail -1 project-spec/meta-user/recipes-bsp/u-boot/files/bsp.cfg 
# CONFIG_DEFINE_TCM_OCM_MMAP is not set

RPU

The RPU applications are simple baremetal applications that will perform memory access operations to pre-determined memory addresses, grouped as allowed addresses and protected addresses. Additionally, a heartbeat print message has been introduced to the end of the application to ensure it keeps running properly once the test has finished.

In order to ensure that all of the printing of the test application is displayed properly, the application will not start performing any memory access until a signal is received from the APU running Linux using the corresponding IPI (Inter-Processor Interrupt) signal. Note that interrupts have not been enabled, instead a polling loop is used on the RPU application to monitor the incoming signal.

Lockstep RPU application source code:

UINTPTR allowedAddr[] = {0xFFE0FFFC, 0xFFE2FFFC, 0x7FE00000};
UINTPTR protectedAddr[] = {0x00040000};

void DataAbortHandler(void *CallBackRef)
{
    /* Do nothing */
}

int main(void) {

    int idx = 0;
    Xil_DCacheDisable();

    /* Register abort handler to check write operation failure */
    Xil_ExceptionRegisterHandler(XIL_EXCEPTION_ID_DATA_ABORT_INT, &DataAbortHandler, NULL);

    /* Wait signal from APU through IPI */
    while(Xil_In32(0x00FF310010) != 0x1);

    /* Print RPU0 test application */
    xil_printf("============================================================\r\n");
    xil_printf("RPU - Test Application\r\n\n");

    xil_printf("RPU - Write to allowed memory areas\r\n");
    xil_printf("--------------\r\n");
    for(idx=0; idx < sizeof(allowedAddr) / sizeof(*allowedAddr); idx++) {
        xil_printf("RPU - Write to 0x%0X\r\n", allowedAddr[idx]);
        usleep(1000);
        Xil_Out32(allowedAddr[idx], 0xDEADBEEF);
    }

    xil_printf("RPU - Write to protected memory areas\r\n");
    xil_printf("--------------\r\n");
    for(uint32_t idx=0; idx < sizeof(protectedAddr) / sizeof(*protectedAddr); idx++) {
        xil_printf("RPU - Write to 0x%0X\r\n", protectedAddr[idx]);
        usleep(1000);
        Xil_Out32(protectedAddr[idx], 0xDEADBEEF);
        sleep(1);
    }

    /* Signal APU through IPI */
    Xil_Out32(0x00FF310000, 0x1);

    /* Loop forever */
    while(1) {
        sleep(20);
        xil_printf("RPU - Hearbeat\r\n");
    }

    return -1;
}

Example Design sources

This example design has been tested using a ZCU102 board and the Vivado/Vitis/PetaLinux 2024.1 release.

Example Design validation

APU Linux only

/ # devmem 0xFFFC0000
============================================================
EM: XMPU/XPPU violation occurred (ErrorId: 16)
EM: XMPU OCM Security violation occurred
EM: Address of poisoned operation: 0xFFFC0XXX
EM: Master Device of poisoned operation: APU
EM: Poison register: 0x100000
============================================================
Bus error

APU Linux and RPU split non-secure

/ # devmem 0xFFE9FFFC
============================================================
EM: XMPU/XPPU violation occurred (ErrorId: 16)
EM: XPPU Master ID access violation
EM: Address of poisoned operation: 0xFFE9FFFC
EM: Master Device of poisoned operation: APU
EM: Poison register : 0xFF9C0
============================================================
Bus error
/ # devmem 0xFFE0FFFC
============================================================
EM: XMPU/XPPU violation occurred (ErrorId: 16)
EM: XPPU Master ID access violation
EM: Address of poisoned operation: 0xFFE0FFFC
EM: Master Device of poisoned operation: APU
EM: Poison register : 0xFF9C0
============================================================
Bus error
/ # devmem 0xFFFC0000
============================================================
EM: XMPU/XPPU violation occurred (ErrorId: 16)
EM: XMPU OCM Security violation occurred
EM: Address of poisoned operation: 0xFFFC0XXX
EM: Master Device of poisoned operation: APU
EM: Poison register: 0x100000
============================================================
Bus error
/ # devmem 0x7FD00000
============================================================
EM: XMPU/XPPU violation occurred (ErrorId: 16)
EM: XMPU DDR1 Security violation occurred
EM: Address of poisoned operation: 0x7FD00XXX
EM: Master Device of poisoned operation: APU
EM: Poison register: 0x100000
============================================================
Bus error
/ # devmem 0x7FE00000
============================================================
EM: XMPU/XPPU violation occurred (ErrorId: 16)
EM: XMPU DDR1 Security violation occurred
EM: Address of poisoned operation: 0x7FE00XXX
EM: Master Device of poisoned operation: APU
EM: Poison register: 0x100000
============================================================
Bus error

/ # devmem 0xFF300000 32 0x100
============================================================
RPU0 - Test Application

RPU0 - Write to allowed memory areas
--------------
RPU0 - Write to 0xFFE0FFFC
RPU0 - Write to 0xFFE2FFFC
RPU0 - Write to 0x7FD00000
RPU0 - Write to protected memory areas
--------------
RPU0 - Write to 0xFFE9FFFC
============================================================
EM: XMPU/XPPU violation occurred (ErrorId: 16)
EM: XPPU Master ID access violation
EM: Address of poisoned operation: 0xFFE9FFFC
EM: Master Device of poisoned operation: RPU0
EM: Poison register : 0xFF9C0
============================================================
/ # RPU0 - Write to 0xFFEBFFFC
============================================================
EM: XMPU/XPPU violation occurred (ErrorId: 16)
EM: XPPU Master ID access violation
EM: Address of poisoned operation: 0xFFEBFFFC
EM: Master Device of poisoned operation: RPU0
EM: Poison register : 0xFF9C0
============================================================
RPU0 - Write to 0x40000
============================================================
EM: XMPU/XPPU violation occurred (ErrorId: 16)
EM: XMPU DDR0 Security violation occurred
EM: Address of poisoned operation: 0x40XXX
EM: Master Device of poisoned operation: RPU0
EM: Poison register: 0x100000
============================================================
RPU0 - Write to 0x7FE00000
============================================================
EM: XMPU/XPPU violation occurred (ErrorId: 16)
EM: XMPU DDR0 Security violation occurred
EM: Address of poisoned operation: 0x7FE00XXX
EM: Master Device of poisoned operation: RPU0
EM: Poison register: 0x100000
============================================================
RPU0 - Write to 0x7FF00000
============================================================
EM: XMPU/XPPU violation occurred (ErrorId: 16)
EM: XMPU DDR0 Security violation occurred
EM: Address of poisoned operation: 0x7FF00XXX
EM: Master Device of poisoned operation: RPU0
EM: Poison register: 0x100000
============================================================
RPU0 - Hearbeat

/ # devmem 0xFF300000 32 0x200
============================================================
RPU1 - Test Application

RPU1 - Write to allowed memory areas
--------------
RPU1 - Write to 0xFFE9FFFC
RPU1 - Write to 0xFFEBFFFC
RPU1 - Write to 0x7FE00000
RPU1 - Write to protected memory areas
--------------
RPU1 - Write to 0xFFE0FFFC
============================================================
EM: XMPU/XPPU violation occurred (ErrorId: 16)
EM: XPPU Master ID access violation
EM: Address of poisoned operation: 0xFFE0FFFC
EM: Master Device of poisoned operation: RPU1
EM: Poison register : 0xFF9C0
============================================================
/ # RPU1 - Write to 0xFFE2FFFC
============================================================
EM: XMPU/XPPU violation occurred (ErrorId: 16)
EM: XPPU Master ID access violation
EM: Address of poisoned operation: 0xFFE2FFFC
EM: Master Device of poisoned operation: RPU1
EM: Poison register : 0xFF9C0
============================================================
RPU1 - Write to 0x40000
============================================================
EM: XMPU/XPPU violation occurred (ErrorId: 16)
EM: XMPU DDR0 Security violation occurred
EM: Address of poisoned operation: 0x40XXX
EM: Master Device of poisoned operation: RPU1
EM: Poison register: 0x100000
============================================================
RPU1 - Write to 0x7FD00000
============================================================
EM: XMPU/XPPU violation occurred (ErrorId: 16)
EM: XMPU DDR0 Security violation occurred
EM: Address of poisoned operation: 0x7FD00XXX
EM: Master Device of poisoned operation: RPU1
EM: Poison register: 0x100000
============================================================
RPU1 - Write to 0x7FF00000
============================================================
EM: XMPU/XPPU violation occurred (ErrorId: 16)
EM: XMPU DDR0 Security violation occurred
EM: Address of poisoned operation: 0x7FF00XXX
EM: Master Device of poisoned operation: RPU1
EM: Poison register: 0x100000
============================================================
RPU1 - Test Application

APU Linux and RPU lockstep secure

# devmem 0xFFE00000
============================================================
EM: XMPU/XPPU violation occurred (ErrorId: 16)
EM: Address of poisoned operation: 0xFFE00000
EM: Master Device of poisoned operation: APU
EM: Poison register : 0xFF9C0
============================================================
Bus error
/ # devmem 0x7FE00000
============================================================
EM: XMPU/XPPU violation occurred (ErrorId: 16)
EM: XMPU DDR1 Security violation occurred
EM: Address of poisoned operation: 0x7FE00XXX
EM: Master Device of poisoned operation: APU
EM: Poison register: 0x100000
============================================================
Bus error
/ # devmem 0xFFFC0000
============================================================
EM: XMPU/XPPU violation occurred (ErrorId: 16)
EM: XMPU OCM Security violation occurred
EM: Address of poisoned operation: 0xFFFC0XXX
EM: Master Device of poisoned operation: APU
EM: Poison register: 0x100000
============================================================
Bus error

# devmem 0xFF300000 32 0x100
============================================================
RPU - Test Application

RPU - Write to allowed memory areas
--------------
RPU - Write to 0xFFE0FFFC
RPU - Write to 0xFFE2FFFC
RPU - Write to 0x7FE00000
RPU - Write to protected memory areas
--------------
RPU - Write to 0x40000
============================================================
EM: XMPU/XPPU violation occurred (ErrorId: 16)
EM: XMPU DDR0 Write permission violation occurred
EM: Address of poisoned operation: 0x40XXX
EM: Master Device of poisoned operation: RPU0
EM: Poison register: 0x100000
============================================================

Xilinx Wiki

Zynq UltraScale+ Isolation Configuration

Table of Contents

Introduction

Subsystems

Isolation Configuration GUI

Default policy settings

Base isolation configuration

Known issues and limitations

Example Subsystems

APU Linux only

APU Linux and RPU split non-secure

APU Linux and RPU lockstep secure

Software

FSBL

PMU Firmware

U-Boot

RPU

Example Design sources

Example Design validation

APU Linux only

APU Linux and RPU split non-secure

APU Linux and RPU lockstep secure

Related content