Zynq Ultrascale Plus Restart Solution Getting Started 2018.3


Table of Contents

Introduction

Zynq UltraScale+ MPSOC supports various type of reset from the simplest system reset to the much more complicated subsystem restart. In any system or subsystem which has a processor component and a programmable logic component, reset must entail both reset to the hardware as well as software. Reset to the hardware includes resetting of the processor, all peripherals associated with the system/subsystem, cleaning up of the memory as needed, and making sure that the interconnect is in a clean state, capable of routing traffic. Reset to the software results in the processor starting from the reset vector. However, designer must make sure that valid and clean code for system/subsystem is located at the reset vector in order to bring the system back to a clean running state.

Resets for Zynq UltraScale+ are broadly divided into two categories, full system resets and subsystem restarts.  Full system resets include Power On Reset (POR), system-reset and PS-only-reset. Subsystem restarts include APU subsystems and RPU subsystem restarts.  A subsystem in Zynq UltraScale+ is composed of all of the components of a particular operating system.  This page describes an example of how to design a project with multiple subsystems in it, and how to cleanly reset each one.


Subsystem Steps


The  following steps describe how to implement subsystem restart.  For more details on any of these steps, refer to UG1137 Software Developer's Guide, Chapter 12.  Refer to the next section for a detailed example.
  • Define the subsystem using the isolation configuration menu in Vivado and export to hdf.
  • Prepare Petalinux project with following changes w.r.t default 2018.3 bsp
    • Modify device tree source file (dts) of Linux to exclude PL IPs that are not part of APU subsystem.
    • Add rpu_firmware application binaries 
    • Add remoteproc.dtsi to allow remoteproc to load the rpu binaries
    • Add pmu-fw and ATF flags to enable warm restart, recovery, escalation
    • Add pmu-fw code patches to idle PL peripheral for given master
  • For RPU subsystem restart, make PM aware rpu applications. Make linker script inline with remoteproc.dtsi
  • Build using Petalinux.

Note: If none of the peripheral idling, recovery or escalation features are needed then there is no requirement to define the subsystem in isolation configuration.
PS only and System restart can be triggered from Linux directly. Though APU subsystem restart without peripheral idling is not recommended and can have adverse effect.

An Example

The following figure shows the example design described by this document.  It consists of an APU subsystem, RPU subsystem, and miscellaneous PL connections.  It was built from the Xilinx ZCU102 Board Support Package (BSP) hardware design, and enhanced to have more hardware components added and isolation enabled.

The APU Subsystem (APU_SS) consists of slaves and masters as shown below.

The APU masters consist of an AXI CDMA and an AXI DMA, each with their own interconnect and firewall as shown below.

APU slaves consist of a BRAM controller, GPIO, and a blinking LED using pl_clk0 from Zynq UltraScale+ MPSoC.

The RPU subsystem also consists of masters and slaves as shown below.

The RPU masters block contains an AXI DMA with an interconnect and a firewall as shown below.

The RPU slaves block contains a BRAM controller, AXI GPIO, and blinking LED as shown below.

The PL subsystem has a blinking LED on its own independent board clock, and miscellaneous logic.


The Subsystems are defined with the following components (some components were left out of the diagram for simplification):
The example design contains the following pieces:
  1. A hardware design file (hdf) which defines three subsystems, APU (including APU_secure), RPU and RPU1.  PMU Firmware subsystem is included by default. Details on subsystems can be found here.
  2. The hdf also contains PL components added to APU and RPU subsystems, including some DMAs to demonstrate how to idle a PL master.
    • Note: Custom hooks are needed to idle the PL masters of the given subsystem.  Refer to the UG1137 Software Developer's Guide, Chapter 12 for description of what is required.
  3. Recovery with healthy bit escalation scheme.
    • Recovery and escalation are enabled with build flags in pmu recipe and warm restart flag in ATF.
    • Includes application (wdt-heartbeat) to provide heartbeats to the FPD wdt.
  4. Enable XMPU / XPPU based configuration only for DDR and OCM with flag in FSBL recipe. No XMPU / XPPU protection for any other components.
  5. Helper script (installed through mpsrm-init) to facilitate setting /unsetting of healthy bit, and starting/stopping of wdt-heartbeat application.
  6. RPU application support. Application recipe to copy prebuild RPU application for r5-0 and r5-1 in the /lib/firmware directory of the rootfs
  • Note: The device-tree changes needed for the R5 application are not needed in meta-user as the default device tree generated with rpu subsystem will generate the necessary overlay.


Download Package

Click the zip icon below to download the package for the example design.

The zip has following directories:

hardware : Contains prebuilt hdf file and project sources to rebuild Vivado project.

ready_to_test: Contains boot image and kernel image, which can be loaded on SD card to test without building the example design.

software: Contains the meta-user patch layer for Petalinux and SDK projects for RPU firmware. 

Following is the complete directory tree of the package:


Below table describes various directories of the package. For more details on the need for modification/addition refer to Zynq Ultrascale Plus Restart Solution,

Top DirectorySub Directory

Description

hardwaresdk_exportContains pre-built hdf file for the hardware design
sourcesContains Tcl and xdc files that can be used for regenerating Vivado project 
ready_to_test-

Contains pre-built boot image (BOOT.bin) and kernel image (image.ub) files

softwarepetalinux-patches/meta-user

Contains the meta-user directory than can be patched over the Petalinux project with default BSP

Following are new/modified recipes in meta-user layer:

  • recipes-apps/rpu-firmware (new)
    • Adds the r50_app and r51_app binaries to rootfs. These binaries are generated separately through the SDK project.
  • recipes-bsp/arm-trusted-firmware (new)
    • Adds WARM_RESTART=1 flag for ATF, which allows ATF to respond to idle request from the pmu-fw.
  • recipes-bsp/device-tree (modified / new)
    • Add remoteproc.dtsi to reserve space for r5 applications and describe r50 and r51 rpu nodes for remoteproc.
    • Modify system-user.dtsi to include remoteproc.dtsi
    • Modify the recipe file to add new file (remoteproc.dtsi)
  • recipes-bsp/fsbl (new)
    • Add Debug flag to enable debug prints at the boot time.
  • recipes-bsp/pmu-firmware (new)
    • Add/Enable warm restart related flags while building pmu-firmware
      • ENABLE_SCHEDULER
      • ENABLE_EM
      • ENABLE_RECOVERY
      • ENABLE_ESCALATION
      • ENABLE_NODE_IDLING
      • REMOVE_GPIO_FROM_NODE_RESET_INFO
      • CHECK_HEALTHY_BOOT
      • IDLE_PERIPHERALS 
    • Add patch to not shutdown rpu during Linux's pm init finalize.
    • Add patch to block PL traffic for subsystem/ps restart
  • recipes-core/images (modified)
    • Add rpu-firmware to install list of petalinux-image-full.bbappend
  • recipes-core/watchdog (new)
    • Enable watchdog-init as Linux init script. This is responsible for setting healthy bit and starting wdt daemon. 
  • recipes-kernel/linux (modified)
    • Enable axidmatest and cdmatest modules. These can be loaded at runtime to test the DMA.
rpu-sdk-project

Contains the SDK projects for building rpu applications. Following SDK projects are provided. 

  • design_1_wrapper_hw_platform_0   – hardware project based on the prebuild hdf file.
  • r50_app   –  Application project for RPU 0 (r5-0) processor
  • r50_bsp   –  BSP project for RPU 0 (r5-0) processor
  • r51_app   –  Application project for RPU 1 (r5-1) processor 
  • r51_bsp   –  BSP project for RPU 1 (r5-1) processor

To build these project just import all of them (with copy option checked) in xsdk workspace and it builds automatically.


Building Petalinux Project with new hardware


  1. Download the default 2018.3 ZCU102 BSP from the Xilinx web page and save to your working directory.   
  2. Create new project from the default BSP.

  3. Change working directory to newly created Petalinux project directory.

  4. Configure Petalinux to use new hardware hdf (pre-built/newly created).

  5. Copy (overwrite) meta-user directory from the package to project-spec.

  6. (optional) Copy newly created RPU firmware images. Default images are already present.

  7. Add rpu-firmware to rootfs

           

  8. Build the petalinux project

    This will generate <plnx-root>/images/linux/image.ub file which is kernel image (along with dtb and rootfs).

  9. Generate the BOOT image.

    This will generate <plnx-root>/images/linux/BOOT.BIN

Copy image.ub and BOOT.BIN to the sd card.


For more information:


How To Run

To run the design, load the SD card with images mentioned above and boot the zcu102 board with sd boot mode.

To log in, use root as login and password.


Following commands can be executed from Linux prompt to perform various subsystem resets.

APU only Subsystem reboot:
PS Only reboot:
System wide reboot (default behavior without set reboot call)

Running DMA's test modules from Linux

Note: If firewall IP is being used for blocking traffic from PL, firewall IP should be cleared to allow traffic start again.

To do so, write 0x1 to MI Unblock Control Register  of firewall IP on particular PL path before starting the traffic again.

In this example design, we are blocking the traffic from APU subsystem's axi dma traffic through the Axi Firewall IP. 

Hence after APU only or PS only reboot, 0x1 should be written to AXI Firewall BaseAddress + 0x8 before starting the Axi DMA operation.

Command line solution is :

Accessing APU subsystem's BRAM from Linux

One can observe BRAM contents not changing across APU only and PS only reboot.

Following commands can be used to load/stop/replace RPU images in /lib/firmware. The images must be present.
To start rpu0 application:
To start rpu1 application:

To stop rpu0 application:

To stop rpu1 application:

Recovery and Escalations


In many use cases, recovery and escalation are turned on as they are critical for getting back into a healthy state when the system is hung. This example design also has them enabled.
To briefly explain:
Recovery: The APU subsystem (including fsbl, uboot, Linux) will be kicking SWDT regularly, but if this is missed there will be APU restart triggered due to watchdog expiry.
Escalation: On top to recovery, if for some reason previous boot was not successful and WDT expired again, there should be higher level restart (PS only or system). This is achieved by setting healthy 
bit by Linux after successful boot.
   

Normal Healthy System


In a recovery enabled system, after Linux boots up it needs to perform the following to mark itself healthy and alive. 
  1. Set the Healthy bit
  2. Start watchdog application as daemon to kick the WDT

The watchdog-init script enabled in the example design does both of these tasks after Linux boot-up process. 


Following are the description of various argument that watchdog-int script (available at /etc/init.d/watchdog-init) can take

Example to explicitly set healthy bit:


How to Build Example Design Hardware

The following steps can be used to generate the hardware description file (HDF) used in this example. 

Copy sources from the package to a working directory, then open the project in Vivado using the provided Tcl script:

When the script finishes, you should see components in your block design for MPSOC, APU_SS, RPU_SS, and PL_SS.  You should also have a constraint file added under the Sources tab.  You can double-click the Zynq UltraScale+ MPSOC component, click on the box for Switch to Advanced Mode, then click on Isolation Configuration to see how the subsystems are configured.

Click on Generate Bitstream using the link under Flow Navigator, saving the block design when prompted.

When the bitstream is ready, go to File → Export → Export Hardware and select Include Bitstream.  Use the default location for the exported hardware. Then go to File → Launch SDK using the default location and workspace to finish creating the HDF needed for building Petalinux.  Follow the steps for Building Petalinux Project with new hardware (Step 4) to use this new hardware in the Petalinux image.