This page describes the process for installing, building, and testing support for the ZU+ Video Codec Unit (VCU) under the QNX RTOS. The https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/1010303044 Design Module #1 is used as the hardware platform for the examples below. This release also supports the Quad Sensor Design Module.

Table of Contents

Introduction

Blackberry QNX provides support for the Zynq UltraScale+VCU when using their ZCU106 BSP for the QNX Neutrino RTOS. This is support is enabled by way of updates to the QNX Multimedia Suite 2.0 packages as referenced in the Required QNX RTOS Software Packages section below. In addition to the VCU support, there is an addition Sensor Framework packages which provides MIPI camera support.

In the sections below, the process for building and evaluating the packages to exercise the VCU and MIPI camera input will be described.

Required QNX Software

It is assumed that the user has installed the following tools from QNX:

• QNX Software Center

• QNX Software Development Platform Version: 7.1.0

  • available through QNX Software Center

• QNX Momentics IDE

For general information about the ZCU106 BSP that is utilized for this project, you should refer to the ZCU106 BSP User Guide.

Xilinx Boot Files Archive

A set of pre-built boot files are available in the following archive:

Please contact your Xilinx FAE or Xilinx Technical Marketing to access to the boot files archive.

The archive also includes QNX .build files.

Package Directory Structure of QNX_VCU_TRD_Package-2020.2_RELEASE.zip , received from AMD-Xilinx

Recreating the Boot Files

The bitstream and FSBL are from the https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/1010303239 design module (DM #1).

To reproduce these files, use the Build flow for Design Module #1 (Multi Stream) and follow the steps below:

• Create a Petalinux project based on BSP in the TRD

• Create a link to the device tree for DM#1

• Configure the Petalinux project to use the new .xsa

• Build the petalinux project

• The new boot artifacts will in the <petalinux project>/images/linux directory

Required QNX RTOS Software Packages

The build the QNX reference design for the ZCU106, the following runtime software packages are required from Blackberry QNX. The packages are available through the QNX Software Center application and are available upon request for current QNX customers. Please contact your QNX or Xilinx FAE for assistance.

Limitations

There are a few outstanding issues related to using the 4k display while encoding/decoding with the test applications below. At this time, it is recommended that users limit the display resolution to 1080p for best overall performance.

Building the QNX BSP Project

The QNX BSP Project can be built from the command line or using the QNX Momentics IDE. The Momentics flow running under Windows will be described below.

A “boot files archive” is needed (See “Xilinx Boot Files” section above) with all necessary components to recreate the BOOT.BIN for staring up the system.

Using the Momentics IDE Flow 

Create the ZCU106 BSP Project

1. Launch Momentics and create a new workspace.

2. Create a new project based on the ZCU106 BSP 

   • File->Import

     Open

     

   • Choose QNX->QNX Source Package and BSP. Click Next

   • Open

     

   • Browse for the ZCU106 BSP (Example steps below)

     • <QNX install path>\qnx710\bsp\BSP_xilinx-xzynq-zcu106_br-710_be-710_SVN947906_JBN0_2022

     • Click Next Twice

     • Give the project a suitable name

     • Click Finish

Modify the Project

1. Replace the default zcu106.build file with the one of the ones provided in the boot files archive. There are two options, depending on which set of applications you want to test:
   
   

   • vcu_trd_dm_1/zcu106.build: Use this if you want to capture, display, and record live video from the FMC MIPI camera (LI-IMX274MIPI-FMC). You’ll also need to install the packages mentioned in the table above.

   • vcu_trd_dm_11/zcu106.build: VCU based design supporting Avnet Quad Sensor video capture through MIPI CSI2-Rx and display through HDMI-Tx for NV12 format
     
     In either case, copy the .build file into the QNX BSP project and rename it zcu106.build. In the QNX project, the .build file is found under xilinx-zynq-zcu106->images in the project explorer.
     
     

2. Copy all of the .conf files (From package directory “QNX_VCU_TRD_Package/graphics_conf/”) in the graphics directory of the boot files archive to the following directory (this only needs to be done once and can be used for other projects): <QNX install path>/qnx710\target\qnx7\aarch64le\usr\lib\graphics\zcu10x
   
   

3. Update the Makefile in the images directory. Modify the QNX-IFS target as shown below so that QNX-IFS gets updated during every build:

Build the Project

1. Right click on the project name and select "Clean Project”

2. Right click on the project name and select "Build Project"

3. Wait for the build to complete. This process will take several minutes.

4. Once complete, the QNX-IFS will be available in the images folder along-with boot.scr.

Prepare the Boot Image

Download the VCU TRD 2020.2 release package here https://www.xilinx.com/cgi-bin/docs/ctdoc?cid=bigfile;d=rdf0428-zcu106-vcu-trd-2020-2.zip

The TRD package contents are placed in the following directory structure as shown below.

• For Quad Sensor design (VCU TRD Design Module #11)

  • The user needs to copy BOOT.bin from the rdf0428-zcu106-vcu-trd-2020.2\images\vcu_quad_sensor to FAT32 formatted SD card directory along with QNX-IFS & boot.scr , generated & available from previous steps.

• For Single Sensor design (VCU TRD Design Module #1)

  • The user needs to copy BOOT.bin from the rdf0428-zcu106-vcu-trd-2020.2\images\vcu_multistream_nv12 to FAT32 formatted SD card directory along with QNX-IFS & boot.scr , generated & available from previous steps.

  • SD Card contents

TRD package contents specific to VCU TRD Multi Stream design are placed in the following directory structure.

Preparing the Hardware

By default, this reference design targets the 2020.2 VCU TRD Design Module #1 , At this moment it, In the provided current reference design of QNX it supports DisplayPort output (DisplayPort in the PS) , With modification, other similar hardware configurations can be supported as well - please refer to Appendix A below. From a QNX software perspective, this reference design supports a subset of the VCU TRD interfaces as defined below:

Supported Hardware Interfaces

Sources

• MIPI CSI-2 Rx capture pipeline implemented in the PL

  • Capture Pipeline: MIPI CSI-2 Subsystem → Demosaic → Gamma → VPSS(CSC) → VPS (Scaler) → Framebuffer Write

• File source: SD card

Sinks

• DP Tx display pipeline in the PS.

• File sink: SD card

VCU Codec

• Video Encode/Decode capability using VCU hard block in PL 

  • AVC/HEVC encoding

  • Encoder/decoder parameter configuration.

Video Input Pipeline

MIPI CSI-2 Subsystem → Demosaic → Gamma → VPSS(CSC) → VPS (Scaler) → Framebuffer Write

Setting up the board

To set the board up for testing, please refer to the Board Setup instructions in the VCU TRD: https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/1010303044/Zynq+UltraScale+MPSoC+VCU+TRD+2020.2#3.5-Board-Setup

Booting up the Board

1. Add BOOT.BIN , QNX-IFS & boot.scr to the FAT partition of an SD Card. For best results, use the SD Memory Card Formatter from the SD Association to format the card.

2. Insert the card in ZCU106 and power on the board.

3. As part of the default boot process, the boot script will attempt to mount the SD cards at /sdcard.

4. Here is a reference command to mount the SD cards (If required) - mount -t dos /dev/hd0t12 /sdcard

Setting the MAC Address

To use the Ethernet interface, you’ll need make sure that the MAC address for the board is programmed into the on-board EEPROM. During boot, if you see an error about the MAC address not being set, refer to the ZCU106 BSP User Guide for details regarding programming the MAC address into the EEPROM.

How to run the #DM1 and DM#11 applications using prebuilt BOOT.bin and QNX-IFS

Example : Running the DM#1 Test Applications

The camera_example application can be used to exercise the available hardware interfaces. The examples in this section are provided by the QNX® Platform packages listed above. The available tests are highlighted below. Source code for the camera_example application is available from QNX upon request.

To run using pre-builts, you should use the QNX-IFS file located in the \vcu_trd_dm_1\prebuilt or vcu_trd_dm_11\prebuilt directory of the QNX Pre-built archive.

Camera Tests

In the serial console, run the camera_example application and follow the default options as marked in - README_QNX_DM1.txt

The camera streaming starts and the viewfinder should show up on the connected DisplayPort Monitor

Camera Recording Test

In the serial console, start the VCU encoder application:

Now run the camera_example application and follow the default options as marked in - README_QNX_DM1.txt

At the final menu, choose “r” to start recording and choose "r" again to stop recording. While recording, you should see the live video on the DisplayPort monitor.

In case you you like to use SD card as storage mechanism for recording instead of NFS server, The steps would be :

• Slay sensor service and restart it with the modified recording path.

• Record using camera_example & follow instructions in README_QNX_DM1.txt

Video File Playback Test

In the serial console, start the VCU decoder application:

Restart the test application by running the vcu-decoder-zcu106 and follow the default options as marked in - README_QNX_DM1.txt for File camera playback

Now run the camera_example application and press 1 for “Camera Viewfinder”. After that, select all default options:

The video at /sdcard/video.mp4 will begin to play on the monitor. Once the end of the video is reached, it will replay continuously until you press 'x' to exit the test.

Running the Multimedia Test Applications

The examples in this section are provided by the QNX® Platform for Multimedia 2.0 packages listed above. The available tests are highlighted below.

To run using pre-builts, you should use the QNX-IFS file located in the vcu_trd_dm_1\prebuilt directory of the boot files archive.

Preparing The System

Once the system is booted, run the following commands:

Xilinx VCU Tests

Included in the multimedia package are four sample applications from Xilinx to exercise the VCU at the Control Software and OpenMax layers. These sample apps operate on raw video files (.hevc, .h264, . yuv, etc. ) instead of containerized formats like .mp4.

VCU Control Software

The VCU control software is a pair of standalone command-line executables that can be used to gain fine-grained control over the operation of the VCU:

• VCU Encode: ctrlsw_encoder

• VCU Decode: ctrlsw_decoder

Command Line Examples

Encode using ctrlsw_encoder - this takes a .cfg file as an input and generates an avc or hevc encoded output file based on the YUV formated input file. The configuration file details can be found in PG252. A sample configuration file can be found in Appendix B below.

For more information on usage of the control software applications, please refer to PG252: VCU Control Software Sample Applications. The source code for the VCU Control Software is available at https://github.com/Xilinx/vcu-ctrl-sw.

VCU OpenMax Sample Applications

The OpenMax Integration Layer (OMXIL) sample applications provide example for utilizing the OpenMax Integration Layer.

• VCU Encode: vcu_omx_encoder

• VCU Decode: vcu_omx_decoder

Command Line Examples

Decode a encoded video to YUV:

Encode a 1080p60 YUV video:

For more information on the usage of the OpenMax sample applications, please refer to PG252: VCU OpenMax Integration Layer. The source code for the OpenMax Integration Layer sample applications is available at https://github.com/Xilinx/vcu-omx-il/tree/master/exe_omx.

QNX Codec Tests

mmrplay

• Used to playback .mp4 videos on the DisplayPort interface
  
  Usage:

Changing the Resolution of the Display

To change the resolution of the DisplayPort interface, you need to restart the screen service and specify a new screen configuration file. Screen configuration files have been provided for the following resolutions:

• 1920x1080 (default)

• 3840x2160

• 1280x720

To restart the screen service, issue the following commands:

Mounting an NFS Partition

For some of the tests above, it’s beneficial to use an NFS mounted filesystem due to the size of the files and the amount of bandwidth necessary. The commands below give an example for connecting to an NFS server. It is assumed that the NFS server is already running and configured correctly.

Other Hardware Configurations

It’s possible to use the QNX software implementation provided with this reference design with other hardware configurations. The sections below describe the modifications need to run this design on the https://xilinx-wiki.atlassian.netnull/pages/createpage.action?spaceKey=XWUC&title=Zynq%20UltraScale%EF%BC%8B%20MPSoC%20VCU%20TRD%202019.1%20-%20Multi%20stream%20Audio-Video%20Design&linkCreation=true&fromPageId=2038464517 design module #11.

Modifying the QNX BSP Project

Changes are required to the zcu106.build file to support the new hardware configuration.

Adding AXI Interrupt Controller

The ZCU106 BSP supports the use of the AXI Interrupt Controller soft IP in the PL to aggregate interrupts before sending a single interrupt to the GIC through the PL->PS interface. In order to use it, we need to enable it at startup and define the characteristics of the interrupt.

In this design, the AXI INTC is configured as follows:

• Base address = 0xA0055000

• Interrupt output is connected to GIC 142 (ps->pl IRQ ).

• The interrupt output is active high, level triggered

In the .boostrap section of the .build file, add the following command:

Updating the Sensor Service Configuration

By default, several xilinx-specific parameters used by the sensor service configuration are set to default values that match the 2020.2 VCU TRD Design Module #1 hardware settings. In order to configure the sensor service to values that don’t match DM#1, we can override the parameter settings in imx274.conf as shown below.

In zcu106.build, replace the /etc/imx274.conf section with:

Please refer to Appendix A: Hardware Configurations for details on the hardware configurations of DM#1,3, and 11.

Appendix A: Hardware Configurations

The following sections provide details on the PL configuration of various Vivado designs that are compatible with the QNX software libraries.

2020.2 VCU TRD Design Module #1 - Single Sensor

Image Sensor Configuration

Sensor Type: IMX274 (LI-IMX274MIPI-FMC)

Sensor I2C Interface

Sensor Video Pipeline

Note: This EMIO for MIPI RX is an active low SPI ChipSelect

Video Codec Unit IP Parameters

AXI Interrupt Controllers (PL Based)

2020.2 VCU TRD Design Module #3

Image Sensor Configuration
Sensor Type: IMX274 (LI-IMX274MIPI-FMC)
Sensor I2C Interface

 Sensor Video Pipeline

Note: This EMIO for MIPI RX is an active low SPI ChipSelect

Video Codec Unit IP Parameters

AXI Interrupt Controllers (PL Based)

2020.2 VCU TRD Design Module #11 - Quad Sensor

Image Sensor Configuration

Sensor Type: 4 x AR0231AT (AES-FMC-MC4-AR0231AT-G)

Sensor I2C Interface

Sensor Video Pipelines

Video Codec Unit IP Parameters

AXI Interrupt Controllers (PL Based)

Common Platform Notes

• For QNX, MIO is mapped to GPIO 0-77, EMIO is mapped to GPIO 78-173

• pl_ps_irq0 = GIC 121:128, pl_ps_irq1 = GIC 136:143 . QNX uses actual GIC number.

• For Interrupts connected through AXI_INTC, the logical interrupt numbers used by QNX are as follows:

Appendix B: Example ctrlsw_encoder Config File Format

The following example configuration file is used to a convert a 1080p YUV420 file avc/h264. For more info, refer to PG252.

ctrlsw_encoder -cfg avc_1080p60.cfg

Input Summary:

• file: csw_AIR_1080p30_nv12_4B_h264.yuv

• Size: 1920x1080

• Color Space: I420

Output Summary:

• file: csw_AIR_1080p30_nv12_4B_h264_yuv.h264

• profile: AVC_MAIN Level 4

Appendix C: Test Videos

Test .mp4 Videos

Big Buck Bunny: https://download.blender.org/demo/movies/BBB/

Elephant’s Dream: https://archive.org/details/ElephantsDream

Creating avc/hevc Encoded Videos From .mp4

Creating .mp4 From avc/hevc Encoded Videos

Raw Videos

https://media.xiph.org/video/derf/

These .y4m videos can be converted to YUV with the following ffmpeg command: