Zynq UltraScale+ MPSoC VCU Single Sensor ROI 2020.1

This page provides all the information related to VCU Single Sensor ROI design.

Table of Contents

1 Overview

The primary goal of this VCU Single Sensor ROI design is to demonstrate the use of DPU (Deep learning Processor Unit) block for extracting the ROI (Region of Interest) from input video frames and to use this information to perform ROI based encoding using VCU (Video Codec Unit) encoder hard block present in Zynq UltraScale+ EV devices.

The design will serve as a platform to accelerate Deep Neural Network inference algorithms using DPU and demonstrate the ROI feature of VCU encoder. The design uses a Deep Convolutional Neural Network (CNN) named Densebox, running on DPU to extract ROI Information (e.g. ‘face’ in this case).
The Design will use Vivado IPI flow for building the hardware platform and Xilinx Yocto Petalinux flow for software design. It will use Xilinx IP and Software driver to demonstrate the capabilities of different components. The Vitis platform will be created from the Vivado/PetaLInux build artifacts, and then Vitis acceleration flow will be used to insert the DPU into the platform to create the final bitstream.

The following figure shows one of the use cases (serial pipeline) with face detection and enhanced ROI quality on ZCU106.

Serial: Face detection and enhanced ROI quality on ZCU106

The following figure shows one of the use cases (streaming pipeline) with face detection and enhanced ROI quality on ZCU106.

Streaming: Face detection and enhanced ROI quality on ZCU106

1.1 System Architecture

The following figure shows the system level diagram which includes the components of the evaluation board.

1.2 Hardware Architecture

This section gives a detailed description of the blocks used in the hardware design. The functional block diagram of the design is shown in the below figure.

There are seven primary Sections in the design.

  • MIPI Capture Pipeline:

    • Captures video frame buffers from Capture source in 4K Resolution, NV12 Format

    •  Writes the buffers into DDR Memory with Frame Buffer Write IP

  • Multi-scaler Block:

    • Reads the Video Buffers from DDR Memory

    • Scales down the buffer to the 640x360 size (suitable for dpu)

    • Converts the format from NV12 to BGR

    • Writes the Down-scaled buffer to DDR Memory

  • DPU Block: 

    • Reads the downscaled buffers from DDR Memory

    • Runs the Densebox algorithm to generate the ROI information for each frame buffer

    • Passes the ROI information to VCU Encoder

  • VCU Encoder: 

    • Reads the 4K NV12 Buffer from DDR Memory

    • Receives the ROI metadata from DPU IP

    • Encodes the video buffers based on the ROI Information

    • Finally writes the encoded stream to DDR Memory

  • PS GEM:

    • Reads the Encoder stream from DDR Memory

    • Streams out the encoded stream via Ethernet

  • VCU Decoder:

    • Decodes the received encoded frame and writes to memory

  • HDMI-Tx:

    • Displays the decoded frames on HDMI Display

This design supports the following video interfaces:

Sources

  • MIPI-Rx capture pipeline implemented in the PL

  • File source (SD card, USB storage, SATA hard disk)

  • Stream-In from network or internet

Sinks

  • HDMI-Tx display pipeline implemented in the PL

  • Stream-out on network or internet

VCU Codec

  • Video Encoder/Decoder capability using VCU hard block in PL 

  • H.264/H.265 encoding

  • Encoder/decoder parameter configuration using OMX interface

DPU

Zynq DPU IP

Streaming Interfaces

1G Ethernet PS GEM

Video Format

NV12

Supported Resolution

4Kp30
1080p30

1.3 VCU ROI Software

1.3.1 GStreamer Pipeline

The GStreamer plugin demonstrates the DPU capabilities with Xilinx VCU encoder’s ROI(Region Of Interest) feature. The plugin will detect ROI(face co-ordinates) from input frames using DPU IP and pass the detected ROI information to the Xilinx VCU encoder. The following figure shows the data flow for GStreamer pipeline of stream-out use case.

Block Diagram of Stream-out Pipeline

fd = v4l2 frame data, fd' = DPU compatible frame data

As shown in the above figure, the stream-out GStreamer pipeline performs the below list of operations:

  1. v4l2src captures the data from MIPI-Rx in NV12 format and pass to xlnxroivideo1detect GStreamer plugin

  2. xlnxroivideo1detect GStreamer plugin will scale down to 640x360 resolution and convert the data into BGR format

  3. 640x360 BGR frame will be provided to DPU IP as an input to find ROI (i.e. face co-ordinates)

  4. Extracted ROI information will be passed to VCU encoder

  5. The encoder will encode the input data by encoding ROI regions with high quality as compared to non-ROI region using received ROI information

  6. Stream-out the encoded data using RTP protocol

The following figure shows the data flow for the GStreamer pipeline of stream-in use cases.

Block Diagram of Stream-in Pipeline

fd = Gst-Omx Frame data

As shown in the above figure, the stream-in GStreamer pipeline performs the below list of operations:

  1. Stream-in the encoded data using RTP protocol

  2. The Xilinx VCU decoder will decode the data

  3. Display the decoded data on HDMI-Tx display

The below figure shows the xlnxroivideo1detect GStreamer plugin data flow.

As shown in the above figure, our xlnxroivideo1detect GStreamer plugin will perform below the list of operations:

  1. DPU is initialized

  2. DPU GStreamer plugin receives the data frame from MIPI-Rx through a v4l2src plugin

  3. Create the DPU task

  4. Scale the input frame to 640x360 resolution using Xilinx Scaler IP

  5. Convert the input frame data format from NV12 to BGR format using Xilinx Color Space Converter(CSC) soft IP

  6. Prepare the OpenCV image using BGR data

  7. Pass the intermediate OpenCV image to the DPU

  8. Run the DPU task

  9. Extract the ROI(face) co-ordinates from the DPU output

  10. Map the detected face co-ordinates to the original input frame resolution

  11. Fill the ROI metadata buffer using extracted ROI (face) co-ordinates

  12. Pass the ROI metadata buffer and input NV12 frame data buffer to the Xilinx VCU encoder

  13. De-initialize the DPU task

1.3.2 DPU(Deep Learning Processor Unit)

DPU is a programmable engine optimized for deep neural networks. It is a group of parameterizable IP cores pre-implemented on the hardware with no place and route required. The DPU is released with the Vitis AI specialized instruction set, allowing efficient implementation of many deep learning networks.

Refer to DPU IP PG338 and UG1354 to know more details on DPU.

The following figure shows the DPU Top-Level Block Diagram.

DPU Top-level Block Diagram

The DPU IP can be implemented in the programmable logic (PL) of the selected Zynq® UltraScale+™ MPSoC device with direct connections to the processing system (PS). The DPU requires instructions to implement a neural network and accessible memory locations for input images as well as temporary and output data. A program running on the application processing unit (APU) is also required to service interrupts and coordinate data transfers.

The following figure shows the sequence of operations performed on the DPU device.

The following sequence of steps are performed to access and run face detection using the DPU device:

  1. DPU device is initialized

  2. Instantiate a DPU Task from DPU Kernel and allocate corresponding DPU memory buffer

  3. Set the input image to created DPU task

  4. Run the DPU task to find the faces from the input image

  5. DPU device is uninitialized

1.4 Software Tools and System Requirements

Hardware

Required:

  • ZCU106 evaluation board (rev C/D/E/F/1.0) with power cable

  • Monitor with HDMI input supporting 3840x2160 resolution or 1920x1080 resolution

  • LI-IMX274MIPI-FMC image sensor daughter card

  • HDMI cable 2.0 certified

  • Class-10 SD card

  • Ethernet cable

Optional:

  • USB pen drive formatted with the FAT32 file system and hub

  • SATA drive formatted with the FAT32 file system, external power supply, and data cable

Software Tools

Required:

Download, Installation, and Licensing of Vivado Design Suite 2020.1

The Vivado Design Suite User Guide explains how to download and install the Vivado® Design Suite tools, which include the Vivado Integrated Design Environment (IDE), High-Level Synthesis tool, and System Generator for DSP. This guide also provides information about licensing and administering evaluation and full copies of Xilinx design tools and intellectual property (IP) products. The Vivado Design Suite can be downloaded from here.

LogiCORE IP Licensing

The following IP cores require a license to build the design.

  • Video Mixer- Included with Vivado - PG243

  • Video PHY Controller - Included with Vivado - PG203

  • HDMI-Rx/Tx Subsystem - Purchase license (Hardware evaluation available) - PG235 & PG236

  • Video Processing Subsystem (VPSS) - Included with Vivado - PG231

  • MIPI CSI Controller Subsystems (mipi_csi2_rx_subsystem) - Purchase license (Hardware evaluation available) - PG232

To obtain the LogiCORE IP license, please visit the respective IP product page and get the license.

Compatibility

The reference design has been tested successfully with the following user-supplied components.

HDMI Monitor:

Make/Model

Resolutions

Make/Model

Resolutions

LG 27UD88

3840 x 2160 @ 30Hz

Samsung LU28ES90DS/XL

3840 x 2160 @ 30Hz

Cable:

  • HDMI 2.0 compatible cable

The below table provides the performance information:

Resolution

FPS Achieved

Resolution

FPS Achieved

4Kp30

27 - 30

1080p30

30

Above FPS are measured withgop-mode=basic gop-length=60 b-frames=0 target-bitrate=1500 num-slices=8 control-rate=constant prefetch-buffer=true low-bandwidth=false qp-mode=roi encoder parameters for AVC and HEVC.

1.5 Board Setup

The below section will provide the information on the ZCU106 board setup for running ROI design.

  1. Connect the Micro USB cable into the ZCU106 Board Micro USB port J83, and the other end into an open USB port on the host PC. This cable is used for UART over USB communication.

  2. Insert the SD card with the images copied into the SD card slot J100. Please find here how to prepare the SD card for a specific design.

  3. Set the SW6 switches as shown in the below Figure. This configures the boot settings to boot from SD.

  4. Connect 12V Power to the ZCU106 6-Pin Molex connector

  5. Connect one end of HDMI cable to the board’s P7 stacked HDMI connector (upper port) and another end to HDMI monitor

  6. For a USB storage device, connect the USB hub along with the mouse. (Optional)

  7. For SATA storage device, connect SATA data cable to SATA 3.0 port. (Optional)

  8. For MIPI CSI-2, Insert the LI-IMX274MIPI-FMC image sensor daughter card into the FMC0 connector and set VADJ to 1.2V

  9. Set up a terminal session between a PC COM port and the serial port on the evaluation board (See the Determine which COM to use to access the USB serial port on the ZCU106 board for more details)

  10. Copy the VCU Single Sensor ROI TRD images into the SD card and insert the SD card on the board

  11. The below images will show how to connect interfaces on the ZCU106 board

ZCU106  board connections

ZCU106 board connector slots

1.6 Run Flow

The VCU Single Sensor ROI package is released with the source code, Vivado project, Petalinux BSP, and SD card image that enables the user to run the demonstration. It also includes the binaries necessary to configure and boot the ZCU106 board. Prior to running the steps mentioned in this wiki page, download the VCU Single Sensor ROI package and extract its contents to a directory referred to as TRD_HOME which is the home directory.

Refer below link to download the VCU Single Sensor ROI package.

TRD package contents are placed in the following directory structure.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 rdf0428-zcu106-vcu-single-sensor-roi-2020-1 ├── apu │   └── vcu_petalinux_bsp │   └── xilinx-vcu-roi-zcu106-v2020.1-final.bsp ├── dpu │   └── 0001-Added-ZCU106-configuration-to-support-DPU-in-ZCU106.patch ├── image │   ├── boot │   │   ├── autostart.sh │   │   ├── bd.hwh │   │   ├── BOOT.BIN │   │   ├── boot.scr │   │   ├── config │   │   ├── dpu.xclbin │   │   ├── Image │   │   ├── setup.sh │   │   ├── system.dtb │   │   ├── vcu │   │   └── vitis │   ├── root │   │   └── rootfs.ext4 │   └── sd_card.img ├── pl │   ├── constrs │   │   ├── roi_async.xdc │   │   └── vcu_roi.xdc │   ├── designs │   │   └── zcu106_ROI_MIPI │   ├── prebuild │   │   └── zcu106_ROI_MIPI_wrapper.xsa │   ├── README.md │   └── srcs │   ├── hdl │   └── ip └── README.txt

The below snippet shows the directory structure of various configuration files which are input for vcu_gst_app to run display and streaming use cases. All these configurations files are placed in the $TRD_HOME/image/boot/config directory.

1 2 3 4 5 6 7 8 9 10 └── config    ├── 1080p30    │   ├── Display    │   ├── Stream-in    │   └── Stream-out    ├── 4kp30    │   ├── Display    │   ├── Stream-in    │   └── Stream-out    └── input.cfg

1.6.1 Preparing the SD card

There are three ways to prepare the SD card for booting. Each method is detailed below.

Using ready to test image

  • sd_card.img is available at rdf0428-zcu106-vcu-single-sensor-roi-2020-1/image/sd_card.img

  • All the required Vitis packages are already installed in ready to test rdf0428-zcu106-vcu-single-sensor-roi-2020-1/image/sd_card.img

Using Pre-built images

  • To Create SD Card with two partitions: Boot(FAT32+Bootable) and Root(EXT4) Refer this Link.

  • Copy boot content fromrdf0428-zcu106-vcu-single-sensor-roi-2020-1/image/boot to Boot partition in SD Card

  • Extract rootfs.ext4 from rdf0428-zcu106-vcu-single-sensor-roi-2020-1/image/rootto Root partition in SD Card

  • Boot the board with Flashed SD Card

Use the Output of the Build Flow

  • To Create SD Card with two partitions: Boot(FAT32+Bootable) and Root(EXT4) Refer this Link.

  • For Build Flow refer this steps and copy mentioned generated dpu build images bd.hwh BOOT.BIN boot.scr dpu.xclbin Image system.dtb into BOOT partition of the SD card and extract generated rootfs.ext4 into ROOT partition of SD Card

  • Copy the mentioned boot content config, vitis, autostart.sh, setup.sh from rdf0428-zcu106-vcu-single-sensor-roi-2020-1/image/boot directory to Boot partition in SD Card

  • Boot the board with Flashed SD Card

  • All the required Vitis AI packages are already available in rdf0428-zcu106-vcu-single-sensor-roi-2020-1/image/boot/vitis directory which are installed automatically during 1st time boot. Please wait till target setup complete and Vitis AI packages are installed.

1.6.2 Using GStreamer Application (vcu_gst_app)

The vcu_gst_app is a command-line multi-threaded Linux application. The command-line application requires an input configuration file (input.cfg) to be provided in the plain text.

After the board gets booted, the display screen turns to the blue screen; which means it is ready to test. Otherwise before the execution of vcu_gst_app, manually run modetest command to set CRTC configurations.

1 $ modetest -D a00c0000.v_mix -s 39:3840x2160-30@AR24 &

Display: Capture → (ROI) → Encode → Decode → Display

  • 4Kp30 HEVC Display Pipeline execution using vcu_gst_app

    1 $ vcu_gst_app /media/card/config/4kp30/Display/Single_4kp30_HEVC.cfg

Stream-out: ( Server )

  • Set IP address for server:

    1 $ ifconfig eth0 192.168.25.90
  • 4Kp30 HEVC Stream-out Pipeline execution using vcu_gst_app

    1 $ vcu_gst_app /media/card/config/4kp30/Stream-out/Single_4kp30_HEVC.cfg

Stream-in: ( Client )

  • Set IP address for the client:

    1 $ ifconfig eth0 192.168.25.89
  • 4Kp30 HEVC Stream-in Pipeline execution using vcu_gst_app

    1 $ vcu_gst_app /media/card/config/4kp30/Stream-in/input.cfg

1.7 Build Flow

Refer below link to download the VCU Single Sensor ROI package.

Unzip the released package.

1 $ unzip </path/to/downloaded/zipfile>/rdf0428-zcu106-vcu-single-sensor-roi-2020-1.zip

The following tutorials assume that the $TRD_HOME environment variable is set as given below.

1 $ export TRD_HOME=</path/to/downloaded/zipfile>/rdf0428-zcu106-vcu-single-sensor-roi-2020-1

1.7.1 HW build Flow

This section explains the steps to build the hardware platform and generate XSA using the Vivado tool.

Refer to the Vivado Design Suite User Guide: Using the Vivado IDE, UG893, for setting up the Vivado environment.

Refer to the vivado-release-notes-install-license(UG973) for installation.

Make sure that the necessary IP licenses are in place

ON LINUX:

  • Open a Linux terminal

  • Change directory to $TRD_HOME/pl folder

  • Source Vivado settings.sh

    1 2 $ bash $ source <path/to/Vivado-installer>/tool/Vivado/2020.1/settings64.sh
  • Run the following command to create the Vivado IPI project and invoke the GUI and generate XSA required for the platform

    1 % vivado -source ./designs/zcu106_ROI_MIPI/project.tcl
  • The project.tcl script does the following

    • Creates project

    • Creates IPI Block design with platform interfaces

    • Runs Synthesis and Implementation

    • Builds bitstream with no accelerators

    • Export the HW to XSA

  • zcu106_ROI_MIPI_wrapper.xsa is created

  • stored in the directory ../zcu106_ROI_MIPI.xsa/

  • at location $TRD_HOME/pl/build/zcu106_ROI_MIPI/zcu106_ROI_MIPI.xsa/

  • This XSA is used by Petalinux for platform creation and also by the Vitis Tool for DPU Kernel Integration.

After executing the script, the Vivado IPI block design comes up.

 

1.7.1.1 Platform Interfaces

The screenshots below show the platform interfaces that have been made available to the Vitis tool for linking in acceleration IP.

In the case of this reference design, the DPU Kernel will be inserted.

 

After the DPU Kernel is integrated dynamically with the platform using Vitis Flow, the connections are as shown below

  • The DPU Data ports are connected to the HP0 Port(S_AXI_HP0_FPD) of PS .

  • The DPU Instruction port is connected to the S_AXI_LPD port of PS through axi_interconnect_lpd

  • The DPU S_AXI_Control port is connected to the M_AXI_HPM0_LPD port of PS through interconnect_axilite

  • The DPU interrupt is connected to the axi interrupt controller dynamically

1.7.2 Petalinux build Flow

This tutorial shows how to build the Linux image and boot image using the PetaLinux build tool.

PetaLinux Installation: Refer to the PetaLinux Tools Documentation (UG1144) for installation.

It is recommended to follow the build steps in sequence

  • Source Petalinux settings.sh

    1 2 $ bash $ source <path/to/petalinux-installer>/tool/petalinux-v2020.1-final/settings.sh
  • Create PetaLinux project

    1 2 $ cd $TRD_HOME/apu/vcu_petalinux_bsp $ petalinux-create -t project -s xilinx-vcu-roi-zcu106-v2020.1-final.bsp
  • Configure the PetaLinux project

    1 2 $ cd xilinx-vcu-roi-zcu106-v2020.1-final $ petalinux-config --silentconfig --get-hw-description=<Path to directory of XSA>
    • For e.g.

    • using the prebuild XSA

      1 $ petalinux-config --silentconfig --get-hw-description=$TRD_HOME/pl/prebuild/
    • using the XSA generated by running the Hardware project.tcl scripts

      1 $ petalinux-config --silentconfig --get-hw-description=$TRD_HOME/pl/build/zcu106_ROI_MIPI/zcu106_ROI_MIPI.xsa/
  • zcu106_ROI_MIPI_wrapper.xsa is available in directory zcu106_ROI_MIPI.xsa

  • Build the PetaLinux project

    1 $ petalinux-build

1.7.3 Prepare Build Artifacts for Platform Creation

To prepare artifacts required for ZCU106 Vitis platform creation, follow below steps after petalinux build

  • Go to the petalinux build image directory

    1 $ cd $TRD_HOME/apu/vcu_petalinux_bsp/xilinx-vcu-roi-zcu106-v2020.1-final/images/linux
  • Create linux.bif file as below in images/linux directory. linux.bif file is required to create ZCU106 Vitis platform which has information related to boot components. After zcu106 Vitis platform creation, this linux.bif file will be part of platform, which is required to build DPU and generate final BOOT.BIN.

    1 2 3 4 5 6 7 8 9 /* linux */ the_ROM_image: { [bootloader, destination_cpu = a53-0] <fsbl.elf> [pmufw_image] <pmufw.elf> [destination_device=pl] <bitstream> [destination_cpu=a53-0, exception_level=el-3, trustzone] <bl31.elf> [destination_cpu=a53-0, exception_level=el-2] <u-boot.elf> }
  • Copy generated images into boot and image directory by following below commands. Use created linux.bif to copy into boot directory.

    1 2 3 4 5 6 7 8 $ mkdir boot $ mkdir image $ cp linux.bif boot/. $ cp bl31.elf pmufw.elf u-boot.elf boot/. $ cp zynqmp_fsbl.elf boot/fsbl.elf $ cp boot.scr Image rootfs.ext4 system.dtb image/.

1.7.4 ZCU106 Platform Creation

This section shows how to create a Vitis acceleration platform for the zcu106 using the Vitis IDE.

It is recommended to follow the build steps in sequence

1 2 $ source <vitis install path>/Vitis/2020.1/settings64.sh $ vitis &

Choose project workspace and click on Launch to begin

User can create Workspace in any directory. For example $TRD_HOME/vitis_workspace

Launch the New Platform Project dialog box using following step:
Go to File > New > Platform Project

Provide a project name “zcu106_dpu“ in the Platform project name field and click Next as shown in below figure

In the Platform Project dialog box, choose Create a new platform from hardware specification (XSA) and provide the XSA path

  • For prebuild XSA, use $TRD_HOME/pl/prebuild/zcu106_ROI_MIPI_wrapper.xsa path

  • For generated XSA using Vivado build, use $TRD_HOME/pl/build/zcu106_ROI_MIPI/zcu106_ROI_MIPI.xsa/zcu106_ROI_MIPI_wrapper.xsapath

Use below settings under Software Specification

  1. Select linux as the operating system, psu_cortexa53 as processor, and 64-bit architecture to create the platform

  2. Uncheck the box for Generate boot components

  3. Click Finish to create your platform project

 

Go to zcu106_dpu > linux on psu_cortexa53 and add path of required files in Domain: linux_domain

Give the bif file, boot directory, image path, and rootfs path as shown in below figure

  • Use <Path to Petalinux Project>/xilinx-vcu-roi-zcu106-v2020.1-final/images/linux/boot/linux.bif for Bif File

  • Use <Path to Petalinux Project>/xilinx-vcu-roi-zcu106-v2020.1-final/images/linux/boot/ for Boot Components Directory

  • Use <Path to Petalinux Project>/xilinx-vcu-roi-zcu106-v2020.1-final/images/linux/image for Linux Image Directory

  • Use <Path to Petalinux Project>/xilinx-vcu-roi-zcu106-v2020.1-final/images/linux/image/rootfs.ext4 for Linux Rootfs

Right click on the zcu106_dpu project in the Explorer tab and click on Build Project to generate the platform as shown in below figure
The Console tab shows the status of the platform generation.

As shown in below image, zcu106_dpu.xpfm is created under zcu106_dpu > export > zcu106_dpu > zcu106_dpu.xpfm

1.7.5 DPU Build

  • It is recommended to follow the build steps in sequence

  • Make sure parted utility is installed for DPU build

  • Clone the Vitis-AI repository and apply patch to add support of ZCU106 in Vitis DPU TRD.

1 2 3 4 $ git clone https://github.com/Xilinx/Vitis-AI.git $ cd Vitis-AI $ git checkout tags/v1.2 -b v1.2 $ git am $TRD_HOME/dpu/0001-Added-ZCU106-configuration-to-support-DPU-in-ZCU106.patch

The following tutorials assume that the $DPU_TRD_HOME environment variable is set as given below.

1 $ export DPU_TRD_HOME =<Vitis AI path>/DPU-TRD

The following tutorials assume that the Vitis and XRT environment variable is set as given below.

  • Open a Linux terminal. Set the Linux as Bash mode.

    1 2 $ source <vitis install path>/Vitis/2020.1/settings64.sh $ source <xrt install path>/xilinx/xrt/setup.sh
  • The default setting of DPU is B4096 with RAM_USAGE_LOW, CHANNEL_AUGMENTATION_ENABLE, DWCV_ENABLE, POOL_AVG_ENABLE, RELU_LEAKYRELU_RELU6, Softmax. Read the $DPU_TRD_HOME/prj/Vitis/dpu_conf.vh file to get the details of DPU. You can get all the configurations form PG338. Modify the $DPU_TRD_HOME/prj/Vitis/dpu_conf.vh file can change the default settings.

    • Enable the URAM for ZCU106 DPU Build, The DPU will replace the bram to the uram.

    • Change 'define URAM_DISABLE to `define URAM_ENABLE

1 `define URAM_ENABLE
  • Build the hardware design

    1 2 3 4 $ cd $DPU_TRD_HOME/prj/Vitis $ export EDGE_COMMON_SW=<Path to Petalinux Project>/xilinx-vcu-roi-zcu106-v2020.1-final/images/linux/image $ export SDX_PLATFORM=<Path to Vitis Workspace>/zcu106_dpu/export/zcu106_dpu/zcu106_dpu.xpfm $ make KERNEL=DPU DEVICE=zcu106
  • Generated SD card files are in $DPU_TRD_HOME/prj/Vitis/binary_container_1/sd_card.(SD Card Format)

  • Copy generated dpu build images from $DPU_TRD_HOME directory to $TRD_HOME directory

    1 2 3 $ cd $DPU_TRD_HOME/prj/Vitis/binary_container_1/sd_card $ cp bd.hwh BOOT.BIN boot.scr dpu.xclbin Image system.dtb $TRD_HOME/image/boot $ cp rootfs.ext4 $TRD_HOME/image/root

2 Other Information

2.1 Known Issues

2.2 Limitations

2.3 Optimum VCU Encoder parameters for use-cases

Video streaming:

  • Video streaming use-case requires a very stable bitrate graph for all pictures

  • It is good to avoid periodic large Intra pictures during the encoding session

  • Good to avoid periodic Intra frames instead use low-delay-p (IPPPPP…)

  • VBR is not a preferred mode of streaming

Performance: AVC Encoder settings:

  • It is preferred to use 8 or higher slices for better AVC encoder performance

  • AVC standard does not support Tile mode processing which results in the processing of MB rows sequentially for entropy coding

Quality: Low bitrate AVC encoding:

  • The high profile enables 8x8 transform which results in better video quality at low bitrates

3 Appendix A - Input Configuration File (input.cfg)

The example configuration files are stored at /media/card/config/ folder.

Common Configuration:
It is the starting point of a common configuration.

Num of Input:
Provide the number of inputs. It will be 1 only due to single-stream support
Options: 1

Output:
Select the video interface.
Options: HDMI

Out Type:
Options: display and stream

Display Rate:
Pipeline frame rate.
Options: 30 FPS

Exit:
It indicates to the application that the configuration is over.

Input Configuration:
It is the starting point of the input configuration.

Input Num:
Due to single stream support, there will be 1 input configuration only
Options: 1

Input Type:
Input source type.
Options: MIPI, Stream

Uri:
Network URL. Applicable for stream-in pipeline only.
Options: udp://192.168.25.89:5004/ (for Network streaming, here 192.168.25.89 is IP address and 5004 is port no)

Raw:
To tell the pipeline is processed. Pass-through is not supported.
Options: False

Width:
The width of the live source.
Options: 3840, 1920

Height:
The height of the live source.
Options: 2160, 1080

Enable Roi:
Enable or Disable ROI in the pipeline.
Options: True, False

Relative QP:
To improve ROI region quality.
Option: 21 to -21 (poor to best)

Exit:
It indicates to the application that the configuration is over.

Encoder Configuration:
It is the starting point of encoder configuration.

Encoder Num:
Due to single stream support, there will be 1 encoder configuration only
Options: 1

Encoder Name:
Name of the encoder.
Options: AVC, HEVC

Profile:
Name of the profile.
Options: baseline, main or high for AVC. Main for HEVC

Rate Control:
Rate control options.
Options: CBR, low-latency

Filler Data:
Options: True, False

QP:
QP control mode used by the VCU encoder.
Options: Uniform, Auto, Roi

L2 Cache:
Enable or Disable L2Cache buffer in the encoding process.
Options: True, False

Latency Mode:
Encoder latency mode.
Options: Normal, sub_frame

Low Bandwidth:
If enabled, decrease the vertical search range used for P-frame motion estimation to reduce the bandwidth.
Options: True, False

Gop Mode:
Group of Pictures mode.
Options: Basic, low_delay_p

Bitrate:
Target bitrate in Kbps
Options: 1000-1500

B Frames:
Number of B-frames between two consecutive P-frames
Options: 0

Slice:
The number of slices produced for each frame. Each slice contains one or more complete macroblock/CTU row(s). Slices are distributed over the frame as regularly as possible. If slice-size is defined as well more slices may be produced to fit the slice-size requirement.

Options:
4-22 4Kp resolution with HEVC codec
4-32 4Kp resolution with AVC codec
4-32 1080p resolution with HEVC codec
4-32 1080p resolution with AVC codec

GoP Length:
The distance between two consecutive I frames
Options: 1-1000

Format:
The format of input data.
Options: NV12

Preset:
Options: custom

Exit
It indicates to the application that the configuration is over.

Streaming Configuration:
It is the starting point of streaming configuration.

Streaming Num:
Due to single stream support, there will be 1 streaming configuration only
Options: 1

Host IP:
The host to send the packets to
Options: 192.168.25.89 or Windows PC IP

Port:
The port to send the packets to
Options: 5004

Exit
It indicates to the application that the configuration is over.

Trace Configuration:
It is the starting point of trace configuration.

FPS Info:
To display fps info on the console
Options: True, False

APM Info:
To display the APM counter number on the console
Options: True, False

Pipeline Info:
To display pipeline info on the console
Options: True, False

Loop Playback:
To play display pipeline in the loop
Options: True, False

Loop Interval:
To repeat the playback of the display pipeline after a specific time duration. The default is False.
Options: 5 - 10 sec

Exit
It indicates to the application that the configuration is over.

4 Appendix B - MIPI-Rx/HDMI-Tx Link-up and GStreamer Commands

This section covers configuration of MIPI-Rx using vcu_gst_app and HDMI-Tx using modetest utility, along with demonstrating MIPI-Rx and HDMI-Tx link-up issues. It also contains sample GStreamer MIPI Video pipelines for Display, Stream-in and Stream-out ROI use-cases.

  • Run the below command for media node to print media device topology; and to check the current resolution and video node of the MIPI input source

    1 $ media-ctl -p -d /dev/media0

When the MIPI source is connected, it shows:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 root@zcu106_vcu_trd:/media/card# media-ctl -p -d /dev/media0 Media controller API version 5.4.0 Media device information ------------------------ driver xilinx-video model Xilinx Video Composite Device serial bus info hw revision 0x0 driver version 5.4.0 Device topology - entity 1: vcap_csi output 0 (1 pad, 1 link) type Node subtype V4L flags 0 device node name /dev/video0 pad0: Sink <- "a0240000.v_proc_ss":1 [ENABLED] - entity 5: IMX274 1-001a (1 pad, 1 link) type V4L2 subdev subtype Sensor flags 0 device node name /dev/v4l-subdev0 pad0: Source [fmt:SRGGB10_1X10/3840x2160@1/60 field:none colorspace:srgb crop.bounds:(0,0)/3840x2160 crop:(0,0)/3840x2160 compose.bounds:(0,0)/3840x2160 compose:(0,0)/3840x2160] -> "a0000000.mipi_csi2_rx_subsystem":1 [ENABLED] - entity 7: a0000000.mipi_csi2_rx_subsystem (2 pads, 2 links) type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev1 pad0: Source [fmt:SRGGB8_1X8/1920x1080 field:none colorspace:srgb] -> "a0060000.v_demosaic":0 [ENABLED] pad1: Sink [fmt:SRGGB8_1X8/1920x1080 field:none colorspace:srgb] <- "IMX274 1-001a":0 [ENABLED] - entity 10: a0060000.v_demosaic (2 pads, 2 links) type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev2 pad0: Sink [fmt:SRGGB8_1X8/1280x720 field:none colorspace:srgb] <- "a0000000.mipi_csi2_rx_subsystem":0 [ENABLED] pad1: Source [fmt:RBG888_1X24/1280x720 field:none colorspace:srgb] -> "a0090000.v_gamma_lut":0 [ENABLED] - entity 13: a0090000.v_gamma_lut (2 pads, 2 links) type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev3 pad0: Sink [fmt:RBG888_1X24/1280x720 field:none colorspace:srgb] <- "a0060000.v_demosaic":1 [ENABLED] pad1: Source [fmt:RBG888_1X24/1280x720 field:none colorspace:srgb] -> "a0040000.v_proc_ss":0 [ENABLED] - entity 16: a0040000.v_proc_ss (2 pads, 2 links) type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev4 pad0: Sink [fmt:VYYUYY8_1X24/1280x720 field:none colorspace:rec709] <- "a0090000.v_gamma_lut":1 [ENABLED] pad1: Source [fmt:VYYUYY8_1X24/1280x720 field:none colorspace:rec709] -> "a0240000.v_proc_ss":0 [ENABLED] - entity 19: a0240000.v_proc_ss (2 pads, 2 links) type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev5 pad0: Sink [fmt:VYYUYY8_1X24/1280x720 field:none colorspace:srgb] <- "a0040000.v_proc_ss":1 [ENABLED] pad1: Source [fmt:VYYUYY8_1X24/1920x1080 field:none colorspace:srgb] -> "vcap_csi output 0":0 [ENABLED]

When the MIPI source is set to 4K resolution, it shows:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 root@zcu106_vcu_trd:/media/card# media-ctl -p -d /dev/media0 Media controller API version 5.4.0 Media device information ------------------------ driver xilinx-video model Xilinx Video Composite Device serial bus info hw revision 0x0 driver version 5.4.0 Device topology - entity 1: vcap_csi output 0 (1 pad, 1 link) type Node subtype V4L flags 0 device node name /dev/video0 pad0: Sink <- "a0240000.v_proc_ss":1 [ENABLED] - entity 5: IMX274 1-001a (1 pad, 1 link) type V4L2 subdev subtype Sensor flags 0 device node name /dev/v4l-subdev0 pad0: Source [fmt:SRGGB8_1X8/3840x2160@1/30 field:none crop.bounds:(0,0)/3840x2160 crop:(0,0)/3840x2160 compose.bounds:(0,0)/3840x2160 compose:(0,0)/3840x2160] -> "a0000000.mipi_csi2_rx_subsystem":1 [ENABLED] - entity 7: a0000000.mipi_csi2_rx_subsystem (2 pads, 2 links) type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev1 pad0: Source [fmt:SRGGB8_1X8/3840x2160 field:none] -> "a0060000.v_demosaic":0 [ENABLED] pad1: Sink [fmt:SRGGB8_1X8/3840x2160 field:none] <- "IMX274 1-001a":0 [ENABLED] - entity 10: a0060000.v_demosaic (2 pads, 2 links) type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev2 pad0: Sink [fmt:SRGGB8_1X8/3840x2160 field:none] <- "a0000000.mipi_csi2_rx_subsystem":0 [ENABLED] pad1: Source [fmt:RBG888_1X24/3840x2160 field:none] -> "a0090000.v_gamma_lut":0 [ENABLED] - entity 13: a0090000.v_gamma_lut (2 pads, 2 links) type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev3 pad0: Sink [fmt:RBG888_1X24/3840x2160 field:none] <- "a0060000.v_demosaic":1 [ENABLED] pad1: Source [fmt:RBG888_1X24/3840x2160 field:none] -> "a0040000.v_proc_ss":0 [ENABLED] - entity 16: a0040000.v_proc_ss (2 pads, 2 links) type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev4 pad0: Sink [fmt:RBG888_1X24/3840x2160 field:none] <- "a0090000.v_gamma_lut":1 [ENABLED] pad1: Source [fmt:RBG888_1X24/3840x2160 field:none] -> "a0240000.v_proc_ss":0 [ENABLED] - entity 19: a0240000.v_proc_ss (2 pads, 2 links) type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev5 pad0: Sink [fmt:RBG888_1X24/3840x2160 field:none] <- "a0040000.v_proc_ss":1 [ENABLED] pad1: Source [fmt:VYYUYY8_1X24/3840x2160 field:none] -> "vcap_csi output 0":0 [ENABLED]

Notes for gst-launch-1.0 commands:

Run below command to set MIPI-Rx nodes to 4K resolution

1 $ vcu_gst_app /media/card/config/4kp30/Display/Single_4kp30_HEVC.cfg

Run below command to set MIPI-Rx nodes to 1080p resolution

1 $ vcu_gst_app /media/card/config/4kp30/Display/Single_1080p30_HEVC.cfg

If HDMI-Tx link-up issue is observed after Linux booting, use the following command:

  • For 4K Resolution:

    1 $ modetest -D a00c0000.v_mix -s 39:3840x2160-30@AR24 &
  • For 1080p Resolution:

    1 $ modetest -D a00c0000.v_mix -s 39:1920x1080-30@AR24 &

GStreamer Static Pipelines:

Display: Capture → (ROI) → Encode → Decode → Display

  • Run the following gst-launch-1.0 command to display processed pipeline (capture → roi_plugin → encode → decode → display) on HDMI-Tx

    1 gst-launch-1.0 v4l2src device=/dev/video0 io-mode=4 ! video/x-raw, width=3840, height=2160, format=NV12, framerate=30/1 ! xlnxroivideo1detect capture-io-mode=4 output-io-mode=5 relative-qp=-21 ! omxh265enc gop-mode=basic gop-length=60 b-frames=0 target-bitrate=1500 num-slices=8 control-rate=constant prefetch-buffer=true low-bandwidth=false filler-data=true cpb-size=1000 initial-delay=500 qp-mode=roi ! video/x-h265, profile=main, alignment=au ! queue ! omxh265dec internal-entropy-buffers=5 low-latency=0 split-input=true ! queue max-size-bytes=0 ! fpsdisplaysink name=fpssink text-overlay=false 'video-sink=kmssink bus-id=a00c0000.v_mix hold-extra-sample=1 show-preroll=false sync=true' sync=true -v

Stream-out: ( Server )

  • Set IP address for server:

    1 $ ifconfig eth0 192.168.25.90
  • Run the following gst-launch-1.0 command for stream-out pipeline

    1 gst-launch-1.0 v4l2src device=/dev/video0 io-mode=4 ! video/x-raw, format=NV12, width=3840, height=2160, framerate=30/1 ! xlnxroivideo1detect capture-io-mode=4 output-io-mode=5 relative-qp=-21 ! omxh265enc gop-mode=basic gop-length=60 b-frames=0 target-bitrate=1500 num-slices=8 control-rate=constant prefetch-buffer=true low-bandwidth=false filler-data=true cpb-size=1000 initial-delay=500 periodicity-idr=60 qp-mode=roi ! video/x-h265, profile=main, alignment=au ! queue ! mpegtsmux alignment=7 name=mux ! rtpmp2tpay ! udpsink host=192.168.25.89 port=5004
  • Here 192.168.25.89 is host/client IP address and 5004 is port number

Stream-in: ( Client )

  • Set IP address for the client:

    1 $ ifconfig eth0 192.168.25.89
  • Run the following gst-launch-1.0 command to display stream-in on HDMI-Tx video using the GStreamer pipeline where 5004 is port number

    1 gst-launch-1.0 udpsrc port=5004 buffer-size=60000000 caps="application/x-rtp, clock-rate=90000" ! rtpjitterbuffer latency=1000 ! rtpmp2tdepay ! tsparse ! video/mpegts ! tsdemux name=demux ! queue ! h265parse ! video/x-h265, profile=main, alignment=au ! omxh265dec internal-entropy-buffers=5 low-latency=0 split-input=true ! queue max-size-bytes=0 ! fpsdisplaysink name=fpssink text-overlay=false 'video-sink=kmssink bus-id=a00c0000.v_mix hold-extra-sample=1 show-preroll=false sync=true' sync=true -v