This page provides all the information related to VCU HDMI ROI TRD design for ZCU106.

Table of Contents

1 Overview

The primary goal of this VCU ROI design is to demonstrate the use of DPU (Deep learning Processor Unit) block for extracting the ROI (Region of Interest) from input video frames and to use this information to perform ROI based encoding using VCU (Video Codec Unit) encoder hard block present in Zynq UltraScale+ EV devices.

The design will serve as a platform to accelerate Deep Neural Network inference algorithms using DPU and demonstrate the ROI feature of VCU encoder. The design uses a Deep Convolutional Neural Network (CNN) named Densebox, running on DPU to extract ROI Information (e.g. ‘face’ in this case).
The Design will use Vivado IPI flow for building the hardware platform and Xilinx Yocto Petalinux flow for software design. It will use Xilinx IP and Software driver to demonstrate the capabilities of different components.

The Vitis platform will be created from the Vivado/PetaLInux build artifacts, and then using the Vitis acceleration flow will be used to insert the DPU into the platform to create the final bitstream.

The following figure shows one of the use cases (Serial pipeline) with face detection with enhanced ROI on ZCU106.

Open

The following figure shows one of the use cases (streaming pipeline) with face detection with enhanced ROI on ZCU106.

Open

1.1 System Architecture

The following figure shows the block diagram of the ROI design

Open

1.2 Hardware Architecture

This section gives a detailed description of the blocks used in the hardware design. The functional block diagram of the design is shown in the below figure.

Open

There are seven primary Sections in the design.

• HDMI Capture Pipeline:

  • Captures video frame buffers from Capture source in 4K Resolution, NV12 Format

  •  Writes the buffers into DDR Memory with Frame Buffer Write IP

• Multi-scaler Block:

  • Reads the Video Buffers from DDR Memory

  • Scales down the buffer to the 640x360 size (suitable for dpu)

  • Converts the format from NV12 to BGR

  • Writes the Down-scaled buffer to DDR Memory

• DPU Block: 

  • Reads the downscaled buffers from DDR Memory

  • Runs the Densebox algorithm to generate the ROI information for each frame buffer

  • Passes the ROI information to VCU Encoder

• VCU Encoder: 

  • Reads the 4K NV12 Buffer from DDR Memory

  • Receives the ROI metadata from DPU IP

  • Encodes the video buffers based on the ROI Information

  • Finally writes the encoded stream to DDR Memory

• PS GEM:

  • Reads the Encoder stream from DDR Memory

  • Streams out the encoded stream via Ethernet

• VCU Decoder:

  • Decodes the received encoded frame and writes to memory

• HDMI-Tx:

  • Displays the decoded frames on HDMI Display

This design supports the following video interfaces:

1.3 VCU ROI Software

1.3.1 GStreamer Pipeline

The GStreamer plugin demonstrates the DPU capabilities with Xilinx VCU encoder’s ROI(Region of Interest) feature. The plugin will detect ROI (i.e. face co-ordinates) from input frames using DPU IP and pass the detected ROI information to the Xilinx VCU encoder. The following figure shows the data flow for GStreamer pipeline of stream-out use case.

Open

As shown in the above figure, the stream-out GStreamer pipeline performs the below list of operations:

1. v4l2src captures the data from HDMI-Rx in NV12 format and pass to xlnxroivideo1detect GStreamer plugin

2. xlnxroivideo1detect GStreamer plugin will scale down to 640x360 resolution and convert the data into BGR format

3. 640x360 BGR frame will be provided to DPU IP as an input to find ROI (i.e. face co-ordinates)

4. Extracted ROI information will be passed to VCU encoder

5. The encoder will encode the input data by encoding ROI regions with high quality as compared to non-ROI region using received ROI information

6. Stream-out the encoded data using RTP protocol

The following figure shows the data flow for the GStreamer pipeline of stream-in use cases.

Open

As shown in the above figure, the stream-in GStreamer pipeline performs the below list of operations:

1. Stream-in the encoded data using RTP protocol

2. The Xilinx VCU decoder will decode the data

3. Display the decoded data on HDMI-Tx display

The below figure shows the xlnxroivideo1detect GStreamer plugin data flow.

Open

As shown in the above figure, our xlnxroivideo1detect GStreamer plugin will perform below the list of operations:

1. DPU is initialized

2. DPU GStreamer plugin receives the data frame from HDMI-Rx through a v4l2src plugin

3. Create the DPU task

4. Scale the input frame to 640x360 resolution using Xilinx Scaler IP

5. Convert the input frame data format from NV12 to BGR format using Xilinx Color Space Converter(CSC) soft IP

6. Prepare the OpenCV image using BGR data

7. Pass the intermediate OpenCV image to the DPU

8. Run the DPU task

9. Extract the ROI(face) co-ordinates from the DPU output

10. Map the detected face co-ordinates to the original input frame resolution

11. Fill the ROI metadata buffer using extracted ROI (face) co-ordinates

12. Pass the ROI metadata buffer and input NV12 frame data buffer to the Xilinx VCU encoder

13. De-initialize the DPU task

1.3.2 DPU(Deep Learning Processor Unit)

DPU is a programmable engine optimized for deep neural networks. It is a group of parameterizable IP cores pre-implemented on the hardware with no place and route required. The DPU is released with the Vitis AI specialized instruction set, allowing efficient implementation of many deep learning networks.

Refer to DPU IP PG338 and UG1354 to know more details on DPU.

The following figure shows the DPU Top-Level Block Diagram.

Open

The DPU IP can be implemented in the programmable logic (PL) of the selected Zynq® UltraScale+™ MPSoC device with direct connections to the processing system (PS). The DPU requires instructions to implement a neural network and accessible memory locations for input images as well as temporary and output data. A program running on the application processing unit (APU) is also required to service interrupts and coordinate data transfers.

The following figure shows the sequence of operations performed on the DPU device.

Open

The following sequence of steps are performed to access and run face detection using the DPU device:

1. DPU device is initialized

2. Instantiate a DPU Task from DPU Kernel and allocate corresponding DPU memory buffer

3. Set the input image to created DPU task

4. Run the DPU task to find the faces from the input image

5. DPU device is uninitialized

1.4 Software Tools and System Requirements

Hardware

Required:

• ZCU106 evaluation board (rev C/D/E/F/1.0) with power cable

• Monitor with HDMI input supporting 3840x2160 resolution or 1920x1080 resolution

• HDMI cable 2.0 certified

• Class-10 SD card

• Ethernet cable

Optional:

• USB pen drive formatted with the FAT32 file system and hub

• SATA drive formatted with the FAT32 file system, external power supply, and data cable

Software Tools

Required:

• Linux host machine for all tool flow tutorials (see UG1144 for detailed OS requirements)

• PetaLinux Tools version 2020.2 (see UG1144 for installation instructions)

• Vivado Design suite version 2020.2

• Vitis Core Development Kit 2020.2

• Xilinx Runtime 2020.2

• Git a distributed version control system

• Serial terminal emulator e.g. teraterm

Download, Installation, and Licensing of Vivado Design Suite 2020.2

The Vivado Design Suite User Guide explains how to download and install the Vivado® Design Suite tools, which include the Vivado Integrated Design Environment (IDE), High-Level Synthesis tool, and System Generator for DSP. This guide also provides information about licensing and administering evaluation and full copies of Xilinx design tools and intellectual property (IP) products. The Vivado Design Suite can be downloaded from here.

LogiCORE IP Licensing

The following IP cores require a license to build the design.

• Video Mixer- Included with Vivado - PG243

• Video PHY Controller - Included with Vivado - PG203

• HDMI-Rx/Tx Subsystem - Purchase license (Hardware evaluation available) - PG235 & PG236

• Video Processing Subsystem (VPSS) - Included with Vivado - PG231

To obtain the LogiCORE IP license, please visit the respective IP product page and get the license.

• AR# 44029 - Licensing - LogiCORE IP Core licensing questions

• Xilinx Licensing FAQ

• LogiCORE IP Project License Terms

Compatibility

The reference design has been tested successfully with the following user-supplied components.

HDMI Monitor:

Cable:

• HDMI 2.0 compatible cable

 The below table provides the performance information:

1.5 Board Setup

The below section will provide the information on the ZCU106 board setup for running ROI design.

1. Connect the Micro USB cable into the ZCU106 Board Micro USB port J83, and the other end into an open USB port on the host PC. This cable is used for UART over USB communication.

2. Insert the SD card with the images copied into the SD card slot J100. Please find here how to prepare the SD card for a specific design.

3. Set the SW6 switches as shown in the below Figure. This configures the boot settings to boot from SD.

   Open

   

4. Connect 12V Power to the ZCU106 6-Pin Molex connector

5. Connect one end of HDMI cable to the board’s P7 stacked HDMI connector (lower port) and another end to HDMI source

6. Connect one end of HDMI cable to the board’s P7 stacked HDMI connector (upper port) and another end to the HDMI monitor

7. For a USB storage device, connect the USB hub along with the mouse. (Optional)

8. For SATA storage device, connect SATA data cable to SATA 3.0 port. (Optional)

9. Set up a terminal session between a PC COM port and the serial port on the evaluation board (See the Determine which COM to use to access the USB serial port on the ZCU106 board for more details).

10. Copy the VCU-HDMI-ROI images into the SD card and insert the SD card on the board

11. The below images will show how to connect interfaces on the ZCU106 board

Open

1.6 Run Flow

The VCU ROI TRD package is released with the source code, Vivado project, Petalinux BSP, and SD card image that enables the user to run the demonstration. It also includes the binaries necessary to configure and boot the ZCU106 board. Prior to running the steps mentioned in this wiki page, download the VCU ROI TRD package and extract its contents to a directory referred to as TRD_HOME which is the home directory.

Refer below link to download the VCU HDMI ROI TRD package.

• Zynq UltraScale+ MPSoC VCU TRD 2020.2 Download zip

TRD package contents are placed in the following directory structure.

The below snippet shows the directory structure of various configuration files which are useful for vcu_gst_app to run Display, and Streaming use cases. All these configurations files are placed in the TRD_HOME/image/sd_card/boot/config directory.

1.6.1 Preparing the SD card

There are three ways to prepare the SD card for booting. Each method is detailed below. 

Using ready to test image

• Flash SD Card with sd_card.img using Etcher or Win32DiskImager

• Boot the board with Flashed SD Card

Using Pre-built images

• To Create SD Card with two partitions: Boot(FAT32+Bootable) and Root(EXT4) Refer this Link.

• Copy boot content fromrdf0617-zcu106_vcu_hdmi_roi_2020_2/image/sd_card/boot to Boot partition in SD Card

• Extract rootfs.ext4 from rdf0617-zcu106_vcu_hdmi_roi_2020_2/image/sd_card/rootto Root partition in SD Card using

• Boot the board with Flashed SD Card

Use the Output of the Build Flow

• To Create SD Card with two partitions: Boot(FAT32+Bootable) and Root(EXT4) Refer this Link.

• For Build Flow refer this steps and copy mentioned generated dpu build images bd.hwh BOOT.BIN boot.scr dpu.xclbin Image system.dtb into BOOT partition of the SD card and extract generated rootfs.ext4 into ROOT partition of SD Card

• Copy the mentioned boot content config, vitis, autostart.sh, setup.sh from rdf0617-zcu106_vcu_hdmi_roi_2020_2/image/sd_card/boot/ directory to Boot partition in SD Card

• Boot the board with Flashed SD Card

1.6.2 Using GStreamer Application (vcu_gst_app)

The vcu_gst_app is a command-line multi-threaded Linux application. The command-line application requires an input configuration file (input.cfg) to be provided in the plain text.

Before Running vcu_gst_app pipelines run below modetest commands to set CRTC configurations.

Display: Capture → (ROI) → Encode → Decode → Display

• 4Kp30 HEVC Display Pipeline execution using vcu_gst_app

  vcu_gst_app /media/card/config/4kp30/Display/Single_4kp30_HEVC.cfg

Stream-out: ( Server )

• Set IP address for server:

  ifconfig eth0 192.168.25.90

• 4Kp30 HEVC Stream-out Pipeline execution using vcu_gst_app

  vcu_gst_app /media/card/config/4kp30/Stream-out/Single_4kp30_HEVC.cfg

Stream-in: ( Client )

• Set IP address for the client:

  ifconfig eth0 192.168.25.89

• 4Kp30 HEVC Stream-in Pipeline execution using vcu_gst_app

  vcu_gst_app /media/card/config/4kp30/Stream-in/input.cfg

1.6.3 GStreamer Pipelines using mediasrcbin plugin

This section covers the GSreamer pipelines using mediasrcbin plugin for serial and streaming ROI use-cases. This mediasrcbin plugin is Xilinx specific plugin which is a bin element on top of v4l2src. It parses and configures the media graph of a media device automatically.

Before Running Gstremer pipelines run below modetest commands to set CRTC configurations.

Serial: Capture → (ROI) → Encode → Decode → Display

• Run the following gst-launch-1.0 command for serial pipeline (capture → roi_plugin → encode → decode → display)

  • Serial Pipeline

    gst-launch-1.0 mediasrcbin media-device=/dev/media0 v4l2src0::io-mode=4 ! video/x-raw, width=3840, height=2160, format=NV12, framerate=30/1 ! xlnxroivideo1detect capture-io-mode=4 output-io-mode=5 relative-qp=-21 ! omxh265enc gop-mode=basic gop-length=60 b-frames=0 target-bitrate=1500 num-slices=8 control-rate=constant prefetch-buffer=true low-bandwidth=false filler-data=true cpb-size=1000 initial-delay=500 qp-mode=roi ! video/x-h265, profile=main, alignment=au ! queue ! omxh265dec internal-entropy-buffers=5 low-latency=0 split-input=true ! queue max-size-bytes=0 ! fpsdisplaysink name=fpssink text-overlay=false 'video-sink=kmssink bus-id=a00c0000.v_mix hold-extra-sample=1 show-preroll=false sync=true' sync=true -v

Stream-out ( Server ): Capture → (ROI) → Encode → Stream-out

• Set IP address for server:

  ifconfig eth0 192.168.25.90

• Run the following gst-launch-1.0 command for stream-out pipeline

  • Stream-out Pipeline

    gst-launch-1.0 mediasrcbin media-device=/dev/media0 v4l2src0::io-mode=4 ! video/x-raw, format=NV12, width=3840, height=2160, framerate=30/1 ! xlnxroivideo1detect capture-io-mode=4 output-io-mode=5 relative-qp=-21 ! omxh265enc gop-mode=basic gop-length=60 b-frames=0 target-bitrate=1500 num-slices=8 control-rate=constant prefetch-buffer=true low-bandwidth=false filler-data=true cpb-size=1000 initial-delay=500 periodicity-idr=60 qp-mode=roi ! video/x-h265, profile=main, alignment=au ! queue ! mpegtsmux alignment=7 name=mux ! rtpmp2tpay ! udpsink host=192.168.25.89 port=5004

Stream-in ( Client ): Stream-in→ Decode → Display

• Set IP address for the client:

  ifconfig eth0 192.168.25.89

• Run the following gst-launch-1.0 command for stream-in pipeline where 5004 is port number

  • Stream-in Pipeline

    gst-launch-1.0 udpsrc port=5004 buffer-size=60000000 caps="application/x-rtp, clock-rate=90000" ! rtpjitterbuffer latency=1000 ! rtpmp2tdepay ! tsparse ! video/mpegts ! tsdemux name=demux ! queue ! h265parse ! video/x-h265, profile=main, alignment=au ! omxh265dec internal-entropy-buffers=5 low-latency=0 split-input=true ! queue max-size-bytes=0 ! fpsdisplaysink name=fpssink text-overlay=false 'video-sink=kmssink bus-id=a00c0000.v_mix hold-extra-sample=1 show-preroll=false sync=true' sync=true -v

1.6.4 GStreamer Pipelines using v4l2src plugin

This section covers the GSreamer pipelines using v4l2src plugin for serial and streaming ROI use-cases. Before Running Gstremer pipelines run modetest command for HDMI-Tx and media-ctl command for HDMI-Rx as mentioned below.

Before Running Gstremer pipelines run below modetest commands to set CRTC configurations.

Run below media-ctl commands to set scaler configuration when HDMI Input Source is configured for 4Kp30 resolution and RBG888_1x24 color format.

Serial: Capture → (ROI) → Encode → Decode → Display

• Run the following gst-launch-1.0 command for serial pipeline (capture → roi_plugin → encode → decode → display)

  • Serial Pipeline

    gst-launch-1.0 v4l2src device=/dev/video0 io-mode=4 ! video/x-raw, width=3840, height=2160, format=NV12, framerate=30/1 ! xlnxroivideo1detect capture-io-mode=4 output-io-mode=5 relative-qp=-21 ! omxh265enc gop-mode=basic gop-length=60 b-frames=0 target-bitrate=1500 num-slices=8 control-rate=constant prefetch-buffer=true low-bandwidth=false filler-data=true cpb-size=1000 initial-delay=500 qp-mode=roi ! video/x-h265, profile=main, alignment=au ! queue ! omxh265dec internal-entropy-buffers=5 low-latency=0 split-input=true ! queue max-size-bytes=0 ! fpsdisplaysink name=fpssink text-overlay=false 'video-sink=kmssink bus-id=a00c0000.v_mix hold-extra-sample=1 show-preroll=false sync=true' sync=true -v

Stream-out ( Server ): Capture → (ROI) → Encode → Stream-out

• Set IP address for server:

  ifconfig eth0 192.168.25.90

• Run the following gst-launch-1.0 command for stream-out pipeline

  • Stream-out Pipeline

    gst-launch-1.0 v4l2src device=/dev/video0 io-mode=4 ! video/x-raw, format=NV12, width=3840, height=2160, framerate=30/1 ! xlnxroivideo1detect capture-io-mode=4 output-io-mode=5 relative-qp=-21 ! omxh265enc gop-mode=basic gop-length=60 b-frames=0 target-bitrate=1500 num-slices=8 control-rate=constant prefetch-buffer=true low-bandwidth=false filler-data=true cpb-size=1000 initial-delay=500 periodicity-idr=60 qp-mode=roi ! video/x-h265, profile=main, alignment=au ! queue ! mpegtsmux alignment=7 name=mux ! rtpmp2tpay ! udpsink host=192.168.25.89 port=5004

Stream-in ( Client ): Stream-in→ Decode → Display

• Set IP address for the client:

  ifconfig eth0 192.168.25.89

• Run the following gst-launch-1.0 command for stream-in pipeline where 5004 is port number

  • Stream-in Pipeline

    gst-launch-1.0 udpsrc port=5004 buffer-size=60000000 caps="application/x-rtp, clock-rate=90000" ! rtpjitterbuffer latency=1000 ! rtpmp2tdepay ! tsparse ! video/mpegts ! tsdemux name=demux ! queue ! h265parse ! video/x-h265, profile=main, alignment=au ! omxh265dec internal-entropy-buffers=5 low-latency=0 split-input=true ! queue max-size-bytes=0 ! fpsdisplaysink name=fpssink text-overlay=false 'video-sink=kmssink bus-id=a00c0000.v_mix hold-extra-sample=1 show-preroll=false sync=true' sync=true -v

1.7 Build Flow

Refer below link to download the VCU HDMI ROI TRD package.

• Zynq UltraScale+ MPSoC VCU TRD 2020.2 Download zip

Unzip the released package.

The following tutorials assume that the $TRD_HOME environment variable is set as given below.

1.7.1 Hardware Build Flow

This section explains the steps to build the hardware platform and generate XSA using the Vivado tool.

Refer to the Vivado Design Suite User Guide: Using the Vivado IDE, UG893, for setting up the Vivado environment.

Refer to the vivado-release-notes-install-license(UG973) for installation.

Make sure that the necessary IP licenses are in place

On Linux:

• Open a Linux terminal

• Change directory to $TRD_HOME/pl folder

• Source Vivado settings.sh

  bash source <path/to/Vivado-installer>/tool/Vivado/2020.2/settings64.sh

• Run the following command to create the Vivado IPI project and invoke the GUI and generate XSA required for the platform

  vivado -source ./designs/zcu106_ROI_HDMI/project.tcl

• The project.tcl script does the following

  • Creates project

  • Creates IPI Block design with platform interfaces

  • Runs Synthesis and Implementation

  • Builds bitstream with no accelerators

  • Export the HW to XSA

• zcu106_ROI_HDMI_wrapper.xsa is created

• stored in the directory ../zcu106_ROI_HDMI.xsa/

• at location TRD_HOME/pl/build/zcu106_ROI_HDMI/zcu106_ROI_HDMI.xsa/

• This XSA is used by Petalinux for platform creation and also by the Vitis Tool for DPU Kernel Integration.

After executing the script, the Vivado IPI block design comes up as shown in the below figure.

Open

1.7.1.1 Platform Interfaces