Zynq UltraScale+ MPSoC ZCU104 VCU HDMI ROI 2020.2

This page provides all the information related to VCU HDMI ROI designs for ZCU104.

  • This design is based on the ZCU106 VCU HDMI ROI TRD which can be found at the following link

  • The two designs are functionally equivalent, please refer to the page linked above for details related to the system architecture, gstreamer plugin info, DPU configuration and usage, etc.  The sections below provide direct links to the relevant information.

Table of Contents

1 Overview

The primary goal of this VCU ROI design is to demonstrate the use of DPU (Deep learning Processor Unit) block for extracting the ROI (Region of Interest) from input video frames and to use this information to perform ROI based encoding using VCU (Video Codec Unit) encoder hard block present in Zynq UltraScale+ EV devices.

  • For detailed overview please refer to this link.

1.1 System Architecture

  • For system architecture please refer to this link. System architecture diagram shown in above refer link is applicable for ZCU104 board.

1.2 Hardware Architecture

This section gives a detailed description of the blocks used in the hardware design. 

  • For detailed hardware architecture please refer to this link.

1.3 VCU ROI Software

1.3.1 GStreamer Pipeline

The GStreamer plugin demonstrates the DPU capabilities with Xilinx VCU encoder’s ROI(Region of Interest) feature. The plugin will detect ROI (i.e. face co-ordinates) from input frames using DPU IP and pass the detected ROI information to the Xilinx VCU encoder.

  • For detailed VCU ROI software please refer to this link.

1.3.2 DPU(Deep Learning Processor Unit)

DPU is a programmable engine optimized for deep neural networks. It is a group of parameterizable IP cores pre-implemented on the hardware with no place and route required. The DPU is released with the Vitis AI specialized instruction set, allowing efficient implementation of many deep learning networks.

  • For detailed Deep Learning Processor Unit please refer to this link.

1.4 Software Tools and System Requirements

Hardware

Required:

  • ZCU104 evaluation board rev 1.0 with power cable

  • Monitor with HDMI input supporting 3840x2160 resolution or 1920x1080 resolution

  • HDMI Receiver - NVIDIA SHIELD Pro for 4Kp60

  • Two HDMI cables 2.0 certified

  • Class-10 Micro-SD card (ZCU104)

  • Ethernet cable

Optional:

  • USB pen drive formatted with the FAT32 file system and hub

  • SATA drive formatted with the FAT32 file system, external power supply, and data cable

Software Tools

Required:

Download, Installation, and Licensing of Vivado Design Suite 2020.2

The Vivado Design Suite User Guide explains how to download and install the Vivado® Design Suite tools, which include the Vivado Integrated Design Environment (IDE), High-Level Synthesis tool, and System Generator for DSP. This guide also provides information about licensing and administering evaluation and full copies of Xilinx design tools and intellectual property (IP) products. The Vivado Design Suite can be downloaded from here.

LogiCORE IP Licensing 

The following IP cores require a license to build the design.

  • Video Mixer- Included with Vivado - PG243

  • Video PHY Controller - Included with Vivado - PG230

  • HDMI-Rx/Tx Subsystem - Purchase license (Hardware evaluation available) - PG235 & PG236

  • Video Processing Subsystem (VPSS) - Included with Vivado - PG231

To obtain the LogiCORE IP license, please visit the respective IP product page and get the license.

Compatibility

The reference design has been tested successfully with the following user-supplied components.

HDMI Monitor:

Make/Model

Resolutions

LG 24UD58

3840 x 2160 @ 30Hz

Samsung LU28E90DS/XL

3840 x 2160 @ 30Hz

Cable:

  • HDMI 2.0 compatible cable

1.5 Board Setup

The below section will provide the information on the ZCU104 boards setup for running ROI design.

  1. ZCU104 - Connect the Micro USB cable into the ZCU104 Board Micro USB port J164, and the other end into an open USB port on the host PC. This cable is used for UART over USB communication.

  2. Insert the SD card with the images copied into the SD card slot J100. Please find here how to prepare the SD card for a specific design.

  3. Set the SW6 switches as shown in the below Figure. This configures the boot settings to boot from SD.

  4. Connect 12V Power to the 6-Pin Molex connector on the board

  5. Connect one end of HDMI cable to the board’s P7 stacked HDMI connector (lower port) and another end to HDMI source

  6. Connect one end of HDMI cable to the board’s P7 stacked HDMI connector (upper port) and another end to the HDMI monitor

  7. For a USB storage device, connect the USB hub along with the mouse. (Optional)

  8. For SATA storage device, connect SATA data cable to SATA 3.0 port. (Optional)

  9. Set up a terminal session between a PC COM port and the serial port on the evaluation board (See the Determine which COM to use to access the USB serial port on the ZCU104 board for more details).

  10. The below images will show how to connect interfaces on the ZCU104 board

ZCU104 Board

1.6 Run Flow

The VCU ROI packages released with the source code, Vivado project, Petalinux BSP, and SD card image that enables the user to run the demonstration. It also includes the binaries necessary to configure and boot the ZCU104 board. Prior to running the steps mentioned in this wiki page, download the desired VCU ROI package and extract its contents to a directory referred to as TRD_HOME which is the home directory.

For ZCU104 Board:

TRD package contents are placed in the following directory structure for ZCU104

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 rdf0428-zcu104-vcu-hdmi-roi-2020-2 ├── apu │   └── vcu_petalinux_bsp │   └── xilinx-vcu-roi-zcu104-v2020.2-final.bsp ├── dpu │   ├── 0001-Added-ZCU104-configuration-to-support-DPU-in-ZCU104.patch │   ├── dpu_conf.vh │   └── vitis_platform │   └── zcu104_dpu ├── image │   ├── bootfiles │   │   ├── bl31.elf │   │   ├── linux.bif │   │   ├── pmufw.elf │   │   ├── system.bit │   │   ├── system.dtb │   │   ├── u-boot.elf │   │   └── zynqmp_fsbl.elf │   ├── README.txt │   ├── sd_card │   │   ├── boot │   │   └── root │   └── sd_card.img └── pl ├── constrs │   ├── roi_async.xdc │   └── vcu_zcu104_roi.xdc ├── designs │   └── zcu104_ROI_HDMI ├── prebuild │   └── zcu104_ROI_HDMI.xsa ├── README.md └── srcs ├── hdl └── ip

1.6.1 Preparing the SD card

There are three ways to prepare the SD card for booting. Each method is detailed below. 

Using ready to test image

  • sd_card.img is available at rdf0428-zcu104-vcu-hdmi-roi-2020-2/image/sd_card.img

  • All the required Vitis packages are already installed in ready to test rdf0428-zcu104-vcu-hdmi-roi-2020-2/image/sd_card.img

Using Pre-built images

  • To Create SD Card with two partitions: Boot(FAT32+Bootable) and Root(EXT4) Refer this Link.

  • Copy boot content fromrdf0428-zcu104-vcu-hdmi-roi-2020-2/image/sd_card/boot to Boot partition in SD Card

  • Extract rootfs.ext4 from rdf0428-zcu104-vcu-hdmi-roi-2020-2/image/sd_card/rootto Root partition in SD Card using

  • Boot the board with Flashed SD Card

Use the Output of the Build Flow

  • To Create SD Card with two partitions: Boot(FAT32+Bootable) and Root(EXT4) Refer this Link.

  • For Build Flow refer this steps and copy mentioned generated dpu build images bd.hwh BOOT.BIN boot.scr dpu.xclbin Image system.dtb into BOOT partition of the SD card and extract generated rootfs.ext4 into ROOT partition of SD Card

  • Copy the mentioned boot content config, vitis, start.sh, setup.sh from rdf0428-zcu104-vcu-hdmi-roi-2020-2/image/sd_card/boot/ directory to Boot partition in SD Card

  • Boot the board with Flashed SD Card

All the required Vitis AI packages are already available in rdf0428-zcu104-vcu-hdmi-roi-2020-2/image/sd_card/boot/vitis directory which needs to be installed using start.sh script after boot-up. Please wait till target setup complete and Vitis AI packages are installed.

1.6.2 GStreamer Pipelines using mediasrcbin plugin

This section covers the GSreamer pipelines using mediasrcbin plugin for serial and streaming ROI use-cases. This mediasrcbin plugin is Xilinx specific plugin which is a bin element on top of v4l2src. It parses and configures the media graph of a media device automatically.

Before Running Gstremer pipelines run below modetest commands to set 4Kp30 video mixer configurations for HDMI-Tx. 

  • First modetest command is used to set resolution of CRTC and alpha property for primary plane

  • Second modetest command is used to get the blue screen once the alpha property for primary plane is set to

1 2 modetest -D a00c0000.v_mix -s 40:3840x2160-30@AR24 -w 36:"alpha":0 & modetest -D a00c0000.v_mix -s 40:3840x2160-30@AR24

Serial: Capture → (ROI) → Encode → Decode → Display

  • Run the following gst-launch-1.0 command for serial pipeline (capture → roi_plugin → encode → decode → display)

    • Serial Pipeline

      1 gst-launch-1.0 mediasrcbin media-device=/dev/media0 v4l2src0::io-mode=4 ! video/x-raw, width=3840, height=2160, format=NV12, framerate=30/1 ! xlnxroivideo1detect capture-io-mode=4 output-io-mode=5 relative-qp=-21 ! omxh265enc gop-mode=basic gop-length=60 b-frames=0 target-bitrate=1500 num-slices=8 control-rate=constant prefetch-buffer=true low-bandwidth=false filler-data=true cpb-size=1000 initial-delay=500 qp-mode=roi ! video/x-h265, profile=main, alignment=au ! queue ! omxh265dec internal-entropy-buffers=5 low-latency=0 split-input=true ! queue max-size-bytes=0 ! fpsdisplaysink name=fpssink text-overlay=false 'video-sink=kmssink bus-id=a00c0000.v_mix hold-extra-sample=1 show-preroll=false sync=true' sync=true -v

Stream-out ( Server ): Capture → (ROI) → Encode → Stream-out

  • Set IP address for server:

    1 ifconfig eth0 192.168.25.90
  • Run the following gst-launch-1.0 command for stream-out pipeline

    • Stream-out Pipeline

      1 gst-launch-1.0 mediasrcbin media-device=/dev/media0 v4l2src0::io-mode=4 ! video/x-raw, format=NV12, width=3840, height=2160, framerate=30/1 ! xlnxroivideo1detect capture-io-mode=4 output-io-mode=5 relative-qp=-21 ! omxh265enc gop-mode=basic gop-length=60 b-frames=0 target-bitrate=1500 num-slices=8 control-rate=constant prefetch-buffer=true low-bandwidth=false filler-data=true cpb-size=1000 initial-delay=500 periodicity-idr=60 qp-mode=roi ! video/x-h265, profile=main, alignment=au ! queue ! mpegtsmux alignment=7 name=mux ! rtpmp2tpay ! udpsink host=192.168.25.89 port=5004
  • Here 192.168.25.89 is host/client IP address and 5004 is port no.

  • Make sure HDMI-Rx source should be configured to 4Kp30 mode

Stream-in ( Client ): Stream-in→ Decode → Display

  • Set IP address for the client:

    1 ifconfig eth0 192.168.25.89
  • Run the following gst-launch-1.0 command for stream-in pipeline where 5004 is port number

    • Stream-in Pipeline

      1 gst-launch-1.0 udpsrc port=5004 buffer-size=60000000 caps="application/x-rtp, clock-rate=90000" ! rtpjitterbuffer latency=1000 ! rtpmp2tdepay ! tsparse ! video/mpegts ! tsdemux name=demux ! queue ! h265parse ! video/x-h265, profile=main, alignment=au ! omxh265dec internal-entropy-buffers=5 low-latency=0 split-input=true ! queue max-size-bytes=0 ! fpsdisplaysink name=fpssink text-overlay=false 'video-sink=kmssink bus-id=a00c0000.v_mix hold-extra-sample=1 show-preroll=false sync=true' sync=true -v

For all 1080p30 commands replace width from 3840 to 1920 and height from 2160 to 1080

1.6.3 GStreamer Pipelines using v4l2src plugin

This section covers the GSreamer pipelines using v4l2src plugin for serial and streaming ROI use-cases. Before Running Gstremer pipelines run modetest command for HDMI-Tx and media-ctl command for HDMI-Rx as mentioned below.

Before Running Gstremer pipelines run below modetest commands to set 4Kp30 video mixer configurations for HDMI-Tx. 

  • First modetest command is used to set resolution of CRTC and alpha property for primary plane

  • Second modetest command is used to get the blue screen once the alpha property for primary plane is set to

1 2 modetest -D a00c0000.v_mix -s 40:3840x2160-30@AR24 -w 36:"alpha":0 & modetest -D a00c0000.v_mix -s 40:3840x2160-30@AR24

Run below media-ctl commands to set scaler configuration when HDMI Input Source is configured for 4Kp30 resolution and RBG888_1x24 color format.

1 2 media-ctl -d /dev/media0 -V "\"a0040000.v_proc_ss\":0 [fmt:RBG888_1X24/3840x2160 field:none]" media-ctl -d /dev/media0 -V "\"a0040000.v_proc_ss\":1 [fmt:VYYUYY8_1X24/3840x2160 field:none]"
  • Make sure HDMI-Rx is configured for 4Kp resolution and RBG888_1x24 color format

  • v_proc_ss:0 should match the HDMI-Rx input source resolution and color format, and that the v_proc_ss:1 should match the Gstremer Pipeline resolution and color format

Serial: Capture → (ROI) → Encode → Decode → Display

  • Run the following gst-launch-1.0 command for serial pipeline (capture → roi_plugin → encode → decode → display)

    • Serial Pipeline

      1 gst-launch-1.0 v4l2src device=/dev/video0 io-mode=4 ! video/x-raw, width=3840, height=2160, format=NV12, framerate=30/1 ! xlnxroivideo1detect capture-io-mode=4 output-io-mode=5 relative-qp=-21 ! omxh265enc gop-mode=basic gop-length=60 b-frames=0 target-bitrate=1500 num-slices=8 control-rate=constant prefetch-buffer=true low-bandwidth=false filler-data=true cpb-size=1000 initial-delay=500 qp-mode=roi ! video/x-h265, profile=main, alignment=au ! queue ! omxh265dec internal-entropy-buffers=5 low-latency=0 split-input=true ! queue max-size-bytes=0 ! fpsdisplaysink name=fpssink text-overlay=false 'video-sink=kmssink bus-id=a00c0000.v_mix hold-extra-sample=1 show-preroll=false sync=true' sync=true -v

Stream-out ( Server ): Capture → (ROI) → Encode → Stream-out

  • Set IP address for server:

    1 ifconfig eth0 192.168.25.90
  • Run the following gst-launch-1.0 command for stream-out pipeline

    • Stream-out Pipeline

      1 gst-launch-1.0 v4l2src device=/dev/video0 io-mode=4 ! video/x-raw, format=NV12, width=3840, height=2160, framerate=30/1 ! xlnxroivideo1detect capture-io-mode=4 output-io-mode=5 relative-qp=-21 ! omxh265enc gop-mode=basic gop-length=60 b-frames=0 target-bitrate=1500 num-slices=8 control-rate=constant prefetch-buffer=true low-bandwidth=false filler-data=true cpb-size=1000 initial-delay=500 periodicity-idr=60 qp-mode=roi ! video/x-h265, profile=main, alignment=au ! queue ! mpegtsmux alignment=7 name=mux ! rtpmp2tpay ! udpsink host=192.168.25.89 port=5004
  • Here 192.168.25.89 is host/client IP address and 5004 is port no.

  • Make sure HDMI-Rx source should be configured to 4Kp30 mode

Stream-in ( Client ): Stream-in→ Decode → Display

  • Set IP address for the client:

    1 ifconfig eth0 192.168.25.89
  • Run the following gst-launch-1.0 command for stream-in pipeline where 5004 is port number

    • Stream-in Pipeline

      1 gst-launch-1.0 udpsrc port=5004 buffer-size=60000000 caps="application/x-rtp, clock-rate=90000" ! rtpjitterbuffer latency=1000 ! rtpmp2tdepay ! tsparse ! video/mpegts ! tsdemux name=demux ! queue ! h265parse ! video/x-h265, profile=main, alignment=au ! omxh265dec internal-entropy-buffers=5 low-latency=0 split-input=true ! queue max-size-bytes=0 ! fpsdisplaysink name=fpssink text-overlay=false 'video-sink=kmssink bus-id=a00c0000.v_mix hold-extra-sample=1 show-preroll=false sync=true' sync=true -v

For all 1080p30 commands replace width from 3840 to 1920 and height from 2160 to 1080

1.7 Build Flow

  • For detailed Build flow please refer to this link.

  • For ZCU104 Board, replace zcu106 with zcu104 in above refer link.

3 Other Information

3.1 Known Issues

3.2 Limitations

4 Appendix A - HDMI-Rx Link-up

  • For Appendix A please refer this link