This page provides all the information related to VCU HDMI ROI designs for ZCU104.

This design is based on the ZCU106 VCU HDMI ROI TRD which can be found at the following link
- Zynq UltraScale+ MPSoC ZCU106 VCU HDMI ROI TRD 2020.2
The two designs are functionally equivalent, please refer to the page linked above for details related to the system architecture, gstreamer plugin info, DPU configuration and usage, etc. The sections below provide direct links to the relevant information.

1 1 Overview
- 1.1 1.1 System Architecture
- 1.2 1.2 Hardware Architecture
- 1.3 1.3 VCU ROI Software
  - 1.3.1 1.3.1 GStreamer Pipeline
  - 1.3.2 1.3.2 DPU(Deep Learning Processor Unit)
- 1.4 1.4 Software Tools and System Requirements
- 1.5 1.5 Board Setup
- 1.6 1.6 Run Flow
- 1.7 1.7 Build Flow
2 3 Other Information
- 2.1 3.1 Known Issues
- 2.2 3.2 Limitations
3 4 Appendix A - HDMI-Rx Link-up

1 Overview

The primary goal of this VCU ROI design is to demonstrate the use of DPU (Deep learning Processor Unit) block for extracting the ROI (Region of Interest) from input video frames and to use this information to perform ROI based encoding using VCU (Video Codec Unit) encoder hard block present in Zynq UltraScale+ EV devices.

For detailed overview please refer to this link.

1.1 System Architecture

For system architecture please refer to this link. System architecture diagram shown in above refer link is applicable for ZCU104 board.

1.2 Hardware Architecture

This section gives a detailed description of the blocks used in the hardware design.

For detailed hardware architecture please refer to this link.

1.3 VCU ROI Software

1.3.1 GStreamer Pipeline

The GStreamer plugin demonstrates the DPU capabilities with Xilinx VCU encoder’s ROI(Region of Interest) feature. The plugin will detect ROI (i.e. face co-ordinates) from input frames using DPU IP and pass the detected ROI information to the Xilinx VCU encoder.

For detailed VCU ROI software please refer to this link.

1.3.2 DPU(Deep Learning Processor Unit)

DPU is a programmable engine optimized for deep neural networks. It is a group of parameterizable IP cores pre-implemented on the hardware with no place and route required. The DPU is released with the Vitis AI specialized instruction set, allowing efficient implementation of many deep learning networks.

For detailed Deep Learning Processor Unit please refer to this link.

1.4 Software Tools and System Requirements

Hardware

Required:

ZCU104 evaluation board rev 1.0 with power cable
Monitor with HDMI input supporting 3840x2160 resolution or 1920x1080 resolution
HDMI Receiver - NVIDIA SHIELD Pro for 4Kp60
Two HDMI cables 2.0 certified
Class-10 Micro-SD card (ZCU104)
Ethernet cable

Optional:

USB pen drive formatted with the FAT32 file system and hub
SATA drive formatted with the FAT32 file system, external power supply, and data cable

Software Tools

Required:

Linux host machine for all tool flow tutorials (see UG1144 for detailed OS requirements)
PetaLinux Tools version 2020.2 (see UG1144 for installation instructions)
Vivado Design suite version 2020.2
Vitis Core Development Kit 2020.2
Xilinx Runtime 2020.2
Git a distributed version control system
Serial terminal emulator e.g. teraterm

Download, Installation, and Licensing of Vivado Design Suite 2020.2

The Vivado Design Suite User Guide explains how to download and install the Vivado® Design Suite tools, which include the Vivado Integrated Design Environment (IDE), High-Level Synthesis tool, and System Generator for DSP. This guide also provides information about licensing and administering evaluation and full copies of Xilinx design tools and intellectual property (IP) products. The Vivado Design Suite can be downloaded from here.

LogiCORE IP Licensing

The following IP cores require a license to build the design.

Video Mixer- Included with Vivado - PG243
Video PHY Controller - Included with Vivado - PG230
HDMI-Rx/Tx Subsystem - Purchase license (Hardware evaluation available) - PG235 & PG236
Video Processing Subsystem (VPSS) - Included with Vivado - PG231

To obtain the LogiCORE IP license, please visit the respective IP product page and get the license.

AR# 44029 - Licensing - LogiCORE IP Core licensing questions
Xilinx Licensing FAQ
LogiCORE IP Project License Terms

Compatibility

The reference design has been tested successfully with the following user-supplied components.

HDMI Monitor:

Make/Model	Resolutions
LG 24UD58	3840 x 2160 @ 30Hz
Samsung LU28E90DS/XL	3840 x 2160 @ 30Hz

Cable:

HDMI 2.0 compatible cable

1.5 Board Setup

The below section will provide the information on the ZCU104 boards setup for running ROI design.

ZCU104 - Connect the Micro USB cable into the ZCU104 Board Micro USB port J164, and the other end into an open USB port on the host PC. This cable is used for UART over USB communication.
Insert the SD card with the images copied into the SD card slot J100. Please find here how to prepare the SD card for a specific design.
Set the SW6 switches as shown in the below Figure. This configures the boot settings to boot from SD.
Connect 12V Power to the 6-Pin Molex connector on the board
Connect one end of HDMI cable to the board’s P7 stacked HDMI connector (lower port) and another end to HDMI source
Connect one end of HDMI cable to the board’s P7 stacked HDMI connector (upper port) and another end to the HDMI monitor
For a USB storage device, connect the USB hub along with the mouse. (Optional)
For SATA storage device, connect SATA data cable to SATA 3.0 port. (Optional)
Set up a terminal session between a PC COM port and the serial port on the evaluation board (See the Determine which COM to use to access the USB serial port on the ZCU104 board for more details).
The below images will show how to connect interfaces on the ZCU104 board

ZCU104 Board

1.6 Run Flow

The VCU ROI packages released with the source code, Vivado project, Petalinux BSP, and SD card image that enables the user to run the demonstration. It also includes the binaries necessary to configure and boot the ZCU104 board. Prior to running the steps mentioned in this wiki page, download the desired VCU ROI package and extract its contents to a directory referred to as TRD_HOME which is the home directory.

For ZCU104 Board:

Refer below link to download the VCU HDMI ROI package
- Zynq UltraScale+ MPSoC VCU ROI 2020.2 Download zip

TRD package contents are placed in the following directory structure for ZCU104

rdf0428-zcu104-vcu-hdmi-roi-2020-2
├── apu
│   └── vcu_petalinux_bsp
│       └── xilinx-vcu-roi-zcu104-v2020.2-final.bsp
├── dpu
│   ├── 0001-Added-ZCU104-configuration-to-support-DPU-in-ZCU104.patch
│   ├── dpu_conf.vh
│   └── vitis_platform
│       └── zcu104_dpu
├── image
│   ├── bootfiles
│   │   ├── bl31.elf
│   │   ├── linux.bif
│   │   ├── pmufw.elf
│   │   ├── system.bit
│   │   ├── system.dtb
│   │   ├── u-boot.elf
│   │   └── zynqmp_fsbl.elf
│   ├── README.txt
│   ├── sd_card
│   │   ├── boot
│   │   └── root
│   └── sd_card.img
└── pl
    ├── constrs
    │   ├── roi_async.xdc
    │   └── vcu_zcu104_roi.xdc
    ├── designs
    │   └── zcu104_ROI_HDMI
    ├── prebuild
    │   └── zcu104_ROI_HDMI.xsa
    ├── README.md
    └── srcs
        ├── hdl
        └── ip

1.6.1 Preparing the SD card

There are three ways to prepare the SD card for booting. Each method is detailed below.

Using ready to test image

Flash SD Card with sd_card.img using Etcher or Win32DiskImager
Boot the board with Flashed SD Card

sd_card.img is available at rdf0428-zcu104-vcu-hdmi-roi-2020-2/image/sd_card.img
All the required Vitis packages are already installed in ready to test rdf0428-zcu104-vcu-hdmi-roi-2020-2/image/sd_card.img

Using Pre-built images

To Create SD Card with two partitions: Boot(FAT32+Bootable) and Root(EXT4) Refer this Link.
Copy boot content fromrdf0428-zcu104-vcu-hdmi-roi-2020-2/image/sd_card/boot to Boot partition in SD Card
Extract rootfs.ext4 from rdf0428-zcu104-vcu-hdmi-roi-2020-2/image/sd_card/rootto Root partition in SD Card using
Boot the board with Flashed SD Card

Use the Output of the Build Flow

To Create SD Card with two partitions: Boot(FAT32+Bootable) and Root(EXT4) Refer this Link.
For Build Flow refer this steps and copy mentioned generated dpu build images bd.hwh BOOT.BIN boot.scr dpu.xclbin Image system.dtb into BOOT partition of the SD card and extract generated rootfs.ext4 into ROOT partition of SD Card
Copy the mentioned boot content config, vitis, start.sh, setup.sh from rdf0428-zcu104-vcu-hdmi-roi-2020-2/image/sd_card/boot/ directory to Boot partition in SD Card
Boot the board with Flashed SD Card

All the required Vitis AI packages are already available in rdf0428-zcu104-vcu-hdmi-roi-2020-2/image/sd_card/boot/vitis directory which needs to be installed using start.sh script after boot-up. Please wait till target setup complete and Vitis AI packages are installed.

1.6.2 GStreamer Pipelines using mediasrcbin plugin

This section covers the GSreamer pipelines using mediasrcbin plugin for serial and streaming ROI use-cases. This mediasrcbin plugin is Xilinx specific plugin which is a bin element on top of v4l2src. It parses and configures the media graph of a media device automatically.

Before Running Gstremer pipelines run below modetest commands to set 4Kp30 video mixer configurations for HDMI-Tx.

First modetest command is used to set resolution of CRTC and alpha property for primary plane
Second modetest command is used to get the blue screen once the alpha property for primary plane is set to

modetest -D a00c0000.v_mix -s 40:3840x2160-30@AR24 -w 36:"alpha":0 &
modetest -D a00c0000.v_mix -s 40:3840x2160-30@AR24

Serial: Capture → (ROI) → Encode → Decode → Display

Run the following gst-launch-1.0 command for serial pipeline (capture → roi_plugin → encode → decode → display)
- Serial Pipeline
  gst-launch-1.0 mediasrcbin media-device=/dev/media0 v4l2src0::io-mode=4 ! video/x-raw, width=3840, height=2160, format=NV12, framerate=30/1 ! xlnxroivideo1detect capture-io-mode=4 output-io-mode=5 relative-qp=-21 ! omxh265enc gop-mode=basic gop-length=60 b-frames=0 target-bitrate=1500 num-slices=8 control-rate=constant prefetch-buffer=true low-bandwidth=false filler-data=true cpb-size=1000 initial-delay=500 qp-mode=roi ! video/x-h265, profile=main, alignment=au ! queue ! omxh265dec internal-entropy-buffers=5 low-latency=0 split-input=true ! queue max-size-bytes=0 ! fpsdisplaysink name=fpssink text-overlay=false 'video-sink=kmssink bus-id=a00c0000.v_mix hold-extra-sample=1 show-preroll=false sync=true' sync=true -v

Stream-out ( Server ): Capture → (ROI) → Encode → Stream-out

Set IP address for server:
ifconfig eth0 192.168.25.90
Run the following gst-launch-1.0 command for stream-out pipeline
- Stream-out Pipeline
  gst-launch-1.0 mediasrcbin media-device=/dev/media0 v4l2src0::io-mode=4 ! video/x-raw, format=NV12, width=3840, height=2160, framerate=30/1 ! xlnxroivideo1detect capture-io-mode=4 output-io-mode=5 relative-qp=-21 ! omxh265enc gop-mode=basic gop-length=60 b-frames=0 target-bitrate=1500 num-slices=8 control-rate=constant prefetch-buffer=true low-bandwidth=false filler-data=true cpb-size=1000 initial-delay=500 periodicity-idr=60 qp-mode=roi ! video/x-h265, profile=main, alignment=au ! queue ! mpegtsmux alignment=7 name=mux ! rtpmp2tpay ! udpsink host=192.168.25.89 port=5004

Here 192.168.25.89 is host/client IP address and 5004 is port no.
Make sure HDMI-Rx source should be configured to 4Kp30 mode

Stream-in ( Client ): Stream-in→ Decode → Display

Set IP address for the client:
ifconfig eth0 192.168.25.89
Run the following gst-launch-1.0 command for stream-in pipeline where 5004 is port number
- Stream-in Pipeline
  gst-launch-1.0 udpsrc port=5004 buffer-size=60000000 caps="application/x-rtp, clock-rate=90000" ! rtpjitterbuffer latency=1000 ! rtpmp2tdepay ! tsparse ! video/mpegts ! tsdemux name=demux ! queue ! h265parse ! video/x-h265, profile=main, alignment=au ! omxh265dec internal-entropy-buffers=5 low-latency=0 split-input=true ! queue max-size-bytes=0 ! fpsdisplaysink name=fpssink text-overlay=false 'video-sink=kmssink bus-id=a00c0000.v_mix hold-extra-sample=1 show-preroll=false sync=true' sync=true -v

For all 1080p30 commands replace width from 3840 to 1920 and height from 2160 to 1080

1.6.3 GStreamer Pipelines using v4l2src plugin

This section covers the GSreamer pipelines using v4l2src plugin for serial and streaming ROI use-cases. Before Running Gstremer pipelines run modetest command for HDMI-Tx and media-ctl command for HDMI-Rx as mentioned below.

Before Running Gstremer pipelines run below modetest commands to set 4Kp30 video mixer configurations for HDMI-Tx.

First modetest command is used to set resolution of CRTC and alpha property for primary plane
Second modetest command is used to get the blue screen once the alpha property for primary plane is set to

modetest -D a00c0000.v_mix -s 40:3840x2160-30@AR24 -w 36:"alpha":0 &
modetest -D a00c0000.v_mix -s 40:3840x2160-30@AR24

Run below media-ctl commands to set scaler configuration when HDMI Input Source is configured for 4Kp30 resolution and RBG888_1x24 color format.

media-ctl -d /dev/media0 -V "\"a0040000.v_proc_ss\":0 [fmt:RBG888_1X24/3840x2160 field:none]"
media-ctl -d /dev/media0 -V "\"a0040000.v_proc_ss\":1 [fmt:VYYUYY8_1X24/3840x2160 field:none]"

Make sure HDMI-Rx is configured for 4Kp resolution and RBG888_1x24 color format
v_proc_ss:0 should match the HDMI-Rx input source resolution and color format, and that the v_proc_ss:1 should match the Gstremer Pipeline resolution and color format