The pages provides detailed instructions for running the VCU Region of Intrest (ROI) Demo on the ZCU104 and ZCU106 boards running Certified Ubuntu 20.04 LTS for Xilinx Devices.

Table of Contents

Introduction

The primary goal of the VCU ROI design is to demonstrate the use of the Xilinx Deep learning Processor Unit (DPU) block for extracting the Region of Interest (ROI) data from input video frames.  This information is then used to perform ROI-based encoding using the Video Codec Unit (VCU) encoder hard block present in Zynq UltraScale+ EV devices.

To run the VCU ROI TRD demo you'll need to connect an HDMI input source to the  bottom HDMI connector on the board, and an HDMI Monitor to the top HDMI connector.  The 2020.2 VCU HDMI ROI TRD wiki page has more information about the VCU ROI TRD and setting up the board. 

The following figure outlines the “Serial Pipeline” usage model with face detection based on ROI detection on ZCU106. This use case is the foundation of this section. 

Running the Demo

There are two ways to run  the VCU ROI TRD Demo:

• Using discrete commands

• Using the xlnx-vcu-roi-trd snap from the Xilinx Snap Store on Snapcraft.io. (coming soon!)

Using Discrete Commands

This demonstration assumes that both the HDMI source and HDMI monitor run at 1080p30 (1920x1080 @ 30fps).

Before getting started, be sure to do the following:

1. Set the HDMI output resolution

2. Launch the gstreamer pipeline

Depending on whether or not you have a USB camera plugged into your system, /dev/media0 in the following steps may need to change to /dev/media1. Use the command media-ctl -p -d /dev/media[X] to determine which device to use for HDMI Rx. 

Setting the HDMI Output Resolution

It is important to set up the video pipeline with a resolution and refresh rate that matches something your monitor is able to support. In order to find out which resolutions your monitor can support, use the modetest command with the syntax below. It is important to use the device a00c0000.v_mix as this corresponds to the video mixer IP.

sudo modetest -cD a00c0000.v_mix | head -n 32

Sample output from this command appears below. Note that in Connector #0, the supported resolution is 1920x1080@60.00Hz. The output of your monitor’s connector section will likely be different.

From the command line, issue the following commands to set the HDMI Monitor to 1080p30 in the AR24 color space. The AR24 color space is a good default to assume. Other common alternatives are BG24 and NV12.

sudo modetest -D a00c0000.v_mix -s 40:1920x1080-30@AR24

The -s flag actively sets the mode with the syntax -s <connector_id>[,<connector_id>][@<crtc_id>]:[#<mode index>]<mode>[-<vrefresh>][@<format>]. In our example, 40: is the connector ID, 1920x1080-30 is the resolution and refresh rate, and @AR24 is the color space mode. For more details, consult the output of modetest --help.

If you are having trouble finding a valid resolution and color space, re-run the modetest command without piping the output to the head command in order to get the full output, including a list of supported color spaces.

Once you have found a valid resolution and color space, you should see the default video test pattern displayed on the screen after issuing the modetest command.

Next, set the display to initialize the display and wait for further input:

sudo modetest -D a00c0000.v_mix -s 40:1920x1080-30@AR24 -w 36:"alpha":0 &

The -w 36:”alpha”:0 set the alpha layer to 0 on object ID 36. This can be seen in the full output of the modetest command:

36      38      46      0,0             0,0     0               0x00000001
  formats: AR24
  props:
        9 type:
                flags: immutable enum
                enums: Overlay=0 Primary=1 Cursor=2
                value: 1
        33 alpha:
                flags: range
                values: 0 256
                value: 256

The ampersand & is a standard Linux operator which makes the modetest command run in the background without exiting (eg, it waits).

Set up the HDMI Rx Pipeline

Configure the VPSS IP color space converter and scaler to support a 1080p input. 

This step is not necessary if using the mediasrcbin plugin below, it is only added for reference. 

media-ctl -d /dev/media0 -V "\"a0040000.v_proc_ss\":0 [fmt:RBG888_1X24/1920x1080 field:none]"
media-ctl -d /dev/media0 -V "\"a0040000.v_proc_ss\":1 [fmt:VYYUYY8_1X24/1920x1080 field:none]"

Run gstreamer

mediasrcbin - The following gstreamer command line uses the mediasrcbin plugin.  This mediasrcbin plugin is Xilinx-specific plugin which is an element on top of v4l2src. It parses and configures the media graph of a media device automatically.

sudo gst-launch-1.0 mediasrcbin media-device=/dev/media0 v4l2src0::io-mode=4 ! video/x-raw, width=1920, height=1080, format=NV12, framerate=30/1 ! xlnxroivideo1detect  capture-io-mode=4 output-io-mode=5 relative-qp=-21 ! omxh265enc gop-mode=basic gop-length=60 b-frames=0 target-bitrate=1500 num-slices=8 control-rate=constant prefetch-buffer=true low-bandwidth=false filler-data=true cpb-size=1000 initial-delay=500 qp-mode=roi ! video/x-h265, profile=main, alignment=au ! queue ! omxh265dec internal-entropy-buffers=5 low-latency=0 split-input=true ! queue max-size-bytes=0 ! fpsdisplaysink name=fpssink text-overlay=false 'video-sink=kmssink bus-id=a00c0000.v_mix hold-extra-sample=1 show-preroll=false sync=true' sync=true -v

v4l2src -  The following gstreamer command line uses the v4l2src plugin. It requires manual commands to setup the VPSS as shown above. 

sudo gst-launch-1.0 v4l2src device=/dev/video0 io-mode=4 ! video/x-raw, width=1920, height=1080, format=NV12, framerate=30/1 ! xlnxroivideo1detect capture-io-mode=4 output-io-mode=5 relative-qp=-21 ! omxh265enc gop-mode=basic gop-length=60 b-frames=0 target-bitrate=1500 num-slices=8 control-rate=constant prefetch-buffer=true low-bandwidth=false filler-data=true cpb-size=1000 initial-delay=500 qp-mode=roi ! video/x-h265, profile=main, alignment=au ! queue ! omxh265dec internal-entropy-buffers=5 low-latency=0 split-input=true ! queue max-size-bytes=0 ! fpsdisplaysink name=fpssink text-overlay=false 'video-sink=kmssink bus-id=a00c0000.v_mix hold-extra-sample=1 show-preroll=false sync=true' sync=true -v

Troubleshooting

Using the xlnx-vcu-roi-trd Snap

Coming Soon!

Demo Operation

Once the gstreamer command or snap starts successfully the output of the HDMI input source is shown on the HDMI monitor. Any faces detected in the input stream will be marked by a red bounding box. If you look closely, you will see that the area inside the bounding box is clearer and includes less compression artifacts than the rest of the image. 

image credit: public domain from RODNAE Productions via http://pexels.com