...
Supported Resolution:
4kp30
1080p30
1.3 VCU ROI Software
1.3.1 GStreamer Pipeline
The GStreamer plugin demonstrates the DPU capabilities with Xilinx VCU encoder’s ROI(Region Of Interest) feature. The plugin will detect ROI(face co-ordinates) from input frames using DPU IP and pass the detected ROI information to the Xilinx VCU encoder. The following figure shows the data flow for GStreamer pipeline of stream-out use case.
...
DPU is initialized and DPU kernel will be loaded using libn2cube APIs - int dpuOpen(), DPUKernel *dpuLoadKernel(const char *networkName)
DPU GStreamer plugin receives data frame from HDMI-Rx through a v4l2src plugin
Create the DPU task - int dpuCreateTask(DPUKernel *kernel, int mode)
Scale the input frame to 640x480 resolution using Xilinx Scaler IP
Convert the input frame data format from NV12 to BGR format using Xilinx Color Space Converter(CSC) soft IP
Prepare the OpenCV image using BGR data
Pass the intermediate OpenCV image to the DPU - int dpuSetInputImage2(DPUTask *task, const char *nodeName, const cv::Mat &image, int idx=0)
Run the DPU task - int dpuRunTask(DPUTask *task)
Extract the ROI(face) co-ordinates from the DPU output
Map the detected face co-ordinates to the original input frame resolution
Fill the ROI metadata buffer using extracted ROI (face) co-ordinates
Pass the ROI meta data buffer and input NV12 frame data buffer to the Xilinx VCU encoder
Destroy the DPU task and kernel - int dpuDestroyTask(DPUTask *task), int dpuDestroyKernel(DPUKernel*kernel)
Close the DPU - int dpuClose()
1.3.
...
2 Deep Learning Processor Unit(DPU)
DPU is a programmable engine dedicated for the convolutional neural network. The unit contains a register configure the module, data controller module, and convolution computing module. There is a specialized instruction set for DPU, which enables DPU to work efficiently for many convolutional neural networks. The deployed convolutional neural network in DPU includes VGG, ResNet, GoogLeNet, YOLO, SSD, MobileNet, FPN, etc.
...
DPU device is initialized by calling dpuOpen() API
From hybrid executable, load the DPU kernel including DPU instructions, weights and biases for specified neural network into dedicated DPU memory space
Instantiate a DPU Task from DPU Kernel and allocate corresponding DPU memory buffer
Set the input image to created DPU task using dpuSetInputImage2() API of libdputils
Run the DPU task to find the faces from the input image
Free the DPU dedicated memory used by the loaded kernel using
DPU device is uninitialized using dpuClose() API of libn2cube
1.
...
3.3 DNNDK Overview
Deep Neural Network Development Kit (DNNDK) is a full-stack deep learning SDK for the Deep-learning Processor Unit(DPU). It provides a unified solution for deep neural network inference applications by providing pruning, quantization, compilation, optimization, and run time support.
...
Refer DNNDK user guide to know more details on DNNDK.
1.
...
4 Software Tools and System Requirements
Hardware
Required:
ZCU106 evaluation board (rev C/D/E/F/1.0) with power cable
Monitor with HDMI input supporting 3840x2160 resolution or 1920x1080 resolution
HDMI cable 2.0 certified
Class-10 SD card
Ethernet cable
...
Resolution | FPS Achieved |
|---|---|
4kp30 | 22 - 25 |
1080p30 | 30 |
1.
...
5 Board Setup
The below section will provide the information on the ZCU106 board setup for running ROI design.
...
The above figure shows all the zcu106 board connector slots
1.
...
6 Run Flow
The VCU ROI TRD package is released with the source code, Vivado project, Petalinux BSP, and SD card image that enables the user to run the demonstration. It also includes the binaries necessary to configure and boot the ZCU106 board. Prior to running the steps mentioned in this wiki page, download the VCU ROI TRD package and extract its contents to a directory referred to as ‘TRD_HOME' which is the home directory.
...
Boot the board with Flash SD Card with VCU ROI images
1.
...
6.1 GStreamer Application (vcu_gst_app)
The vcu_gst_app is a command-line multi-threaded Linux application. The command-line application requires an input configuration file (input.cfg) to be provided in the plain text.
...
NOTE: Make sure HDMI-Rx should be configured to 4kp30 mode.
1.
...
7 Build Flow
Refer below link to download the VCU ROI TRD package.
...
| Code Block |
|---|
% export TRD_HOME=</path/to/downloaded/zipfile>/vcu_roi_2019.2_v1.0 |
1.
...
7.1 HW build Flow
This tutorial shows how to build the hardware and generating XSA using the Vivado tool. Refer to the vivado-release-notes-install-license(UG973) for installation.
...
By default, the XSA is created at pl/prebuild/zcu106_dpu_hdmi_wrapper.xsa
1.
...
7.2 Petalinux build Flow
This tutorial shows how to build the Linux image and boot image using the PetaLinux build tool.
...
| Code Block |
|---|
% cp BOOT.BIN image.ub $TRD_HOME/images |
1.
...
8 VCU GST APP
The vcu_gst_app and supporting libraries will be built as a "vcu-gst-app" recipe inside petalinux-project. Refer "project-spec/meta-user/recipes-apps/vcu-gst-app" directory inside petalinux-project for vcu-gst-app recipe. Source of vcu_apm_lib, vcu_video_lib, vcu_gst_lib and vcu_gst_app is provided as zip inside "project-spec/meta-user/recipes-apps/vcu-gst-app/files/" directory. vcu_gst_app will be built as part of petalinux project and the executable is placed in /usr/bin location of rootfs. User can update the zip file if any source code modifications need to be and run following command to build vcu-gst-app recipe.
...