Zynq UltraScale+ MPSoC VCU TRD 2021.1 - VCU TRD Multi Stream Video Capture and Display
- Ronak Shah (Unlicensed)
- Nirmal Luhana
- Nayan Bhavsar
- Sunil Vaghela
This page provides all the information related to Design Module 1 - VCU TRD Muti Stream design.
Table of Contents
1 Overview
The primary goal of this Design is to demonstrate the capabilities of VCU hard block present in Zynq UltraScale+ EV devices. The TRD will serve as a platform to tune the performance parameters of VCU and arrive at optimal configurations for encoder and decoder blocks. This module supports seven hdmi video streams using AXI4 Stream Broadcaster IP at capture side and eight video streams using mixer at display side for NV12 pixel format.
This design supports the following video interfaces:
Sources:
Test pattern generator (TPG) implemented in the PL
HDMI-Rx capture pipeline implemented in the PL
MIPI CSI-2 Rx capture pipeline implemented in the PL
File source (SD card, USB storage, SATA hard disk)
Stream-In from network or internet
Sinks:
DP-Tx display pipeline in the PS
HDMI-Tx display pipeline implemented in the PL
VCU Codec:
Video Encode/Decode capability using VCU hard block in PL
AVC/HEVC encoding
Encoder/decoder parameter configuration.
Demonstrate the multi-stream capability of VCU at 4K 60 Hz throughput
Streaming Interfaces:
1G Ethernet PS GEM
Video format:
NV12
This design demonstrates the multi-stream capability of VCU at 4K 60 Hz throughput.
Supports 2-4Kp30 multi-stream feature with any 2 of HDMI-Rx, TPG, and MIPI as the input source and HDMI-Tx as display pipeline.
Supports 4-1080p60 multi-stream feature with 3 HDMI-Rx and 1 MIPI as the input source and HDMI-Tx as display pipeline.
Supports 8-1080p30 multi-stream feature with 7 HDMI-Rx and 1 MIPI as the input source and HDMI-Tx as display pipeline.
Other features:
This design supports 8 channel memory-based SCD(Scene Change Detection) IP. SCD can be enabled or disabled through configuration.
Supported Resolution:
The table below provides the supported resolution from GUI and command-line app in this design.
Resolution | GUI | Command Line | |
Single Stream | Single Stream | Multi-stream | |
4Kp60 | X | √ | NA |
4Kp30 | √ | √ | √ (Max 2) |
1080p60 | √ | √ | √ (Max 4) |
1080p30 | X | √ | √ (Max 8) |
√- Supported
x - Not supported
NA - Not applicable
The below table gives information about the features supported in this design.
Pipeline | Input Source | Output Type | Resolution | VCU Codec |
|---|---|---|---|---|
PASSTHROUGH Pipeline: Capture → Display | HDMI-Rx / MIPI / TPG | HDMI-Tx / DP | 4K / 1080p | None |
Single Stream: Capture → SCD → Encode → Decode → Display | HDMI-Rx / MIPI / TPG | HDMI-Tx / DP | 4K / 1080p | HEVC / AVC |
Multi-Stream (2 input sources): Capture → SCD → Encode → Decode → Display | HDMI-Rx / MIPI / TPG | HDMI-Tx | 4Kp30 | HEVC / AVC |
Multi-Stream(4 input sources): Capture → SCD → Encode → Decode → Display | HDMI-Rx / MIPI / TPG | HDMI-Tx | 1080p60 | HEVC / AVC |
Multi-Stream(8 input sources): Capture → SCD → Encode → Decode → Display | 7-HDMI-Rx + 1 MIPI | HDMI-Tx | 1080p30 | HEVC / AVC |
Single Stream: Record/Stream-Out pipeline | HDMI-Rx / MIPI / TPG | File Sink / Stream-Out | 4K / 1080p | HEVC / AVC |
Multi-Stream(2 or 4 i/p sources): Record / Stream-Out pipeline | HDMI-Rx / MIPI / TPG | File Sink / Stream-Out | 2-4Kp30 / 4-1080p60 | HEVC / AVC |
Multi-Stream(8 input sources): Record / Stream-Out pipeline | 7-HDMI-Rx + 1 MIPI | File Sink / Stream-Out | 8-1080p30 | HEVC / AVC |
File Playback / Streaming pipeline | File Source / Stream-In | HDMI-Tx / DP | 4K / 1080p | HEVC / AVC |
DP supports a max resolution of 4Kp30.
TPG will not support 1080p30 resolution mode.
The below figure shows the VCU TRD Multi Stream design hardware block diagram.
The below figure shows the VCU TRD Multi Stream design software block diagram.
1.1 Board Setup
Refer below link for Board Setup
1.2 Run Flow
The TRD package is released with the source code, Vivado project, Petalinux BSP, and SD card image that enables the user to run the demonstration. It also includes the binaries necessary to configure and boot the ZCU106 board. Prior to running the steps mentioned in this wiki page, download the TRD package and extract its contents to a directory referred to as TRD_HOME which is the home directory.
Refer below link to download all TRD contents.
Refer Section 4.1 : Download the TRD of
Zynq UltraScale+ MPSoC VCU TRD 2021.1wiki page to download all TRD contents.
TRD package contents are placed in the following directory structure. The user needs to copy all the files from the $TRD_HOME/images/vcu_trd/ to FAT32 formatted SD card directory.
rdf0428-zcu106-vcu-trd-2021-1/
├── apu
│ └── vcu_petalinux_bsp
├── images
│ ├── vcu_10g
│ ├── vcu_audio
│ ├── vcu_llp2_hdmi_nv12
│ ├── vcu_llp2_hdmi_nv16
│ ├── vcu_llp2_hdmi_xv20
│ ├── vcu_llp2_sdi_xv20
│ ├── vcu_multistream_nv12
│ ├── vcu_pcie
│ ├── vcu_plddrv1_hdr10_hdmi
│ ├── vcu_plddrv2_hdr10_hdmi
│ └── vcu_sdi_xv20
├── pcie_host_package
│ ├── COPYING
│ ├── include
│ ├── LICENSE
│ ├── readme.txt
│ ├── RELEASE
│ ├── tests
│ ├── tools
│ └── xdma
├── pl
│ ├── constrs
│ ├── designs
│ ├── prebuild
│ ├── README.md
│ └── srcs
└── README.txt
└── zcu106_vcu_trd_sources_and_licenses.tar.gzTRD package contents specific to VCU TRD Multi Stream design are placed in the following directory structure.
rdf0428-zcu106-vcu-trd-2021-1
├── apu
│ └── vcu_petalinux_bsp
│ └── xilinx-vcu-zcu106-v2021.1-final.bsp
├── images
│ ├── vcu_multistream_nv12
│ │ ├── autostart.sh
│ │ ├── BOOT.BIN
│ │ ├── boot.scr
│ │ ├── config
│ │ ├── Image
│ │ ├── rootfs.cpio.gz.u-boot
│ │ ├── system.dtb
│ │ └── vcu
├── pcie_host_package
├── pl
│ ├── constrs
│ ├── designs
│ │ └── zcu106_trd
│ ├── prebuild
│ │ └── zcu106_trd
│ ├── README.md
│ └── srcs
│ ├── hdl
│ └── ip
└── README.txt
└── zcu106_vcu_trd_sources_and_licenses.tar.gzThe below snippet shows the configuration files (input.cfg) for running various multistream Display, Record, and Streaming use cases. All these configurations files are placed in the images folder mentioned above. The directory structure in /media/card.
config
├── 1-4kp60
│ ├── Display
│ ├── Record
│ ├── Stream-out
│ └── Stream-in
├── 2-4kp30
│ ├── Display
│ ├── Record
│ ├── Stream-out
│ └── Stream-in
├── 4-1080p60
│ ├── Display
│ ├── Record
│ ├── Stream-out
│ └── Stream-in
├── 8-1080p30
│ ├── Display
│ ├── Record
│ ├── Stream-out
│ └── Stream-in
└── input.cfg1.2.1 GStreamer Application (vcu_gst_app)
The vcu_gst_app is a command-line multi-threaded Linux application. The command-line application requires an input configuration file (input.cfg) to be provided in the plain text.
Run below modetest command to set CRTC configurations for 4Kp60:
Run below modetest command to set CRTC configurations for 4Kp30:
Execution of the application is shown below:
Example:
4kp60 HEVC_HIGH Display Pipeline Execution
4kp60 HEVC_HIGH Record Pipeline Execution
4kp60 HEVC_HIGH Stream-out Pipeline Execution
4kp60 HEVC_HIGH Stream-in Pipeline Execution
Make sure HDMI-Rx should be configured to 4kp60 mode
Latency Measurement: To measure the latency of the pipeline, run the below command. The latency data is huge, so dump it to a file.
Refer below link for detailed run flow steps
1.3 Build Flow
Refer below link for detailed build flow steps
2 Other Information
2.1 Known Issues
Frame drop observed in TPG pipeline
Frequency: Always
Workaround: None
56 fps in 4Kp60/1080p60, and 28 fps in 4Kp30 resolution
Frame drops and jerkiness is observed with TPG source with some test patterns generated by TPG in VCU TRD design
VCU QT: Green shade on DP when playing 4Kp30 HDMI/MIPI pipeline with SCD
Frequency: At random interval
Workaround: None
Frame drops observed with 8x1080p30 (7-HDMI-Rx + 1 MIPI) AVC serial use-case due to system level limitation
For Petalinux related known issues please refer: PetaLinux 2021.1 - Product Update Release Notes and Known Issues
For VCU related known issues please refer AR# 76600: LogiCORE H.264/H.265 Video Codec Unit (VCU) - Release Notes and Known Issues and Xilinx Zynq UltraScale+ MPSoC Video Codec Unit.
To reduce performance issues with 8x serial pipelines, please refer to chapter# 40 of Section VI: Appendices for IRQ Balancing scheme in PG252.
2.2 Limitations
For playback in DP, video input resolution should match to DP's native resolution. This constraint is due to the support of the GUI. In the GUI case if we allow video source other than native resolution(by setting full screen overlay) then the graphics layer will disappear. To recover back GUI user need to kill and relaunch the GUI app. To avoid such condition TRD only supports video input resolution which is equal to DP's native resolution.
For Petalinux related limitations please refer: PetaLinux 2021.1 - Product Update Release Notes and Known Issues
For VCU related limitations please refer AR# 76600: LogiCORE H.264/H.265 Video Codec Unit (VCU) - Release Notes and Known Issues, Xilinx Zynq UltraScale+ MPSoC Video Codec Unit and PG252.
2.3 Optimum VCU Encoder parameters for use-cases:
Video streaming:
Video streaming use-case requires a very stable bitrate graph for all pictures
It is good to avoid periodic large Intra pictures during the encoding session
Low-latency rate control (hardware RC) is the preferred control-rate for video streaming; as it tries to maintain equal amount frame sizes for all pictures.
Good to avoid periodic Intra frames instead use low-delay-p (IPPPPP…)
VBR is not a preferred mode of streaming
Performance: AVC Encoder settings:
It is preferred to use 8 or higher slices for better AVC encoder performance
AVC standard does not support Tile mode processing, which results in the processing of MB rows sequentially for entropy coding.
Quality: Low bitrate AVC encoding:
Enable
profile=highand useqp-mode=autofor low-bitrate encoding use-casesThe high profile enables 8x8 transform which results in better video quality at low bitrates
3 Appendix A - Input Configuration File (input.cfg)
The example configuration files are stored at /media/card/config/ folder.
Configuration Type | Configuration Name | Description |
|---|