This page provides all of the information related to Design Module 1 - VCU TRD Multi Stream Video capture and display design.

1 Overview

The primary goal of this design is to demonstrate the capabilities of the VCU hard block present in Zynq UltraScale+ EV devices. The TRD will serve as a platform to tune the performance parameters of the VCU and arrive at optimal configurations for encoder and decoder blocks. This module supports seven HDMI video streams using AXI4 Stream Broadcaster IP at the capture side and eight video streams using mixer at the display side for NV12 pixel format.

This design supports the following video interfaces:

Sources:

Test pattern generator (TPG) implemented in the PL
HDMI-Rx capture pipeline implemented in the PL
MIPI CSI-2 Rx capture pipeline implemented in the PL
File source (SD card, USB storage, SATA hard disk)
Stream-In from network or internet

Sinks:

DP-Tx display pipeline in the PS
HDMI-Tx display pipeline implemented in the PL

VCU Codec:

Video Encode/Decode capability using VCU hard block in PL
- AVC/HEVC encoding
- Encoder/decoder parameter configuration.
- Demonstrate the multi-stream capability of VCU at 4K 60 Hz throughput

Streaming Interfaces:

1G Ethernet PS GEM

Video format:

NV12

This design demonstrates the multi-stream capability of VCU at 4K 60 Hz throughput.

Supports 2-4Kp30 multi-stream feature with any 2 of HDMI-Rx, TPG, and MIPI as the input source and HDMI-Tx as the display pipeline.
Supports 4-1080p60 multi-stream feature with 3 HDMI-Rx and 1 MIPI as the input source and HDMI-Tx as the display pipeline.
Supports 8-1080p30 multi-stream feature with 7 HDMI-Rx and 1 MIPI as the input source and HDMI-Tx as the display pipeline.

Other features:

This design supports 8 channel memory-based Scene Change Detection (SCD) IP. SCD can be enabled or disabled through configuration.

Supported Resolution:

The table below provides the supported resolution from the GUI and command-line app in this design.

Resolution	GUI	Command Line
Resolution	Single Stream	Single Stream	Multi-stream
4Kp60	X	√	NA
4Kp30	√	√	√ (Max 2)
1080p60	√	√	√ (Max 4)
1080p30	X	√	√ (Max 8)

√- Supported
x - Not supported
NA - Not applicable

The below table gives information about the features supported in this design.

Pipeline	Input Source	Output Type	Resolution	VCU Codec

Pipeline	Input Source	Output Type	Resolution	VCU Codec
Pass-through/RAW Pipeline: Capture -> Display	HDMI-Rx / MIPI / TPG	HDMI-Tx / DP	4K / 1080p	None
Single Stream: Capture -> SCD -> Encode -> Decode -> Display	HDMI-Rx / MIPI / TPG	HDMI-Tx / DP	4K / 1080p	HEVC / AVC
Multi-Stream (2 input sources): Capture -> SCD -> Encode -> Decode -> Display	HDMI-Rx / MIPI / TPG	HDMI-Tx	4Kp30	HEVC / AVC
Multi-Stream(4 input sources): Capture -> SCD -> Encode -> Decode -> Display	HDMI-Rx / MIPI / TPG	HDMI-Tx	1080p60	HEVC / AVC
Multi-Stream(8 input sources): Capture -> SCD -> Encode -> Decode -> Display	7-HDMI-Rx + 1 MIPI	HDMI-Tx	1080p30	HEVC / AVC
Single Stream: Record/Stream-Out pipeline	HDMI-Rx / MIPI / TPG	File Sink / Stream-Out	4K / 1080p	HEVC / AVC
Multi-Stream(2 or 4 i/p sources): Record / Stream-Out pipeline	HDMI-Rx / MIPI / TPG	File Sink / Stream-Out	2-4Kp30 / 4-1080p60	HEVC / AVC
Multi-Stream(8 input sources): Record / Stream-Out pipeline	7-HDMI-Rx + 1 MIPI	File Sink / Stream-Out	8-1080p30	HEVC / AVC
File Playback / Streaming pipeline	File Source / Stream-In	HDMI-Tx / DP	4K / 1080p	HEVC / AVC

DP supports a max resolution of 4Kp30.
TPG will not support 1080p30 resolution mode.

The below figure shows the VCU TRD Multi Stream Video capture and display design hardware block diagram.

The below figure shows the VCU TRD Multi Stream Video capture and display design software block diagram.

1.1 Board Setup

Refer to the below link for Board Setup

Zynq UltraScale+ MPSoC VCU TRD 2024.1 Board Setup

1.2 Run Flow

The TRD package is released with the source code, Vivado project, PetaLinux BSP, and SD card image that enables the user to run the demonstration. It also includes the binaries necessary to configure and boot the ZCU106 board. Prior to running the steps mentioned in this wiki page, download the TRD package and extract its contents to a directory referred to as TRD_HOME - which is the home directory.

Refer to the below link to download all TRD contents.

Refer to Section 4.1 : Download the TRD of the Zynq UltraScale+ MPSoC VCU TRD 2024.1 wiki page to download all TRD contents.

TRD package contents are placed in the following directory structure. The user needs to copy all of the files from the $TRD_HOME/images/vcu_trd/ to a FAT32 formatted SD card directory.

rdf0428-zcu106-vcu-trd-2024-1/
├── apu
│   └── vcu_petalinux_bsp
├── images
│   ├── vcu_audio
│   ├── vcu_llp2_hdmi_nv12
│   ├── vcu_llp2_hlg_sdi
│   ├── vcu_llp2_plddr_hdmi
│   ├── vcu_multistream_nv12
│   ├── vcu_plddrv1_hdr10_hdmi
│   ├── vcu_plddrv2_hdr10_hdmi
│   └── vcu_yuv444
├── pl
│   ├── constrs
│   ├── designs
│   ├── prebuild
│   ├── README.md
│   └── srcs
├── README.txt
└── zcu106_vcu_trd_sources_and_licenses.tar.gz

16 directories, 3 files

TRD package contents specific to the VCU TRD Multi Stream design are placed in the following directory structure.

rdf0428-zcu106-vcu-trd-2024-1
├── apu
│   └── vcu_petalinux_bsp
│       └── xilinx-vcu-zcu106-v2024.1-final.bsp
├── images
│   ├── vcu_multistream_nv12
│   │   ├── autostart.sh
│   │   ├── BOOT.BIN
│   │   ├── bootfiles/
│   │   ├── boot.scr
│   │   ├── config/
│   │   ├── Image
│   │   ├── rootfs.cpio.gz.u-boot
│   │   ├── system.dtb
│   │   └── vcu/
├── pl
│   ├── constrs/
│   ├── designs
│   │   └── zcu106_trd/
│   ├── prebuild
│   │   └── zcu106_trd/
│   ├── README.md
│   └── srcs
├── README.txt
└── zcu106_vcu_trd_sources_and_licenses.tar.gz

The below snippet shows the configuration files (input.cfg) for running various multistream Display, Record, and Streaming use cases. All of these configurations files are placed in the images folder mentioned above. The directory structure in /media/card.

config
├── 1-4kp60
│   ├── Display
│   ├── Record
│   ├── Stream-out
│   └── Stream-in
├── 2-4kp30
│   ├── Display
│   ├── Record
│   ├── Stream-out
│   └── Stream-in 
├── 4-1080p60
│   ├── Display
│   ├── Record
│   ├── Stream-out
│   └── Stream-in 
├── 8-1080p30 
│   ├── Display 
│   ├── Record
│   ├── Stream-out 
│   └── Stream-in 
└── input.cfg

1.2.1 GStreamer Application (vcu_gst_app)

The vcu_gst_app is a command-line multi-threaded Linux application. The command-line application requires an input configuration file (input.cfg) to be provided in the plain text.

Run the below modetest command to set CRTC configurations for 4Kp60:

modetest -D a0070000.v_mix -s 54:3840x2160-60@BG24

Run the below modetest command to set CRTC configurations for 4Kp30:

modetest -D a0070000.v_mix -s 54:3840x2160-30@BG24

Execution of the application is shown below:

vcu_gst_app <path to *.cfg file>

Example:

4kp60 HEVC_HIGH Display Pipeline Execution

vcu_gst_app /media/card/config/1-4kp60/Display/Single_4kp60_HEVC_HIGH.cfg

4kp60 HEVC_HIGH Record Pipeline Execution

vcu_gst_app /media/card/config/1-4kp60/Record/Single_4kp60_HEVC_HIGH.cfg

4kp60 HEVC_HIGH Stream-out Pipeline Execution

vcu_gst_app /media/card/config/1-4kp60/Stream-out/Single_4kp60_HEVC_HIGH.cfg

4kp60 HEVC_HIGH Stream-in Pipeline Execution

vcu_gst_app /media/card/config/1-4kp60/Stream-in/input.cfg

The HDMI-Rx should be configured to 4kp60 mode

Latency Measurement: To measure the latency of the pipeline, run the below command. The latency data is huge, so dump it to a file.

GST_DEBUG="GST_TRACER:7" GST_TRACERS="latency" GST_DEBUG_FILE=/run/latency.log vcu_gst_app /media/card/config/input.cfg

Refer to the below link for detailed run flow steps

Zynq UltraScale+ MPSoC VCU TRD 2024.1 - Run Flow

1.3 Build Flow

Refer to the below link for detailed build flow steps

Zynq UltraScale+ MPSoC VCU TRD 2024.1 - Build Flow

2 Other Information

2.1 Known Issues

Frame drop observed in TPG pipeline
- Frequency: Always
- Work-around: None

56 fps in 4Kp60/1080p60 resolution

Frame drops and jerkiness are observed with the TPG source with some test patterns generated by the TPG in the VCU TRD design
VCU QT: Green shade on DP when playing 4Kp30 HDMI/MIPI pipeline with SCD
- Frequency: At random interval
- Work-around: None
Frame drops observed with 8x1080p30 (7-HDMI-Rx + 1 MIPI) AVC serial use-case due to system level limitation
For PetaLinux related known issues please refer to: PetaLinux 2024.1 - Product Update Release Notes and Known Issues
For VCU related known issues please refer AR# 76600: LogiCORE H.264/H.265 Video Codec Unit (VCU) - Release Notes and Known Issues and Xilinx Zynq UltraScale+ MPSoC Video Codec Unit.
To reduce performance issues with 8x serial pipelines, please refer to IRQ Balancing in PG252.

2.2 Limitations

For playback in DP, video input resolution should match the DP's native resolution. This constraint is in support of the GUI. In the GUI if we allow a video source other than native resolution (by setting full screen overlay) then the graphics layer will disappear. To recover, the GUI user need to kill and relaunch the GUI app. To avoid this condition the TRD only supports video input resolution which is equal to the DP's native resolution.
For PetaLinux related limitations please refer: PetaLinux 2024.1 - Product Update Release Notes and Known Issues
For VCU related limitations please refer AR# 76600: LogiCORE H.264/H.265 Video Codec Unit (VCU) - Release Notes and Known Issues, Xilinx Zynq UltraScale+ MPSoC Video Codec Unit and PG252.

2.3 Optimum VCU Encoder parameters for use-cases:

Video streaming:

Video streaming use-case requires a very stable bitrate graph for all pictures
It is good to avoid periodic large Intra pictures during the encoding session
Low-latency rate control (hardware RC) is the preferred control-rate for video streaming; as it tries to maintain equal frame sizes for all pictures.
It is good to avoid periodic Intra frames and instead use low-delay-p (IPPPPP…)
VBR is not a preferred mode of streaming

Performance: AVC Encoder settings:

It is preferred to use 8 or higher slices for better AVC encoder performance
AVC standard does not support Tile mode processing, which results in the processing of MB rows sequentially for entropy coding.

Quality: Low bitrate AVC encoding:

Enable profile=high and use qp-mode=auto for the low-bitrate encoding use-cases
The high profile enables 8x8 transform which results in better video quality at low bitrates

3 Appendix A - Input Configuration File (input.cfg)

The example configuration files are stored at /media/card/config/ folder.

Configuration Type	Configuration Name	Description	Available Options

Configuration Type	Configuration Name	Description	Available Options
Common	Common Configuration	It is the starting point of common configuration
	Num of Input	Provide the number of inputs	1 to 8
	Output	Select the video interface	HDMI or DP
	Out Type	Type of output	display, record, stream
	Display Rate	Pipeline frame rate	30 or 60 fps
	Exit	It indicates to the application that the configuration is over
Input	Input Configuration	It is the starting point of the input configuration
	Input Num	Starting N^th input configuration	1 to 8
	Input Type	Input source type	TPG, HDMI, HDMI_2, HDMI_3, HDMI_4, HDMI_5, HDMI_6, HDMI_7, MIPI, File, Stream
	Uri	File path or Network URL. Applicable for file playback and stream-in pipeline only. Supported file formats for playback are ts, mp4, and mkv.	file:///run/media/vcu-sda/abc.ts OR file:///run/abc.ts OR file:///media/usb/abc.ts (for file path), udp://192.168.25.89:5004/ (for Network streaming, Here 192.168.25.89 is Client's IP address and 5004 is port number)
	Raw	To tell the pipeline is processed or pass-through	True, False
	Width	The width of the live source	3840, 1920
	Height	The height of the live source	2160, 1080
	Format	The format of input data	NV12
	Enable SCD	Enable or Disable Memory based SCD in the pipeline	True, False
	Exit	It indicates to the application that the configuration is over
Encoder	Encoder Configuration	It is the starting point of encoder configuration
	Encoder Num	Starting N^th encoder configuration	1 to 8
	Encoder Name	Name of the encoder	AVC, HEVC
	Profile	Name of the profile	AVC: baseline, main or high HEVC: Main
	Rate Control	Rate control options	CBR, VBR, and Low_Latency
	Filler Data	Filler Data NAL units for CBR rate control	True, False
	QP	QP control mode used by the VCU encoder	Uniform, Auto
	L2 Cache	Enable or Disable L2Cache buffer in encoding process	True, False
	Latency Mode	Encoder latency mode.	Normal, sub_frame
	Low Bandwidth	If enabled, decrease the vertical search range used for P-frame motion estimation to reduce the bandwidth.	True, False
	GOP Mode	Group of Pictures mode.	Basic, low_delay_p, low_delay_b
	Bitrate	Target bitrate in Kbps	1-60000
	B Frames	Number of B-frames between two consecutive P-frames	0-4
	Slice	The number of slices produced for each frame. Each slice contains one or more complete macroblock/CTU row(s). Slices are distributed over the frame as regularly as possible. If slice-size is defined as well more slices may be produced to fit the slice-size requirement.	4-22 4Kp resolution with HEVC codec 4-32 4Kp resolution with AVC codec 4-32 1080p resolution with HEVC codec 4-32 1080p resolution with AVC codec
	GOP Length	The distance between two consecutive I frames	1-1000
	GDR Mode	It specifies which Gradual Decoder Refresh(GDR) scheme should be used when gop-mode = low_delay_p GDR mode is currently supported with LLP1/LLP2 low-delay-p use-cases only	Horizontal, Vertical, Disabled
	Entropy Mode	It specifies the entropy mode for H.264 (AVC) encoding process	CAVLC, CABAC, Default
	Max Picture Size	It is used to curtail instantaneous peak in the bit-stream using this parameter. It works in CBR/VBR rate-control only. When it is enabled, max-picture-size value is calculated and set with 10% of AllowedPeakMargin. i.e. max-picture-size = (TargetBitrate / FrameRate) * 1.1 It works in CBR/VBR rate-control only	True, False
	Format	The format of input data	NV12
	Preset	Based on provided six presets, predefined configuration will be set for encoder parameters. Select custom to provide user-specific options for encoder parameters.	HEVC_HIGH, HEVC_MEDIUM, HEVC_LOW, AVC_HIGH, AVC_MEDIUM, AVC_LOW, Custom
	Exit	It indicates to the application that the configuration is over

Zynq UltraScale+ MPSoC VCU TRD 2024.1 - VCU TRD Multi Stream Video Capture and Display

Table of Contents