This page provides all the information related to Design Module 3 - VCU TRD Multi stream Audio-Video design.

1 Overview

The primary goal of this Design is to demonstrate the capabilities of VCU hard block present in Zynq UltraScale+ EV devices with soft audio codec. The TRD will serve as a platform to tune the performance parameters of VCU and arrive at optimal configurations for encoder and decoder blocks with audio-video synchronization.

...

Codec: AAC
Format: S24_32LE
Channel: 2
Sampling rate: 48 kHz
Source: HDMI-Rx/ I2S-Rx
Render: HDMI-Tx/ I2S-Tx/ DP

Audio Deliverables:

Pipeline	Video-Input Source	Audio Input Source	Video Output Type	Audio Output Type	ALSA Drivers	Resolution	Audio Codec Type	Audio Configuration	Video Codec Type	Deliverables
Record / Stream-Out pipeline	HDMI-Rx MIPI-Rx	HDMI-Rx I2S-Rx	File-Sink Stream-Out	File-Sink Stream-Out	HDMI-Rx ALSA drivers	4K / 1080p	AAC	2 channel @ 48 kHz	HEVC / AVC	HDMI-Rx Audio encode with soft codec and video with VCU and store it in a container format.
Playback pipeline	File Source/ Stream-In	File Source/ Stream-In	DP HDMI –Tx	HDMI-Tx I2S-Tx DP	HDMI-Tx ALSA drivers	4K / 1080p	AAC	2 channel @ 48 kHz	HEVC / AVC	Playback of the local-file / stream-in with video decoded using VCU and Audio using GStreamer soft codec.
Capture → Display	HDMI-Rx MIPI-Rx	HDMI-Rx I2S-Rx	DP HDMI -Tx	HDMI-Tx I2S-Tx DP	HDMI-Rx/Tx ALSA drivers	4K / 1080p	NA	2 channel @ 48 kHz	HEVC / AVC	HDMI-Rx Audio / Video pass to HDMI-Tx without VCU/Audio-Codec.
Capture → Encod → Decode → Display	HDMI-Rx MIPI-Rx	HDMI-Rx I2S-Rx	DP HDMI -Tx	HDMI-Tx I2S-Tx DP	HDMI-Rx/Tx ALSA drivers	4K / 1080p	NA	2 channel @ 48 kHz	HEVC / AVC	HDMI-Rx raw audio and video with VCU encoder and decode to achieve AV sync.

Supports 1-4Kp60 Single Stream with either HDMI-Rx or I2S-Rx as input Audio source + HDMI-Rx / MIPI Rx as input Video source; and HDMI-Tx / I2S-Tx as Output Audio Sink + HDMI-Tx / DP as Output Video sink pipeline
Supports 1-4Kp30 Single Stream with either HDMI-Rx or I2S-Rx as input Audio source + HDMI-Rx / MIPI Rx as input Video source; and HDMI-Tx / I2S-Tx as Output Audio Sink + HDMI-Tx / DP as Output Video sink pipeline
Supports 1-1080p60 Single Stream with either HDMI-Rx or I2S-Rx as input Audio source + HDMI-Rx / MIPI Rx as input Video source; and HDMI-Tx / I2S-Tx as Output Audio Sink + HDMI-Tx / DP as Output Video sink pipeline
Supports 2-4Kp30 multi-stream feature with HDMI-Rx and I2S-Rx as input Audio sources, with HDMI-Rx and MIPI Rx as an input Video sources; and with HDMI-Tx and I2S-Tx as Output Audio Sink + HDMI-Tx as Output Video sink pipeline
Supports 2-1080p60 multi-stream feature with HDMI-Rx and I2S-Rx as input Audio sources with HDMI-Rx and MIPI Rx as an input Video sources; and with HDMI-Tx and I2S-Tx as Output Audio Sink + HDMI-Tx as Output Video sink pipeline

...

The table below provides the supported resolution from GUI and command-line app in this design.

Resolution	GUI	Command Line
Resolution	Single Stream	Single Stream	Multi-stream
4Kp60	X	√	NA
4Kp30	√	√	√ (Max 2)
1080p60	√	√	√ (Max 2)

√ - Supported
x – Not supported
NA – Not applicable

The below sections describe the HDMI / MIPI Video Capture and HDMI Display with the Audio from HDMI / I2S sources. It is VCU TRD design supporting HDMI-Rx audio/video + HDMI-Tx with Audio/video and MIPI-Rx video + I2S-Rx audio with HDMI-Tx video + I2S-Tx audio.

...

The below figure shows the HDMI, MIPI Video Capture along with HDMI, I2S Audio Capture and HDMI Display with Audio design software block diagram.

1.1 Board Setup

Refer below link for Board Setup

Zynq UltraScale+ MPSoC VCU TRD 2020.1 Board Setup
I2S Audio. signals from MPSoC PL fabric are connected to PMOD0 GPIO Header (J55 - right angle female connector )
PMOD I2S2 Add on card connects to J55 connector and its Master/Slave select jumper (JP1) should be placed into the Slave (SLV) position

1.2 Run Flow

The TRD package is released with the source code, Vivado project, Petalinux BSP, and SD card image that enables the user to run the demonstration. It also includes the binaries necessary to configure and boot the ZCU106 board. Prior to running the steps mentioned in this wiki page, download the TRD package and extract its contents to a directory referred to as TRD_HOME which is the home directory.

...

Code Block

rdf0428-zcu106-vcu-trd-2020.-1
    ├── apu
    │   └── vcu_petalinux_bsp
    ├── images
    │   ├── vcu_10g
    │   ├── vcu_audio
    │   ├── vcu_hdmi_multistream_xv20
    │   ├── vcu_hdmi_rx
    │   ├── vcu_hdmi_tx
    │   ├── vcu_llp2_hdmi_nv12
    │   ├── vcu_llp2_hdmi_nv16
    │   ├── vcu_llp2_hdmi_xv20
    │   ├── vcu_llp2_sdi_xv20
    │   ├── vcu_multistream_nv12
    │   ├── vcu_pcie
    │   ├── vcu_sdirx
    │   ├── vcu_sditx
    │   └── vcu_sdi_xv20
    ├── pcie_host_package
    │   ├── COPYING
    │   ├── include
    │   ├── libxdma
    │   ├── LICENSE
    │   ├── readme.txt
    │   ├── RELEASE
    │   ├── tests
    │   ├── tools
    │   └── xdma
    ├── pl
    │   ├── constrs
    │   ├── designs
    │   ├── prebuild
    │   ├── README.md
    │   └── srcs
    └── README.txt

...

Code Block

└── rdf0428-zcu106-vcu-trd-2020.-1
	├── apu
	│   └── vcu_petalinux_bsp
	│       └── xilinx-vcu-zcu106-v2020.1-final.bsp
	├── images
	│   ├── vcu_audio
	│   │	├── autostart.sh
	│   │   ├── bin
	│   │	├── BOOT.BIN
	│   │   ├── boot.scr
	│   │	├── config
	│   │	├── image.ub
	│   │	├── system.dtb
	│   │	└── vcu
	├── pcie_host_package
	├── pl
	│   ├── constrs
	│   ├── designs
	│   │	├── zcu106_audio 
	│   ├── prebuild
	│   │	├── zcu106_audio 
	│   ├── README.md
	│   └── srcs
	│	├── hdl
	│	└── ip
	└── README.txt

...

Code Block

config
├── 1-4kp60
│   ├── Display
│   ├── Record
│   ├── Stream-in
│   └── Stream-out
├── 2-1080p60
│   ├── Display
│   ├── Record
│   ├── Stream-in
│   └── Stream-out
├── 2-4kp30
│   ├── Display
│   ├── Record
│   ├── Stream-in
│   └── Stream-out
└── input.cfg

1.2.1 GStreamer Application (vcu_gst_app)

The vcu_gst_app is a command-line multi-threaded Linux application. The command-line application requires an input configuration file (input.cfg) to be provided in the plain text.

...

Zynq UltraScale+ MPSoC VCU TRD 2020.1 - Run Flow

1.3 Build Flow

Refer below link for Build Flow

Zynq UltraScale+ MPSoC VCU TRD 2020.1 - Build Flow

...

2 Other Information

2.1 Known Issues

Block Noise is observed in AVC_MEDIUM and AVC_LOW in 4Kp60 pipelines
The digilent PMOD card cannot support the passive source like MICROPHONES. Only active sources are to be connected. Here the source is from the Aux cable which is connected in between the source (laptop) and PMOD card.
For Petalinux related known issues please refer: PetaLinux 2020.1 - Product Update Release Notes and Known Issues
For VCU related known issues please refer AR# 66763: LogiCORE H.264/H.265 Video Codec Unit (VCU) - Release Notes and Known Issues and Xilinx Zynq UltraScale+ MPSoC Video Codec Unit.

2.2 Limitations

For playback in DP, video input resolution should match to DP's native resolution. This constraint is due to the support of the GUI. In the GUI case if we allow video source other than native resolution(by setting full screen overlay) then the graphics layer will disappear. To recover back GUI user need to kill and relaunch the GUI app. To avoid such condition TRD only supports video input resolution which is equal to DP's native resolution.
For Petalinux related limitations please refer: PetaLinux 2020.1 - Product Update Release Notes and Known Issues
For VCU related limitations please refer AR# 66763: LogiCORE H.264/H.265 Video Codec Unit (VCU) - Release Notes and Known Issues, Xilinx Zynq UltraScale+ MPSoC Video Codec Unit and PG252

2.3 Optimum VCU Encoder parameters for use-cases:

Video streaming:

Video streaming use-case requires a very stable bitrate graph for all pictures
It is good to avoid periodic large Intra pictures during the encoding session
Low-latency rate control (hardware RC) is the preferred control-rate for video streaming, it tries to maintain equal amount frame sizes for all pictures.
Good to avoid periodic Intra frames instead use low-delay-p (IPPPPP…)
VBR is not a preferred mode of streaming

...

Enable profile=high and use qp-mode=auto for low-bitrate encoding use-cases
The high profile enables 8x8 transform which results in better video quality at low bitrate

2.4 Audio-Video Synchronization

Clocks and synchronization in GStreamer

...

For more detail please refer: https://gstreamer.freedesktop.org/documentation/application-development/advanced/clocks.html?gi-language=c

...

3 Appendix A - Input Configuration File (input.cfg)

The example configuration files are stored at /media/card/config/ folder.

...

Exit
It indicates to the application that the configuration is over.

Mount Locations:

The mount locations for various devices can be found in the below table.
The mount locations can vary. Users can use lsblk or mount to find the location of the mounted devices.

Below are some example mount points

Device	Mount Location
SD Card	/run/media/mmcblk0p2
Sata Drive USB Drive	/run/media/sda /run/media/usb
RAM Disk	/run/media/

...

4 Appendix B - HDMI-Rx/Tx Link-up and GStreamer Commands

This section covers configuration of HDMI-Rx using media-ctl utility and HDMI-Tx using modetest utility, along with demonstrating HDMI-Rx/Tx link-up issues and steps to switch HDMI-Rx resolution. It also contains sample GStreamer HDMI Audio+Video (also I2S Audio + MIPI CSI Video pipelines for Display, Record & Playback, Stream-in and Stream-out use-cases.

...

Code Block

$ gst-launch-1.0 v4l2src device=/dev/video0 io-mode=4 num-buffers=3600 ! video/x-raw, format=NV12, width=3840, height=2160, framerate=60/1 ! omxh265enc qp-mode=auto gop-mode=basic gop-length=60 b-frames=0 target-bitrate=60000 num-slices=8 control-rate=constant prefetch-buffer=true low-bandwidth=false filler-data=true cpb-size=1000 initial-delay=500 ! video/x-h265, profile=main, alignment=au ! queue ! mux. alsasrc device=hw:2,1 num-buffers=3600 ! audio/x-raw, format=S24_32LE, rate=48000, channels=2 ! queue ! audioconvert ! audioresample ! faac ! aacparse ! mpegtsmux name=mux ! filesink location = "/run/media/sda/test.ts"

...

Code Block

$ gst-launch-1.0 v4l2src device=/dev/video1 io-mode=4 num-buffers=3600 ! video/x-raw, format=NV12, width=3840, height=2160, framerate=60/1 ! omxh265enc qp-mode=auto gop-mode=basic gop-length=60 b-frames=0 target-bitrate=60000 num-slices=8 control-rate=constant prefetch-buffer=true low-bandwidth=false filler-data=true cpb-size=1000 initial-delay=500 ! queue ! video/x-h265, profile=main, alignment=au ! mux. alsasrc device=hw:1,1 num-buffers=3600 ! audio/x-raw, format=S24_32LE, rate=48000, channels=2 ! queue ! audioconvert ! audioresample ! faac ! mpegtsmux name=mux ! filesink location = "/run/media/sda/test.ts"

...

Versions Compared

Old Version 44

New Version Current

Key

Table of Contents

1 Overview

1.1 Board Setup

1.2 Run Flow

1.2.1 GStreamer Application (vcu_gst_app)

1.3 Build Flow

2 Other Information

2.1 Known Issues

2.2 Limitations

2.3 Optimum VCU Encoder parameters for use-cases:

2.4 Audio-Video Synchronization

3 Appendix A - Input Configuration File (input.cfg)

Mount Locations:

4 Appendix B - HDMI-Rx/Tx Link-up and GStreamer Commands

Page Comparison

Versions Compared

Old Version 44

New Version Current

Key

Table of Contents

1 Overview

1.1 Board Setup

1.2 Run Flow

1.2.1 GStreamer Application (vcu_gst_app)

1.3 Build Flow

2 Other Information

2.1 Known Issues

2.2 Limitations

2.3 Optimum VCU Encoder parameters for use-cases:

2.4 Audio-Video Synchronization

3 Appendix A - Input Configuration File (input.cfg)

Mount Locations:

4 Appendix B - HDMI-Rx/Tx Link-up and GStreamer Commands