This page provides all the information related to Design Module 7 - VCU TRD Xilinx low latency(LLP2) PS DDR NV12 HDMI design.

...

• Zynq UltraScale+ MPSoC VCU TRD 2020.2 - Build Flow

...

2

...

V4l2 Capture Control Software Encoder Application

2.

...

For Petalinux related known issues please refer: PetaLinux 2020.2 - Product Update Release Notes and Known Issues

...

1 Overview

The primary goal of v4l2 capture control software encoder application is to demonstrate the Xilinx’s Ultra Low-Latency feature using the VCU ctrlsw APIs. This application (v4l2_capture_ctrlsw_enc) is an enhanced version of normal VCU ctrlsw app (ctrlsw_encoder). Normal ctrlsw_encoder application is only capable for file-based encoding, while this app captures data from the HDMI source and does stream-out using Gstreamer libraries.

The v4l2 capture ctrlsw encoder application has following features:

• Stream out encoded data captured from HDMI source using RTP streaming.

• Record encoded data captured from HDMI source to a file.

• Supports various encoding options, can be set by config file as an input to the application, similar as how config file is used for ctrlsw_encoder.

• Supports various latency modes e.g. Xilinx’s Ultra Low Latency (LLP2) mode via --xlnx-slicelat and Low Latency (LLP1) mode via --slicelat, can be set via command line

The following figure shows one of the use cases (Xilinx Low-Latency streaming):

...

As shown in the above figure, the app performs the below list of operations in case of Xilinx’s Ultra Low Latency (LLP2) mode:

1. Application enables syncip and programs address ranges as per input video format and resolution.

2. Application calls VIDIOC_DQBUF and sends empty input buffer to encoder using early dequeue mechanism.

3. Encoder receives this empty buffer and starts generating read request.

4. Start DMA command is issued to v4l2 capture driver and capture starts filling the buffer.

5. SyncIP blocks the encoder until framebuffer-write is done writing data corresponding to read request made by encoder.

6. Once encoder is unblocked, it starts encoding data and generating output slices corresponding to unblocked input read requests.

7. Encoded data is feed to Gstreamer AppSrc, and it passed to UDP sink through RTP payloader to stream-out the encoded data.

8. Similarly, for consecutive buffers v4l2 programs SyncIP, submits buffer to encoder using VIDIOC_DQBUF and syncip block the encoder until v4l2 has written sufficient data. This way syncip maintains the synchronization between producer (v4l2) and consumer (encoder).

The below figure shows the v4l2 capture control software encoder application software block diagram:

...

2.2 Downloads

• Download AR package for V4l2 Control Software Encoder Application from below AR link:

  • AR#76096: 2020.2 Zynq UltraScale+ MPSoC VCU - Live capture encoder application for Xilinx Low latency use case (To be published soon)

• You can download VCU TRD 2020.2 rdf package - containing VCU TRD bsp and designs from below link:

  • https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/1010303044/Zynq+UltraScale+MPSoC+VCU+TRD+2020.2#4.1-Download-the-TRD

2.3 Build Flow

This tutorial shows how to build the above v4l2 control software encoder application’s AR package to generate Linux and boot image using the PetaLinux build tool. It assume that the $TRD_HOME environment variable is set as given below.

$ export TRD_HOME=</path/to/downloaded/zipfile>/rdf0428-zcu106-vcu-trd-2020-2

• Source the Petalinux tool-chain using below command

$ source <path/to/petalinux-installer>/tool/petalinux-v2020.2-final/settings.sh
$ echo $PETALINUX

• Create a PetaLinux project.

$ cd $TRD_HOME/apu/vcu_petalinux_bsp
$ petalinux-create -t project -s xilinx-vcu-zcu106-v2020.2-final.bsp

• Configure the PetaLinux project.

$ cd xilinx-vcu-zcu106-v2020.2-final
$ petalinux-config --get-hw-description=$TRD_HOME/pl/prebuild/zcu106_llp2_audio_nv12/

• If the Vivado project is modified, the user is expected to configure the bsp with the modified .xsa file and build. e.g.

$ petalinux-config --get-hw-description=$TRD_HOME/pl/build/zcu106_llp2_audio_nv12/

• Extract the downloaded AR package in the petalinux project (xilinx-vcu-zcu106-v2020.2-final/) directory using below command

$ unzip -o <path-to-AR-package>/vcu_trd_v4l2_ctrlsw_enc_AR76096.zip

• Create a soft link of llp2 psddr nv12 design dtsi file to system-user.dtsi using below command

$ cd project-spec/meta-user/recipes-bsp/device-tree/files/
$ ln -sf vcu_llp2_psddr_hdmi.dtsi system-user.dtsi
$ cd ../../../../../

• Build the PetaLinux project

$ petalinux-build

• Build SDK components to use it as sysroot for application development (Optional).

$ petalinux-build --sdk
$ petalinux-package --sysroot

• Create a boot image (BOOT.BIN) including FSBL, ATF, bitstream, and u-boot.

$ cd images/linux
$ petalinux-package --boot --fsbl zynqmp_fsbl.elf --u-boot u-boot.elf --pmufw pmufw.elf --fpga system.bit

• Copy the generated boot image and Linux image to the SD card directory.

$ cp BOOT.BIN image.ub boot.scr $TRD_HOME/images/vcu_llp2_hdmi_nv12/

2.4 Run Flow

The v4l2_capture_ctrlsw_enc is a command-line Linux application. It requires an input configuration file to be provided in the plain text with the necessary encoder configuration options.

Run below command to check all available configuration options:

$ v4l2_capture_ctrlsw_enc --help-cfg
OR
$ v4l2_capture_ctrlsw_enc --help-cfg-json

Using above configuration options, create a required configuration file according to your use-case. You can also use sample configuration files from /usr/local/etc/v4l2-capture-ctrlsw-enc/sample_cfg/ and tweak parameters as required.

Below are some example pipelines using v4l2_capture_ctrlsw_enc.

• Run following v4l2_capture_ctrlsw_enc command to run Low-Latency (LLP1) stream-out use-case. where 192.168.25.89 is host/client ip address.

$ v4l2_capture_ctrlsw_enc -cfg /usr/local/etc/v4l2-capture-ctrlsw-enc/sample_cfg/4kp60_HEVC.cfg --slicelat --hostip 192.168.25.89

• Run following gst-launch-1.0 command on client to display stream-in NV12 video on HDMI-Tx using Low-Latency(LLP1) GStreamer pipeline.

$ gst-launch-1.0 udpsrc port=5004 buffer-size=60000000 caps="application/x-rtp, media=video, clock-rate=90000, payload=96, encoding-name=H265" ! rtpjitterbuffer latency=7 ! rtph265depay ! h265parse ! video/x-h265, alignment=nal ! omxh265dec low-latency=1 ! video/x-raw ! queue max-size-bytes=0 ! fpsdisplaysink name=fpssink text-overlay=false 'video-sink=kmssink bus-id=a0070000.v_mix hold-extra-sample=1 show-preroll-frame=false sync=true ' sync=true -v 

• Run following v4l2_capture_ctrlsw_enc command to run Xilinx’s Ultra Low-Latency (LLP2) stream-out use-case. where 192.168.25.89 is host/client ip address.

$ v4l2_capture_ctrlsw_enc -cfg /usr/local/etc/v4l2-capture-ctrlsw-enc/sample_cfg/4kp60_HEVC.cfg --xlnx-slicelat --hostip 192.168.25.89

• Run following gst-launch-1.0 command on client to display stream-in NV12 video on HDMI-Tx using Xilinx’s Ultra Low-Latency(LLP2) GStreamer pipeline.

$ gst-launch-1.0 udpsrc port=5004 buffer-size=60000000 caps="application/x-rtp, media=video, clock-rate=90000, payload=96, encoding-name=H265" ! rtpjitterbuffer latency=7 ! rtph265depay ! h265parse ! video/x-h265, alignment=nal ! omxh265dec low-latency=1 ! video/x-raw\(memory:XLNXLL\) ! queue max-size-bytes=0 ! fpsdisplaysink name=fpssink text-overlay=false 'video-sink=kmssink bus-id=a0070000.v_mix hold-extra-sample=1 show-preroll-frame=false sync=true ' sync=true -v

• Run following v4l2_capture_ctrlsw_enc command to run Xilinx’s Ultra Low-Latency (LLP2) record use-case.

$ v4l2_capture_ctrlsw_enc -cfg /usr/local/etc/v4l2-capture-ctrlsw-enc/sample_cfg/4kp60_HEVC.cfg --xlnx-slicelat --record

3 Other Information

3.1 Known Issues

• For Petalinux related known issues please refer: PetaLinux 2020.2 - Product Update Release Notes and Known Issues

• For VCU related known issues please refer AR# 66763: LogiCORE H.264/H.265 Video Codec Unit (VCU) - Release Notes and Known Issues and Xilinx Zynq UltraScale+ MPSoC Video Codec Unit.

• To reduce performance issues with llp2 4x serial pipelines, please refer to chapter# 40 of Section VI: Appendices for IRQ Balancing scheme in PG252.

• For Out of Memory(OOM) killer error on UART console in long run, please refer to AR# 75900: Why do I see out of memory messages in UART console in long run of VCU TRD multi-stream designs?

...

3.2 Limitations

• For Petalinux related limitations please refer: PetaLinux 2020.2 - Product Update Release Notes and Known Issues

• For VCU related limitations please refer AR# 66763: LogiCORE H.264/H.265 Video Codec Unit (VCU) - Release Notes and Known Issues , Xilinx Zynq UltraScale+ MPSoC Video Codec Unit and PG252.

...

3.3 Optimum VCU Encoder parameters for use-cases

Video streaming:

• Video streaming use-case requires very stable bitrate graph for all pictures.

• It is good to avoid periodic large Intra pictures during the encoding session

• Low-latency rate control (hardware RC) is the preferred control-rate for video streaming, it tries to maintain equal amount frame sizes for all pictures.

• Good to avoid periodic Intra frames instead use low-delay-p (IPPPPP…)

• VBR is not a preferred mode of streaming.

...

• Enable profile=high and use qp-mode=auto for low-bitrate encoding use-cases.

• The high profile enables 8x8 transform which results in better video quality at low bitrates.

...

4 Appendix A - Input Configuration File (input.cfg)

The example configuration files are stored at /media/card/config/ folder.

...

...

5 Appendix B - HDMI-Rx/Tx Link-up and GStreamer Commands

This section covers configuration of HDMI-Rx using media-ctl utility and HDMI-Tx using modetest utility, along with demonstrating HDMI-Rx/Tx link-up issues and steps to switch HDMI-Rx resolution. It also contains sample GStreamer Low-Latency NV12 and Xilinx’s Ultra Low-Latency NV12 Audio+Video pipelines for Display, Stream-In and Stream-Out use-cases.

...

...