Zynq UltraScale＋ MPSoC VCU TRD 2019.2 - Multi stream Audio-Video Design

...

1.1 Board Setup

Refer below link for Board Setup

Zynq UltraScale+ MPSoC VCU TRD 2019.2 Board Setup
I2S Audio. signals from MPSoC PL fabric are connected to PMOD0 GPIO Header (J55 - right angle female connector )
Pmod I2S2 Add on card connects to J55 connector and its Master/Slave select jumper (JP1) should be placed into the Slave (SLV) position

1.2 Run Flow

The TRD package is released with the source code, Vivado project, Petalinux BSP, and SD card image that enables the user to run the demonstration. It also includes the binaries necessary to configure and boot the ZCU106 board. Prior to running the steps mentioned in this wiki page, download the TRD package and extract its contents to a directory referred to as ‘TRD_HOME' which is the home directory.

Refer below link to download all TRD contents.

Zynq UltraScale+ MPSoC VCU TRD 2019.2 Download zip

TRD package contents are placed in the following directory structure. The user needs to copy all the files from the $TRD_HOME/images/vcu_audio/ to FAT32 formatted SD card directory.

Code Block

theme	Midnight

└── rdf0428-zcu106-vcu-trd-2019-2
	├── apu
	│   └── vcu_petalinux_bsp
	├── images
	│   ├── vcu_10g
	│   ├── vcu_audio
	│   ├── vcu_hdmi_multistream_xv20
	│   ├── vcu_hdmi_rx
	│   ├── vcu_hdmi_tx
	│   ├── vcu_llp2_hdmi_nv12
	│   ├── vcu_llp2_hdmi_nv16
	│   ├── vcu_llp2_hdmi_xv20
	│   ├── vcu_llp2_sdi_xv20
	│   ├── vcu_multistream_nv12
	│   ├── vcu_pcie
	│   ├── vcu_sdirx
	│   ├── vcu_sditx
	│   └── vcu_sdi_xv20
	├── pcie_host_package
	│   ├── COPYING
	│   ├── etc
	│   ├── include
	│   ├── libxdma
	│   ├── LICENSE
	│   ├── README.md
	│   ├── tools
	│   └── xdma
	├── pl
	│   ├── constrs
	│   ├── designs
	│   ├── prebuild
	│   ├── README.md
	│   └── srcs
	└── README.txt

TRD package contents specific to Multistream Audio-Video design is placed in the following directory structure.

Code Block

theme	Midnight

└── rdf0428-zcu106-vcu-trd-2019-2
	├── apu
	│   └── vcu_petalinux_bsp
	├── images
	│   ├── vcu_audio
	│	│	├── autostart.sh
	│	│	├── bin
	│	│	├── BOOT.BIN
	│	│	├── config
	│	│	├── image.ub
	│	│	├── system.dtb
	│	│	└── vcu
	├── pcie_host_package
	│   ├── COPYING
	│   ├── etc
	│   ├── include
	│   ├── libxdma
	│   ├── LICENSE
	│   ├── README.md
	│   ├── tools
	│   └── xdma
	├── pl
	│   ├── constrs
	│   ├── designs
	│   ├── prebuild
	│   ├── README.md
	│   └── srcs
	└── README.txt

configuration files(input.cfg) for various Resolutions are placed in the following directory structure in /media/card.

Code Block

theme	Midnight

config
├── 1-4kp60
│   ├── Display
│   ├── Record
│   ├── Stream-in
│   └── Stream-out
├── 2-1080p60
│   ├── Display
│   ├── Record
│   ├── Stream-in
│   └── Stream-out
├── 2-4kp30
│   ├── Display
│   ├── Record
│   ├── Stream-in
│   └── Stream-out
└── input.cfg

1.2.1 GStreamer Application (vcu_gst_app)

The vcu_gst_app is a command-line multi-threaded Linux application. The command-line application requires an input configuration file (input.cfg) to be provided in the plain text.

Execution of the application is shown below:

Code Block

theme	Midnight

% vcu_gst_app <path to *.cfg file>

Example:

4kp60 HEVC_HIGH Display Pipeline execution

Code Block

theme	Midnight

% vcu_gst_app /media/card/config/Single_4kp60_HDMI_HDMI/Display/Single_HDMI_HDMI_4kp60_HEVC_HIGH.cfg

4kp60 HEVC_HIGH Record Pipeline execution

Code Block

theme	Midnight

% vcu_gst_app /media/card/config/Single_4kp60_HDMI_HDMI/Record/Single_HDMI_HDMI_4kp60_HEVC_HIGH.cfg

4kp60 HEVC_HIGH Stream-out Pipeline execution

Code Block

theme	Midnight

% vcu_gst_app /media/card/config/Single_4kp60_HDMI_HDMI/Stream-out/Single_HDMI_HDMI_4kp60_HEVC_HIGH.cfg

4kp60 HEVC_HIGH Stream-in Pipeline execution

Code Block

theme	Midnight

% vcu_gst_app /media/card/config/Single_4kp60_HDMI_HDMI/Stream-in/Single_HDMI_HDMI_4kp60_HEVC_HIGH.cfg

NOTE: Make sure HDMI-Rx should be configured to 4kp60 mode.

To measure the latency of the pipeline, run the below command. The latency data is huge, so dump it to a file.

Code Block

theme	Midnight

% GST_DEBUG="GST_TRACER:7" GST_TRACERS="latency;scheduletime" ./vcu_gst_app ./input.cfg >& dump_log.txt

...

Follow the below steps to switch the HDMI-Rx resolution from 1080p60 to 4kp60.
- Check current HDMI Input Source Resolution (1080p60) by following the above-mentioned steps.
- Run vcu_gst_app for current HDMI resolution (1080p60) by executing the following command.

Code Block

theme	Midnight

$ vcu_gst_app /media/card/config/input.cfg

Below configurations needs to be set in input.cfg for HDMI-1080p60.

Code Block

theme	Midnight

Common Configuration    : START
Num Of Input            : 1
Output                  : HDMI
Out Type                : Display
Frame Rate              : 60
Exit

Input Configuration     : START
Input Num               : 1
Input Type              : hdmi
Raw                     : TRUE
Width                   : 1920
Height                  : 1080
Enable SCD              : TRUE
Exit

- Change Resolution of HDMI Input Source from 1080p60 to 4kp60 by following the below steps.
  - Set the HDMI source resolution to 4kp60 (Home page → settings → display & Sound → Resolution → change to 4kp60).
  - Save the configuration to take place the change.
- Verify the desired HDMI Input Source Resolution (4kp60) by following the above-mentioned steps.

If HDMI Tx link-up issue is observed after Linux booting, use the following command:

Code Block

theme	Midnight

$ modetest -D a0070000.v_mix -s 40:3840x2160-60@AR24 -w 36:"alpha":0

Run the following gst-launch-1.0 command to capture and play raw HDMI video and Audio using the GStreamer pipeline.

Code Block

theme	Midnight

$ gst-launch-1.0 v4l2src device=/dev/video0 io-mode=4 ! video/x-raw, width=3840, height=2160, framerate=60/1, format=NV12 ! queue max-size-bytes=0 ! kmssink bus-id="a0070000.v_mix" alsasrc device=hw:2,1 ! audio/x-raw, rate=48000, channels=2, format=S24_32LE ! audioconvert ! audioresample ! audio/x-raw, rate=48000, channels=2, format=S24_32LE ! queue ! alsasink device="hw:2,0"

Run the following gst-launch-1.0 command to capture and play processed(capture → encode → decode → display) HDMI video and raw HDMI Audio using the GStreamer pipeline.

Code Block

theme	Midnight

$ gst-launch-1.0 v4l2src device=/dev/video0 io-mode=4 ! video/x-raw, width=3840, height=2160, framerate=60/1, format=NV12 ! omxh265enc qp-mode=auto gop-mode=basic gop-length=60 b-frames=0 target-bitrate=60000 num-slices=8 control-rate=constant prefetch-buffer=true low-bandwidth=false filler-data=true cpb-size=1000 initial-delay=500 ! video/x-h265, profile=main, alignment=au ! queue ! omxh265dec internal-entropy-buffers=5 low-latency=0 ! queue max-size-bytes=0 ! kmssink bus-id="a0070000.v_mix" alsasrc device=hw:2,1 ! audio/x-raw, rate=48000, channels=2, format=S24_32LE ! audioconvert ! audioresample ! audio/x-raw, rate=48000, channels=2, format=S24_32LE ! queue ! alsasink device="hw:2,0"

Run the following gst-launch-1.0 command to record HDMI video and audio using the GStreamer pipeline.

Code Block

theme	Midnight

$ gst-launch-1.0 v4l2src device=/dev/video0 io-mode=4 num-buffers=3600 ! video/x-raw, format=NV12, width=3840, height=2160, framerate=60/1 ! omxh265enc qp-mode=auto gop-mode=basic gop-length=60 b-frames=0 target-bitrate=60000 num-slices=8 control-rate=constant prefetch-buffer=true low-bandwidth=false filler-data=true cpb-size=1000 initial-delay=500 ! video/x-h265, profile=main, alignment=au ! queue ! mux. alsasrc device=hw:2,1 num-buffers=3600 ! audio/x-raw, format=S24_32LE, rate=48000, channels=2 ! queue ! audioconvert ! audioresample ! faac ! aacparse ! mpegtsmux name=mux ! filesink location = "/media/usb/test.ts"

NOTE: File location should be USB-3.0 to avoid the read-write bandwidth issue.

Run the following gst-launch-1.0 command to play the recorded file using the GStreamer pipeline.

Code Block

theme	Midnight

$ gst-launch-1.0 uridecodebin uri="file:///media/usb/test.ts" name=decode ! queue max-size-bytes=0 ! kmssink bus-id="a0070000.v_mix" decode. ! audioconvert ! audioresample ! audio/x-raw, rate=48000, channnels=2, format=S24_32LE ! queue ! alsasink device="hw:2,0"

NOTE: File location should be USB-3.0 to avoid the read-write bandwidth issue.

Run the following gst-launch-1.0 command to stream-out HDMI video and audio using the GStreamer pipeline.

Code Block

theme	Midnight

$ gst-launch-1.0 v4l2src device=/dev/video0 io-mode=4 ! video/x-raw, format=NV12, width=3840, height=2160, framerate=60/1 ! omxh265enc qp-mode=auto gop-mode=basic gop-length=60 b-frames=0 target-bitrate=60000 num-slices=8 control-rate=constant prefetch-buffer=true low-bandwidth=false filler-data=true cpb-size=1000 initial-delay=500 periodicity-idr=60 ! video/x-h265, profile=main, alignment=au ! queue ! mux. alsasrc device=hw:2,1 ! audio/x-raw, format=S24_32LE, rate=48000, channels=2 ! queue ! audioconvert ! audioresample ! faac ! aacparse ! mpegtsmux name=mux ! rtpmp2tpay ! udpsink host=192.168.25.89 port=5004

NOTE: Here 192.168.25.89 is host/client IP address and 5004 is port no.

Run the following gst-launch-1.0 command to play stream-in video and audio using the Gstreamer pipeline where 5004 is port no.

Code Block

theme	Midnight

$ gst-launch-1.0 udpsrc port=5004 buffer-size=60000000 caps="application/x-rtp, clock-rate=90000" ! rtpjitterbuffer latency=1000 ! rtpmp2tdepay ! tsparse ! video/mpegts ! tsdemux name=demux demux. ! queue ! h265parse ! video/x-h265, profile=main, alignment=au ! omxh265dec internal-entropy-buffers=5 low-latency=0 ! queue max-size-bytes=0 ! kmssink bus-id="a0070000.v_mix" demux. ! queue ! aacparse ! faad ! audioconvert ! audioresample ! audio/x-raw, rate=48000, channnels=2, format=S24_32LE ! alsasink device="hw:2,0"

GST LAUNCH COMMANDS FOR MIPI video, I2S Audio:

Run the following gst-launch-1.0 command to capture and play raw MIPI video and I2S Audio using the GStreamer pipeline.

Code Block

theme	Midnight

$ gst-launch-1.0 v4l2src device=/dev/video1 io-mode=4 ! video/x-raw, width=3840, height=2160, framerate=60/1, format=NV12 ! queue max-size-bytes=0 ! kmssink bus-id="a0070000.v_mix" alsasrc device=hw:2,1 ! audio/x-raw, rate=48000, channels=2, format=S24_32LE ! audioconvert ! audioresample ! audio/x-raw, rate=48000, channels=2, format=S24_32LE ! queue ! alsasink device="hw:0,0"

Run the following gst-launch-1.0 command to capture and play processed(capture → encode → decode → display) MIPI video and raw I2S Audio using the GStreamer pipeline.

Code Block

theme	Midnight

$ gst-launch-1.0 v4l2src device=/dev/video1 io-mode=4 ! video/x-raw, width=3840, height=2160, framerate=60/1, format=NV12 ! omxh265enc qp-mode=auto gop-mode=basic gop-length=60 b-frames=0 target-bitrate=60000 num-slices=8 control-rate=constant prefetch-buffer=true low-bandwidth=false filler-data=true cpb-size=1000 initial-delay=500 ! video/x-h265, profile=main, alignment=au ! queue ! omxh265dec internal-entropy-buffers=5 low-latency=0 ! queue max-size-bytes=0 ! kmssink bus-id="a0070000.v_mix" alsasrc device=hw:2,1 ! audio/x-raw, rate=48000, channels=2, format=S24_32LE ! audioconvert ! audioresample ! audio/x-raw, rate=48000, channels=2, format=S24_32LE ! queue ! alsasink device="hw:0,0"

Run the following gst-launch-1.0 command to record MIPI video and I2S audio using the GStreamer pipeline.

Code Block

theme	Midnight

$ gst-launch-1.0 v4l2src device=/dev/video1 io-mode=4 num-buffers=3600 ! video/x-raw, format=NV12, width=3840, height=2160, framerate=60/1 ! omxh265enc qp-mode=auto gop-mode=basic gop-length=60 b-frames=0 target-bitrate=60000 num-slices=8 control-rate=constant prefetch-buffer=true low-bandwidth=false filler-data=true cpb-size=1000 initial-delay=500 ! queue ! video/x-h265, profile=main, alignment=au ! mux. alsasrc device=hw:0,1 num-buffers=3600 ! audio/x-raw, format=S24_32LE, rate=48000, channels=2 ! queue ! audioconvert ! audioresample ! faac ! mpegtsmux name=mux ! filesink location = "/media/usb/test.ts"

NOTE: File location should be USB-3.0 to avoid the read-write bandwidth issue.

Run the following gst-launch-1.0 command to play the recorded file using the GStreamer pipeline.

Code Block

theme	Midnight

$ gst-launch-1.0 uridecodebin uri="file:///media/usb/test.ts" name=decode ! queue max-size-bytes=0 ! kmssink bus-id="a0070000.v_mix" decode. ! audioconvert ! audioresample ! audio/x-raw, rate=48000, channnels=2, format=S24_32LE ! queue ! alsasink device="hw:0,0"

NOTE: File location should be USB-3.0 to avoid the read-write bandwidth issue.

Run the following gst-launch-1.0 command to stream-out MIPI video and I2S audio using the GStreamer pipeline.

Code Block

theme	Midnight

$ gst-launch-1.0 v4l2src device=/dev/video1 io-mode=4 ! video/x-raw, format=NV12, width=3840, height=2160, framerate=60/1 ! omxh265enc qp-mode=auto gop-mode=basic gop-length=60 b-frames=0 target-bitrate=60000 num-slices=8 control-rate=constant prefetch-buffer=true low-bandwidth=false filler-data=true cpb-size=1000 initial-delay=500 periodicity-idr=60 ! video/x-h265, profile=main, alignment=au ! queue ! mux. alsasrc device=hw:0,1 ! audio/x-raw, format=S24_32LE, rate=48000, channels=2 ! queue ! audioconvert ! audioresample ! faac ! aacparse ! mpegtsmux name=mux ! rtpmp2tpay ! udpsink host=192.168.25.89 port=5004

NOTE: Here 192.168.25.89 is host/client IP address and 5004 is port no.

Run the following gst-launch-1.0 command to play stream-in video and audio using the Gstreamer pipeline where 5004 is port no.

Code Block

theme	Midnight

$ gst-launch-1.0 udpsrc port=5004 buffer-size=60000000 caps="application/x-rtp, clock-rate=90000" ! rtpjitterbuffer latency=1000 ! rtpmp2tdepay ! tsparse ! video/mpegts ! tsdemux name=demux demux. ! queue ! h265parse ! video/x-h265, profile=main, alignment=au ! omxh265dec internal-entropy-buffers=5 low-latency=0 ! queue max-size-bytes=0 ! kmssink bus-id="a0070000.v_mix" demux. ! queue ! aacparse ! faad ! audioconvert ! audioresample ! audio/x-raw, rate=48000, channnels=2, format=S24_32LE ! alsasink device="hw:0,0"

Notes for gst-launch-1.0 commands:

Make sure the HDMI-Rx media pipeline is configured for 4kp60 resolution and source/sink has the same color format. Run below xmedia-ctl commands to set the resolution and format of the HDMI scaler node.

When HDMI Input Source is NVIDIA SHIELD

Code Block

theme	Midnight

$ xmedia-ctl -d /dev/media1 -V "\"a0080000.v_proc_ss\":0  [fmt:RBG888_1X24/3840x2160 field:none]"
$ xmedia-ctl -d /dev/media1 -V "\"a0080000.v_proc_ss\":1  [fmt:VYYUYY8_1X24/3840x2160 field:none]"

NOTE: Make sure NVIDIA SHIELD is configured for 4kp resolution and RGB888 color format.

When HDMI Input Source is ABOX

Code Block

theme	Midnight

$ xmedia-ctl -d /dev/media1 -V "\"a0080000.v_proc_ss\":0  [fmt:VYYUYY8_1X24/3840x2160 field:none]"
$ xmedia-ctl -d /dev/media1 -V "\"a0080000.v_proc_ss\":1  [fmt:VYYUYY8_1X24/3840x2160 field:none]"

NOTE: Make sure ABOX is configured for 4kp resolution and VYYUYY8 color format.

Versions Compared

Old Version 20

New Version Current

Key

Zynq UltraScale＋ MPSoC VCU TRD 2019.2 - Multi stream Audio-Video Design

1.1 Board Setup

1.2 Run Flow

1.2.1 GStreamer Application (vcu_gst_app)

Page Comparison

Versions Compared

Old Version 20

New Version Current

Key

1.1 Board Setup

1.2 Run Flow

1.2.1 GStreamer Application (vcu_gst_app)