Zynq UltraScale+ MPSoC ZCU106 VCU Multi-Stream ROI TRD using Avnet Quad Sensor 2021.1

This page provides all the information related to VCU Multi-Stream ROI TRD using Avnet Quad Sensor design for ZCU106.

Table of Contents

1 Overview

The primary goal of this VCU Multi-Stream ROI TRD using Avnet Quad Sensor design is to demonstrate the use of Deep learning Processor Unit (DPU) block for extracting the Region of Interest (ROI) from input video frames and to use this information to perform ROI based encoding using Video Codec Unit (VCU) encoder hard block present in Zynq UltraScale+ EV devices. Video captures from the quad sensor connected through MIPI CSI-2 Rx which is implemented in the PL. The Avnet Multi-Camera FMC module is used to capture four video streams through a MIPI CSI-2 interface.

The design will serve as a platform to accelerate Deep Neural Network inference algorithms using DPU and demonstrate the ROI feature of VCU encoder. The design uses a Deep Convolutional Neural Network (CNN) named Densebox, running on DPU to extract ROI Information (e.g. ‘face’ in this case). The design will also use Vitis Video Analytics SDK (VVAS) framework to take leverage of its rich set of highly optimized and ready to use - Kernels and GStreamer plugins.

The Design will use Vivado IPI flow for building the hardware platform and Xilinx Yocto Petalinux flow for software design. It will use Xilinx IP and Software driver to demonstrate the capabilities of different components.

The Vitis platform will be created from the Vivado/PetaLinux build artifacts, and then using the Vitis acceleration flow will be used to insert the DPU into the platform to create the final bitstream.

The following figure shows streaming pipeline use-case with enhanced ROI + face detection model on ZCU106. For a detailed view of VVAS block, please refer to Section 1.3.3 - GStreamer Pipeline Flow.

Streaming: Face detection with enhanced ROI on ZCU106.

This ZCU106 VCU Multi Stream ROI TRD design supports only encoding feature of VCU. For decoding on ZCU106 Board-2 setup, needs to use VCU HDMI Single Stream ROI TRD design.

1.1 System Architecture

The following figure shows the block diagram of the ROI design

1.2 Hardware Architecture

This section gives a detailed description of the blocks used in the hardware design. The functional block diagram of the design is shown in the below figure.

There are Five primary Sections in the design.

  • MIPI Capture Pipeline:

    • Captures video frame buffers from Capture source (Avnet Quad Sensor ) at 1080p30 resolution

    • The AXI Switch sends the Captured videos to Multiple streams in Round-Robin method

    •  Each stream writes the frame buffers into DDR Memory with Frame Buffer Write IP

  • Multi-Scaler Block:

    • Reads the Video Buffers from DDR Memory only for first two sensors

    • Scales down the buffer to the 640x360 size (suitable for DPU)

    • Converts the format from NV12 to BGR

    • Writes the downscaled buffer to DDR Memory

  • DPU Block: 

    • Reads the downscaled buffers from DDR Memory only for first two sensors

    • Runs the Densebox algorithm to generate the ROI information for each frame buffer

    • Passes the ROI information to VCU Encoder

  • VCU Encoder: 

    • Reads the 4 x 1080p30 NV12 Buffer from DDR Memory

    • Receives the ROI metadata from DPU IP only from first two sensors.

    • Encodes the video buffers based on the ROI Information for first two sensors

    • Encodes the video buffers for other two sensors

    • Finally writes the encoded stream to DDR Memory

  • PS GEM:

    • Reads the Encoder stream from DDR Memory

    • Stream-out the encoded stream via Ethernet

This design supports the following video interfaces:

Sources

  • MIPI capture pipeline implemented in the PL

Sinks

  • Stream-out on network or internet

VCU Codec

  • Video Encoder capability using VCU hard block in PL 

  • H.264/H.265 encoding

  • Encoder parameter configuration using OMX interface

DPU

Zynq DPU IP

Streaming Interfaces

1G Ethernet PS GEM

Video Format

NV12

Supported Resolution

4 x 1080P30

1.3 VCU ROI Software

1.3.1 Vitis Video Analytics SDK (VVAS)

VVAS is being developed to provide easy to use and scalable framework using which users will be able to build their solutions on Xilinx FPGA. VVAS provides infrastructure that will be covering a wide variety of applications in Embedded, Vision, Datacenter, Machine Learning, Automotive and many other domains.

VVAS Provides a set of generic framework plugins that abstracts the complexities of writing GStreamer plugin. These framework plugins interacts with the kernel libraries through a simple VVAS kernel interface. Using this VVAS Kernel interface, user can easily integrate and test their kernels in GStreamer framework.

VVAS also provides a rich set of highly optimized, ready to use, Kernels and GStreamer plugins, like video encoder, video decoder, multiscalar, ML, bounding box cropping etc. so that users can create his/her applications in very short span of time.

VVAS will also provide the infrastructure needed to bridge the gap between Edge and Cloud solutions.

In VCU Multi Stream ROI TRD, which is using Avnet Quad Sensor design, we’ve used the following VVAS plugins:

  • ivas_xfilter: The ivas_xfilter efficiently works with hard-kernel/soft-kernel/software (user-space) acceleration software library types. It can operate in Passthrough/in-place/transform mode. In Multi-Stream ROI TRD using Avnet Quad Sensor design, it was used as in-place mode so that the acceleration software library can alter the input buffer.

  • ivas_xmetaaffixer: It is used to scale the incoming metadata information for the different resolutions, where metadata received on the master sink pad is scaled in relation to the resolution output slave pads.

  • ivas_xroigen: This plug-in generates ROI metadata information, which is expected by GStreamer OMX encoder plug-ins to encode raw frames with the desired quality parameters (QP) values/ level for specified ROIs.

Refer to VVAS document for more detail on VVAS.

VVAS Top-level Block diagram

1.3.2 Deep Learning Processor Unit (DPU)

DPU is a programmable engine optimized for deep neural networks. It is a group of parameterizable IP cores pre-implemented on the hardware with no place and route required. The DPU is released with the Vitis AI specialized instruction set, allowing efficient implementation of many deep learning networks.

Refer to DPU IP PG338 and UG1354 to know more details on DPU.

The following figure shows the DPU Top-Level Block Diagram.

DPU Top-level Block Diagram

PE - Processing Engine, DPU - Deep Learning Processor Unit, APU - Application Processing Unit

The DPU IP can be implemented in the programmable logic (PL) of the selected Zynq® UltraScale+™ MPSoC device with direct connections to the processing system (PS). The DPU requires instructions to implement a neural network and accessible memory locations for input images as well as temporary and output data. A program running on the Application Processing Unit (APU) is also required to service interrupts and coordinate data transfers.

The following figure shows the sequence of operations performed on the DPU device.

The following sequence of steps are performed to access and run face detection using the DPU device:

  1. DPU device is initialized

  2. Instantiate a DPU Task from DPU Kernel and allocate corresponding DPU memory buffer

  3. Set the input image to created DPU task

  4. Run the DPU task to find the faces from the input image

  5. DPU device is uninitialized

1.3.3 GStreamer Pipelines Flow

The GStreamer plugin demonstrates the DPU capabilities with Xilinx VCU encoder’s ROI (Region of Interest) feature. The plugin will detect ROI (i.e. face co-ordinates) from input frames using DPU IP and pass the detected ROI information to the Xilinx VCU encoder. The following figure shows the data flow for GStreamer pipeline of stream-out use case.

Block Diagram of Stream-out Pipeline

fd = v4l2 frame data, fd' = DPU compatible frame data

As shown in the above figure, the stream-out GStreamer pipeline performs the below list of operations:

  1. Sensors capture the stream through camera and pass it to the FMC module which further pass it to the MIPI CSI-2 Rx interface on ZCU106 Board.

  2. MIPI CSI-2 Rx interface will capture the data in NV12 format and pass it to the tee element, which will split the input stream to metaaffixer and preprocessor elements.

  3. Preprocessors (v4l2convert GStreamer plugin) will scale-down the input frame resolution to 640x360 and convert the data into BGR format as per the input requirement of DPU.

  4. 360p BGR frame will provide to DPU IP (via xfilter plugin) as an input to find ROI(i.e. face co-ordinates).

  5. Extracted ROI information will be passed to VVAS Metaaffixer plug-in along with the original capture stream (via tee), which will embed ROI metadata with the original stream.

  6. ROI Generator will generate ROI SEI events in the stream based on ROI metadata and it is given to VCU encoder, which will encode the input data by encoding ROI regions with high quality as compared to non-ROI region using received ROI information.

  7. Stream-out the encoded data using RTP protocol.

The following figure shows the data flow for the GStreamer pipeline of stream-in use case.

Block Diagram of Stream-in Pipeline

fd = VCU decoded frame data

As shown in the above figure, the stream-in GStreamer pipeline performs the below list of operations:

  1. Stream-in the encoded data using RTP protocol

  2. The Xilinx VCU decoder will decode the data

  3. Display the decoded data on HDMI-Tx display

1.4 Software Tools and System Requirements

Hardware

Required:

  • Two ZCU106 evaluation board rev 1.0 with power cable

  • Monitor with HDMI input supporting 3840x2160 resolution or 1920x1080 resolution (e.g. LG 27UD88, Samsung LU28ES90DS/XL)

  • HDMI 2.0 certified cable

  • Class-10 SD card

  • Avnet Multi-Camera FMC module

  • Ethernet cable

Optional:

  • USB pen drive formatted with the FAT32 file system and hub

  • SATA drive formatted with the FAT32 file system, external power supply, and data cable

Software Tools

Required:

Download, Installation, and Licensing of Vivado Design Suite 2021.1

The Vivado Design Suite User Guide explains how to download and install the Vivado® Design Suite tools, which include the Vivado Integrated Design Environment (IDE), High-Level Synthesis tool, and System Generator for DSP. This guide also provides information about licensing and administering evaluation and full copies of Xilinx design tools and intellectual property (IP) products. The Vivado Design Suite can be downloaded from here.

LogiCORE IP Licensing

The following IP cores require a license to build the design.

  • Video Processing Subsystem (VPSS) - Included with Vivado - PG231

  • MIPI CSI Controller Subsystems (mipi_csi2_rx_subsystem) - Purchase license (Hardware evaluation available) - PG232

To obtain the LogiCORE IP license, please visit the respective IP product page and get the license.

The below table provides the performance information:

Resolution

FPS Achieved

Resolution

FPS Achieved

4 x 1080p30

30

1.5 Board Setup

The below section will provide the information on the ZCU106 board setup for running ROI design.

  1. Connect the Micro USB cable into the ZCU106 Board Micro USB port J83, and the other end into an open USB port on the host PC. This cable is used for UART over USB communication.

  2. Insert the SD card with the images copied into the SD card slot J100. Please find here how to prepare the SD card for a specific design.

  3. Set the SW6 switches as shown in the below Figure. This configures the boot settings to boot from SD.

  4. Connect 12V Power to the ZCU106 6-Pin Molex connector

  5. For a USB storage device, connect the USB hub along with the mouse. (Optional)

  6. For SATA storage device, connect SATA data cable to SATA 3.0 port. (Optional)

  7. For MIPI CSI-2, Insert the Avnet Multi-Camera FMC module into the FMC0 connector and set VADJ to 1.2V

  8. Set up a terminal session between a PC COM port and the serial port on the evaluation board (See the Determine which COM to use to access the USB serial port on the ZCU106 board for more details).

  9. Copy the VCU Multi Stream ROI TRD images into the SD card and insert the SD card on the board

  10. The below images will show how to connect interfaces on the ZCU106 board

1.6 Run Flow

The VCU Multi Stream ROI TRD package is released with the source code, Vivado project, Petalinux BSP, and SD card image that enables the user to run the demonstration. It also includes the binaries necessary to configure and boot the ZCU106 board. Prior to running the steps mentioned in this wiki page, download the VCU Multi Stream ROI TRD package and extract its contents to a directory referred to as $TRD_HOME which is the home directory.

Refer below link to download the VCU Multi Stream ROI TRD package.

  • Zynq UltraScale+ MPSoC VCU Multi Stream ROI TRD 2021.1 Download zip

TRD package contents are placed in the following directory structure.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 rdf0617-zcu106-vcu-multi-stream-roi-2021-1/ ├── apu │   └── vcu_petalinux_bsp │   └── xilinx-vcu-multi-stream-roi-zcu106-v2021.1-final.bsp ├── dpu │   ├── 0001-Added-ZCU106-configuration-to-support-DPU-in-ZCU106.patch │   ├── dpu_conf.vh │   └── vitis_platform │   └── zcu106_dpu ├── image │   ├── bootfiles │   │   ├── bl31.elf │   │   ├── linux.bif │   │   ├── pmufw.elf │   │   ├── system.bit │   │   ├── system.dtb │   │   ├── u-boot.elf │   │   └── zynqmp_fsbl.elf │   ├── license_zcu106_multistream_roi_trd_dpu_xclbin.txt │   ├── README.txt │   ├── sd_card │   │   ├── boot │   │   └── root │   └── sd_card.img ├── pl │   ├── constrs │   │   ├── quad_mipi_rx_ROI.xdc │   │   └── quad_sensor_async.xdc │   ├── designs │   │   └── zcu106_Quad_Sensor_ROI │   ├── prebuild │   │   └── zcu106_Quad_Sensor_ROI_wrapper.xsa │   ├── README.md │   └── srcs │   └── hdl ├── README.txt └── zcu106_vcu_multistream_roi_trd_sources_and_licenses.tar.gz 17 directories, 19 files

The below snippet shows the directory structure of various binary files placed in the $TRD_HOME/image/sd_card/boot directory.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 ├── image └──sd_card └──boot ├── autostart.sh ├── bd.hwh ├── BOOT.BIN ├── boot.scr ├── dpu.xclbin   ├── Image   ├── quad_sensor_isp_tuning.sh ├── quad_sensor_media_graph_setting.sh ├── setup.sh ├── system.dtb ├── vcu │   └── configure_qos.sh ├── vitis │   └── densebox_640_360-zcu102_zcu104_kv260-r1.4.0.tar.gz └── vvas └── json └── kernel_ML.json

1.6.1 Preparing the SD card

There are three ways to prepare the SD card for booting. Each method is detailed below.

Using ready to test image

  • sd_card.img is available at rdf0617-zcu106-vcu-multi-stream-roi-2021-1/image/sd_card.img

  • All the required Vitis packages are already installed in ready to test rdf0617-zcu106-vcu-multi-stream-roi-2021-1/image/sd_card.img

Using Pre-built images

  • To Create SD Card with two partitions: Boot(FAT32+Bootable) and Root(EXT4) Refer this Link.

  • Copy boot content fromrdf0617-zcu106-vcu-multi-stream-roi-2021-1/image/sd_card/boot to Boot partition in SD Card

  • Extract rootfs.ext4 from rdf0617-zcu106-vcu-multi-stream-roi-2021-1/image/sd_card/rootto Root partition in SD Card using

  • Boot the board with Flashed SD Card

Use the Output of the Build Flow

  • To Create SD Card with two partitions: Boot(FAT32+Bootable) and Root(EXT4) Refer this Link.

  • For Build Flow refer this steps and copy mentioned generated dpu build images bd.hwh BOOT.BIN boot.scr dpu.xclbin Image system.dtb into BOOT partition of the SD card and extract generated rootfs.ext4 into ROOT partition of SD Card

  • Copy the mentioned boot content vcu, vitis, vvas, autostart.sh, setup.sh from rdf0617-zcu106-vcu-multi-stream-roi-2021-1/image/sd_card/boot/ directory to Boot partition in SD Card

  • Boot the board with Flashed SD Card

  • All the required Densebox models are already available in rdf0617-zcu106-vcu-multi-stream-roi-2021-1/image/sd_card/boot/vitis directory which are installed automatically during 1st time boot. Please wait till the target setup completes and models are installed.

1.6.2 GStreamer Pipelines using mediasrcbin plugin

This section covers the GSreamer pipelines using mediasrcbin plugin for serial and streaming ROI use-cases. This mediasrcbin plugin is Xilinx specific plugin which is a bin element on top of v4l2src. It parses and configures the media graph of a media device automatically.

Stream-out ( Server ):
→ v4l2convert → ivas_xfilter (DPU) →
Capture (Sensor-1) → tee -| |- ivas_xmetaaffixer → ivas_xroigen → Encode → Stream-out
→→→→→→→→→→→→→→→→→→→

→ v4l2convert → ivas_xfilter (DPU) →
Capture (Sensor-2) → tee -| |- ivas_xmetaaffixer → ivas_xroigen → Encode → Stream-out
→→→→→→→→→→→→→→→→→→→

Capture (Sensor-3) → Encode → Stream-out

Capture (Sensor-4) → Encode → Stream-out

  • Set IP address for server:

    1 ifconfig eth0 192.168.25.90
  • Run the following gst-launch-1.0 command for stream-out pipeline

    • Stream-out Pipeline

      1 gst-launch-1.0 mediasrcbin media-device=/dev/media0 v4l2src0::io-mode=4 v4l2src1::io-mode=4 v4l2src2::io-mode=4 v4l2src3::io-mode=4 name=src src. ! video/x-raw, format=NV12, width=1920, height=1080, framerate=30/1 ! tee name=t0 t0. ! queue ! v4l2convert capture-io-mode=4 output-io-mode=5 ! video/x-raw, width=640, height=360, format=BGR ! queue ! ivas_xfilter kernels-config="/media/card/vvas/json/kernel_ML.json" ! scalem0.sink_master ivas_xmetaaffixer name=scalem0 t0. ! queue ! scalem0.sink_slave_0 scalem0.src_slave_0 ! queue ! ivas_xroigen roi-type=2 roi-qp-delta=-21 roi-max-num=50 insert-roi-sei=true ! queue ! omxh265enc qp-mode=roi gop-mode=basic gop-length=60 b-frames=0 target-bitrate=1500 num-slices=8 control-rate=constant prefetch-buffer=true low-bandwidth=false filler-data=true cpb-size=1000 initial-delay=500 periodicity-idr=60 ! video/x-h265, profile=main, alignment=au ! h265parse ! queue ! mpegtsmux alignment=7 name=mux0 ! rtpmp2tpay ! udpsink host=192.168.25.89 port=5004 src. ! video/x-raw, format=NV12, width=1920, height=1080, framerate=30/1 ! tee name=t1 t1. ! queue ! v4l2video5convert capture-io-mode=4 output-io-mode=5 ! video/x-raw, width=640, height=360, format=BGR ! queue ! ivas_xfilter kernels-config="/media/card/vvas/json/kernel_ML.json" ! scalem1.sink_master ivas_xmetaaffixer name=scalem1 t1. ! queue ! scalem1.sink_slave_0 scalem1.src_slave_0 ! queue ! ivas_xroigen roi-type=2 roi-qp-delta=-21 roi-max-num=50 insert-roi-sei=true ! queue ! omxh265enc qp-mode=roi gop-mode=basic gop-length=60 b-frames=0 target-bitrate=1500 num-slices=8 control-rate=constant prefetch-buffer=true low-bandwidth=false filler-data=true cpb-size=1000 initial-delay=500 periodicity-idr=60 ! video/x-h265, profile=main, alignment=au ! h265parse ! queue ! mpegtsmux alignment=7 name=mux1 ! rtpmp2tpay ! udpsink host=192.168.25.89 port=5008 src. ! video/x-raw, format=NV12, width=1920, height=1080, framerate=30/1 ! omxh265enc qp-mode=auto gop-mode=basic gop-length=60 b-frames=0 target-bitrate=15000 num-slices=8 control-rate=constant prefetch-buffer=true low-bandwidth=false filler-data=true cpb-size=1000 initial-delay=500 periodicity-idr=60 ! video/x-h265, profile=main, alignment=au ! h265parse ! queue ! mpegtsmux alignment=7 name=mux2 ! rtpmp2tpay ! udpsink host=192.168.25.89 port=5012 src. ! video/x-raw, format=NV12, width=1920, height=1080, framerate=30/1 ! omxh265enc qp-mode=auto gop-mode=basic gop-length=60 b-frames=0 target-bitrate=15000 num-slices=8 control-rate=constant prefetch-buffer=true low-bandwidth=false filler-data=true cpb-size=1000 initial-delay=500 periodicity-idr=60 ! video/x-h265, profile=main, alignment=au ! h265parse ! queue ! mpegtsmux alignment=7 name=mux3 ! rtpmp2tpay ! udpsink host=192.168.25.89 port=5016

Here 192.168.25.89 is host/client IP address and 5004, 5008, 5012 & 5016 are port no.

Stream-in ( Client ): Stream-in→ Decode → Display

  • Set IP address for the client:

    1 ifconfig eth0 192.168.25.89
  • Run the following gst-launch-1.0 command for stream-in pipeline where 5004 is port number

    • Stream-in Pipeline

      1 gst-launch-1.0 udpsrc port=5004 buffer-size=60000000 caps="application/x-rtp, clock-rate=90000" ! rtpjitterbuffer latency=1000 ! rtpmp2tdepay ! tsparse ! video/mpegts ! tsdemux name=demux ! queue ! h265parse ! video/x-h265, profile=main, alignment=au ! omxh265dec internal-entropy-buffers=5 low-latency=0 split-input=true ! queue max-size-bytes=0 ! fpsdisplaysink name=fpssink text-overlay=false 'video-sink=kmssink bus-id=a00c0000.v_mix hold-extra-sample=1 show-preroll=false sync=true draw-roi=true roi-rectangle-thickness=3 roi-rectangle-color=<0,90,240>' sync=true -v

1.6.3 GStreamer Pipelines using v4l2src plugin

This section covers the GStreamer pipelines using v4l2src plugin for serial and streaming ROI use-cases.

  • Make sure MIPI CSI-2 Rx media pipeline is configured for 1080p resolution and source/sink have the same color format. Run below script to set resolution and format of MIPI CSI-2 Rx media pipeline nodes where "media0" indicates media node for MIPI CSI-2 Rx input source.

1 $ sh /media/card/quad_sensor_media_graph_setting.sh

Stream-out ( Server ):
→ v4l2convert → ivas_xfilter (DPU) →
Capture (Sensor-1) → tee -| |- ivas_xmetaaffixer → ivas_xroigen → Encode → Stream-out
→→→→→→→→→→→→→→→→→→→

→ v4l2convert → ivas_xfilter (DPU) →
Capture (Sensor-2) → tee -| |- ivas_xmetaaffixer → ivas_xroigen → Encode → Stream-out
→→→→→→→→→→→→→→→→→→→

Capture (Sensor-3) → Encode → Stream-out

Capture (Sensor-4) → Encode → Stream-out

  • Set IP address for server:

    1 ifconfig eth0 192.168.25.90
  • Run the following gst-launch-1.0 command for stream-out pipeline

    • Stream-out Pipeline

      1 gst-launch-1.0 v4l2src device=/dev/video0 io-mode=4 ! video/x-raw, format=NV12, width=1920, height=1080, framerate=30/1 ! tee name=t0 t0. ! queue ! v4l2convert capture-io-mode=4 output-io-mode=5 ! video/x-raw, width=640, height=360, format=BGR ! queue ! ivas_xfilter kernels-config="/media/card/vvas/json/kernel_ML.json" ! scalem0.sink_master ivas_xmetaaffixer name=scalem0 t0. ! queue ! scalem0.sink_slave_0 scalem0.src_slave_0 ! queue ! ivas_xroigen roi-type=2 roi-qp-delta=-21 roi-max-num=50 insert-roi-sei=true ! queue ! omxh265enc qp-mode=roi gop-mode=basic gop-length=60 b-frames=0 target-bitrate=1500 num-slices=8 control-rate=constant prefetch-buffer=true low-bandwidth=false filler-data=true cpb-size=1000 initial-delay=500 periodicity-idr=60 ! video/x-h265, profile=main, alignment=au ! h265parse ! queue ! mpegtsmux alignment=7 name=mux0 ! rtpmp2tpay ! udpsink host=192.168.25.89 port=5004 v4l2src device=/dev/video1 io-mode=4 ! video/x-raw, format=NV12, width=1920, height=1080, framerate=30/1 ! tee name=t1 t1. ! queue ! v4l2video5convert capture-io-mode=4 output-io-mode=5 ! video/x-raw, width=640, height=360, format=BGR ! queue ! ivas_xfilter kernels-config="/media/card/vvas/json/kernel_ML.json" ! scalem1.sink_master ivas_xmetaaffixer name=scalem1 t1. ! queue ! scalem1.sink_slave_0 scalem1.src_slave_0 ! queue ! ivas_xroigen roi-type=2 roi-qp-delta=-21 roi-max-num=50 insert-roi-sei=true ! queue ! omxh265enc qp-mode=roi gop-mode=basic gop-length=60 b-frames=0 target-bitrate=1500 num-slices=8 control-rate=constant prefetch-buffer=true low-bandwidth=false filler-data=true cpb-size=1000 initial-delay=500 periodicity-idr=60 ! video/x-h265, profile=main, alignment=au ! h265parse ! queue ! mpegtsmux alignment=7 name=mux1 ! rtpmp2tpay ! udpsink host=192.168.25.89 port=5008 v4l2src device=/dev/video2 io-mode=4 ! video/x-raw, format=NV12, width=1920, height=1080, framerate=30/1 ! omxh265enc qp-mode=auto gop-mode=basic gop-length=60 b-frames=0 target-bitrate=15000 num-slices=8 control-rate=constant prefetch-buffer=true low-bandwidth=false filler-data=true cpb-size=1000 initial-delay=500 periodicity-idr=60 ! video/x-h265, profile=main, alignment=au ! h265parse ! queue ! mpegtsmux alignment=7 name=mux2 ! rtpmp2tpay ! udpsink host=192.168.25.89 port=5012 v4l2src device=/dev/video3 io-mode=4 ! video/x-raw, format=NV12, width=1920, height=1080, framerate=30/1 ! omxh265enc qp-mode=auto gop-mode=basic gop-length=60 b-frames=0 target-bitrate=15000 num-slices=8 control-rate=constant prefetch-buffer=true low-bandwidth=false filler-data=true cpb-size=1000 initial-delay=500 periodicity-idr=60 ! video/x-h265, profile=main, alignment=au ! h265parse ! queue ! mpegtsmux alignment=7 name=mux3 ! rtpmp2tpay ! udpsink host=192.168.25.89 port=5016

Here 192.168.25.89 is host/client IP address and 5004, 5008, 5012 & 5016 are port no.

Stream-in ( Client ): Stream-in→ Decode → Display

  • Set IP address for the client:

    1 ifconfig eth0 192.168.25.89
  • Run the following gst-launch-1.0 command for stream-in pipeline where 5004 is port number

    • Stream-in Pipeline

      1 gst-launch-1.0 udpsrc port=5004 buffer-size=60000000 caps="application/x-rtp, clock-rate=90000" ! rtpjitterbuffer latency=1000 ! rtpmp2tdepay ! tsparse ! video/mpegts ! tsdemux name=demux ! queue ! h265parse ! video/x-h265, profile=main, alignment=au ! omxh265dec internal-entropy-buffers=5 low-latency=0 split-input=true ! queue max-size-bytes=0 ! fpsdisplaysink name=fpssink text-overlay=false 'video-sink=kmssink bus-id=a00c0000.v_mix hold-extra-sample=1 show-preroll=false sync=true draw-roi=true roi-rectangle-thickness=3 roi-rectangle-color=<0,90,240>' sync=true -v

1.7 Build Flow

Refer below link to download the VCU Multi Stream ROI TRD package.

  • Zynq UltraScale+ MPSoC VCU Multi Stream ROI TRD 2021.1 Download zip

Unzip the released package.

1 unzip </path/to/downloaded/zipfile>/rdf0617-zcu106-vcu-multi-stream-roi-2021-1.zip

The following tutorials assume that the $TRD_HOME environment variable is set as given below.

1 export TRD_HOME=</path/to/downloaded/zipfile>/rdf0617-zcu106-vcu-multi-stream-roi-2021-1

1.7.1 Hardware Build Flow

This section explains the steps to build the hardware platform and generate XSA using the Vivado tool.

Refer to the Vivado Design Suite User Guide: Using the Vivado IDE, UG893, for setting up the Vivado environment.

Refer to the vivado-release-notes-install-license(UG973) for installation.

Make sure that the necessary IP licenses are in place

On Linux:

  • Open a Linux terminal

  • Change directory to $TRD_HOME/pl folder

  • Source Vivado settings.sh

    1 2 bash source <path/to/Vivado-installer>/tool/Vivado/2021.1/settings64.sh
  • Run the following command to create the Vivado IPI project and invoke the GUI and generate XSA required for the platform

    1 vivado -source ./designs/zcu106_Quad_Sensor_ROI/project.tcl
  • The project.tcl script does the following

    • Creates project in the ../pl/build/zcu106_Quad_Sensor_ROI directory

    • Creates IPI Block design with platform interfaces

    • Runs Synthesis and Implementation

    • Builds bitstream with no accelerators

    • Export the HW to XSA (zcu106_Quad_Sensor_ROI_wrapper.xsa)

  • zcu106_Quad_Sensor_ROI_wrapper.xsa is stored at location $TRD_HOME/pl/build/zcu106_Quad_Sensor_ROI/zcu106_Quad_Sensor_ROI.xsa/

  • This XSA is used by Petalinux for platform creation and also by the Vitis Tool for DPU Kernel Integration.

After executing the script, the Vivado IPI block design comes up as shown in the below figure.

 

 

The Platform Setup tab has the settings and AXI Ports, as shown in below image

1.7.1.1 Platform Interfaces

The screenshots below show the platform interfaces that have been made available to the Vitis tool for linking the acceleration IP dynamically

In the case of this reference design, the DPU Kernel will be inserted.

 

After the DPU Kernel is integrated dynamically with the platform using Vitis Flow, the connections are as shown below

  • The DPU Data ports are connected to the HP0 Port(S_AXI_HP0_FPD) of PS .

  • The DPU Instruction port is connected to the S_AXI_HPC1 port of PS

  • The DPU S_AXI_Control port is connected to the M_AXI_HPM0_LPD port of PS through interconnect_hpm0_lpd

  • The DPU interrupt is connected to the axi interrupt controller dynamically

1.7.2 Petalinux build Flow

This tutorial shows how to build the Linux image and boot image using the PetaLinux build tool.

PetaLinux Installation: Refer to the PetaLinux Tools Documentation (UG1144) for installation.

Kernel patches Documentation: Refer this article for Kernel patches required for ZCU106 VCU Multi-Stream ROI TRD using Avnet Quad Sensor BSP.

It is recommended to follow the build steps in sequence

  • Source Petalinux settings.sh

    1 2 bash source <path/to/petalinux-installer>/tool/petalinux-v2021.1-final/settings.sh
  • Create PetaLinux project

    1 2 cd $TRD_HOME/apu/vcu_petalinux_bsp petalinux-create -t project -s xilinx-vcu-multi-stream-roi-zcu106-v2021.1-final.bsp
  • Configure the PetaLinux project

    1 2 cd xilinx-vcu-multi-stream-roi-zcu106-v2021.1-final petalinux-config --silentconfig --get-hw-description=<Path to directory of XSA>
    • For e.g.

    • using the prebuild XSA

      1 petalinux-config --silentconfig --get-hw-description=$TRD_HOME/pl/prebuild/
    • using the XSA generated by running the Hardware project.tcl scripts

      1 petalinux-config --silentconfig --get-hw-description=$TRD_HOME/pl/build/zcu106_Quad_Sensor_ROI/zcu106_Quad_Sensor_ROI.xsa/
  • zcu106_Quad_Sensor_ROI_wrapper.xsa is available in directory zcu106_Quad_Sensor_ROI.xsa

  • Build the PetaLinux project

    1 petalinux-build

1.7.3 Prepare Build Artifacts for Platform Creation

To prepare artifacts required for ZCU106 Vitis platform creation, follow the below steps after petalinux build

  • Go to the petalinux build image directory

    1 cd $TRD_HOME/apu/vcu_petalinux_bsp/xilinx-vcu-multi-stream-roi-zcu106-v2021.1-final/images/linux
  • Create linux.bif file as below in images/linux directory. linux.bif file is required to create ZCU106 Vitis platform which has information related to boot components. After zcu106 vitis platform creation this linux.bif file will be part of the platform, which is required to build DPU and generate final BOOT.BIN

    1 2 3 4 5 6 7 8 9 10 /* linux */ the_ROM_image: { [bootloader, destination_cpu = a53-0] <zynqmp_fsbl.elf> [pmufw_image] <pmufw.elf> [destination_device=pl] <bitstream> [destination_cpu=a53-0, exception_level=el-3, trustzone] <bl31.elf> [destination_cpu=a53-0, load=0x00100000] <system.dtb> [destination_cpu=a53-0, exception_level=el-2] <u-boot.elf> }
  • Copy generated images into boot and image directory by following the below commands. Use created linux.bif to copy into boot directory.

    1 2 3 4 mkdir boot image cp linux.bif bl31.elf pmufw.elf system.dtb u-boot.elf zynqmp_fsbl.elf boot/. cp boot.scr system.dtb image/.

1.7.4 ZCU106 Platform Creation

This section shows how to create a Vitis acceleration platform for the zcu106 using the Vitis IDE.

  • Ready to use vitis platform is available at $TRD_HOME/dpu/vitis_platform/zcu106_dpu

  • It is recommended to follow the build steps in sequence

1 2 source <vitis install path>/Vitis/2021.1/settings64.sh vitis &

Choose project workspace and click on Launch to begin

 

User can create Workspace in any directory. For example $TRD_HOME/vitis_workspace

Launch the New Platform Project dialog box using the following step:
Go to File > New > Platform Project

Provide a project name “zcu106_dpu“ in the Platform project name field and click Next as shown in the below figure

In the Platform Project dialog box, choose Create a new platform from hardware specification (XSA) and provide the XSA path

  • For prebuild XSA use $TRD_HOME/pl/prebuild/ path

  • For generated XSA using vivado build use $TRD_HOME/pl/build/zcu106_Quad_Sensor_ROI/zcu106_Quad_Sensor_ROI.xsa/ path

Use the below setting under Software Specification

  1. Select linux as the operating system, psu_cortexa53 as a processor, and 64-bit architecture to create the platform

  2. Uncheck the box for Generate boot components

  3. Click Finish to create your platform project

 

Go to zcu106_dpu > psu_cortexa53 > linux on psu_cortexa53 and add path of required files in Domain: linux_domain

Give the bif file, boot directory and image path as shown in below figure

  • Use <Path to Petalinux Project>/xilinx-vcu-multi-stream-roi-zcu106-v2021.1-final/images/linux/boot/linux.bif for Bif File

  • Use <Path to Petalinux Project>/xilinx-vcu-multi-stream-roi-zcu106-v2021.1-final/images/linux/boot/ for Boot Components Directory

  • Use <Path to Petalinux Project>/xilinx-vcu-multi-stream-roi-zcu106-v2021.1-final/images/linux/image for FAT32 Partition Directory

Right click on the zcu106_dpu project in the Explorer tab and click on Build Project to generate the platform as shown in the below figure
The Console tab shows the status of the platform generation.

As shown in below image, zcu106_dpu.xpfm is created under zcu106_dpu > export > zcu106_dpu > zcu106_dpu.xpfm

1.7.5 DPU Build

  • It is recommended to follow the build steps in sequence

  • Make sure parted utility is installed for DPU build

  • Clone the Vitis-AI repository and apply patch to add support of ZCU106 in Vitis DPU TRD.

1 2 3 4 git clone https://github.com/Xilinx/Vitis-AI.git cd Vitis-AI git checkout tags/v1.4 git am $TRD_HOME/dpu/0001-Added-ZCU106-configuration-to-support-DPU-in-ZCU106.patch

The following tutorials assume that the DPU_TRD_HOME environment variable is set as given below.

1 export DPU_TRD_HOME =<Vitis AI path>/dsa/DPU-TRD

The following tutorials assume that the Vitis and XRT environment variable is set as given below.

  • Open a Linux terminal. Set the Linux as Bash mode.

    1 2 source <vitis install path>/Vitis/2021.1/settings64.sh source <xrt install path>/packages/setenv.sh
  • The default setting of DPU is B4096 with RAM_USAGE_LOW, CHANNEL_AUGMENTATION_ENABLE, DWCV_ENABLE, POOL_AVG_ENABLE, RELU_LEAKYRELU_RELU6. Read the $DPU_TRD_HOME/prj/Vitis/dpu_conf.vh file to get the details of DPU. You can get all the configurations from PG338. Modify the $DPU_TRD_HOME/prj/Vitis/dpu_conf.vh file can change the default settings.

    • Copy dpu_conf.vh file to enable the URAM for ZCU106 DPU Build, The DPU will replace the bram to the uram.

      1 cp $TRD_HOME/dpu/dpu_conf.vh $DPU_TRD_HOME/prj/Vitis/dpu_conf.vh
  • Build the hardware design

    1 2 3 4 cd $DPU_TRD_HOME/prj/Vitis export EDGE_COMMON_SW=<Path to Petalinux Project>/xilinx-vcu-multi-stream-roi-zcu106-v2021.1-final/images/linux/ export SDX_PLATFORM=<Path to Vitis Workspace>/zcu106_dpu/export/zcu106_dpu/zcu106_dpu.xpfm make KERNEL=DPU DEVICE=zcu106_dpu
  • Generated SD card files are in $DPU_TRD_HOME/prj/Vitis/binary_container_1/sd_card(SD card Format)

  • Copy generated dpu build images from $DPU_TRD_HOME directory to $TRD_HOME directory

    1 2 3 cd $DPU_TRD_HOME/prj/Vitis/binary_container_1/sd_card cp BOOT.BIN boot.scr dpu.xclbin Image system.dtb $TRD_HOME/image/sd_card/boot cp $TRD_HOME/apu/vcu_petalinux_bsp/xilinx-vcu-multi-stream-roi-zcu106-v2021.1-final/images/linux/rootfs.ext4 $TRD_HOME/image/sd_card/root

2 Other Information

2.1 Known Issues

2.2 Limitations

3 Appendix A - MIPI CSI-2 Rx Link-up

  • This section covers configuration of CSI-2 Rx using media-ctl utility, along with demonstrating CSI-2 Rx link-up issues.

  • Run the below command for media node to print media device topology where "media0" represents media node.

    1 media-ctl -p -d /dev/media0
  • When MIPI CSI-2 Rx source is connected, it shows as below:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 root@vcu_roi_trd-2021_1:/media/card# media-ctl -p -d /dev/media0 Media controller API version 5.10.0 Media device information ------------------------ driver xilinx-video model Xilinx Video Composite Device serial bus info hw revision 0x0 driver version 5.10.0 Device topology - entity 1: vcap_csi_p0_scalar_0 output 0 (1 pad, 1 link) type Node subtype V4L flags 0 device node name /dev/video0 pad0: Sink <- "a0040000.v_proc_ss":1 [ENABLED] - entity 5: vcap_csi_p0_scalar_0 output 1 (1 pad, 1 link) type Node subtype V4L flags 0 device node name /dev/video1 pad0: Sink <- "a0080000.v_proc_ss":1 [ENABLED] - entity 9: vcap_csi_p0_scalar_0 output 2 (1 pad, 1 link) type Node subtype V4L flags 0 device node name /dev/video2 pad0: Sink <- "a00c0000.v_proc_ss":1 [ENABLED] - entity 13: vcap_csi_p0_scalar_0 output 3 (1 pad, 1 link) type Node subtype V4L flags 0 device node name /dev/video3 pad0: Sink <- "a0140000.v_proc_ss":1 [ENABLED] - entity 17: amba_pl@0:axis_switch_csi_axis_ (5 pads, 5 links) type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev0 pad0: Sink [fmt:SGRBG8_1X8/1920x1080 field:none colorspace:srgb] <- "a0000000.mipi_csi2_rx_subsystem":1 [ENABLED] pad1: Source [fmt:SGRBG8_1X8/1920x1080 field:none colorspace:srgb] -> "a0190000.v_demosaic":0 [ENABLED] pad2: Source [fmt:SGRBG8_1X8/1920x1080 field:none colorspace:srgb] -> "a01c0000.v_demosaic":0 [ENABLED] pad3: Source [fmt:SGRBG8_1X8/1920x1080 field:none colorspace:srgb] -> "a01f0000.v_demosaic":0 [ENABLED] pad4: Source [fmt:SGRBG8_1X8/1920x1080 field:none colorspace:srgb] -> "a0220000.v_demosaic":0 [ENABLED] - entity 23: a0000000.mipi_csi2_rx_subsystem (2 pads, 2 links) type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev1 pad0: Sink [fmt:SGRBG8_1X8/1920x1080 field:none] <- "max9286 23-0048":4 [ENABLED] pad1: Source [fmt:SGRBG8_1X8/1920x1080 field:none] -> "amba_pl@0:axis_switch_csi_axis_":0 [ENABLED] - entity 26: a0190000.v_demosaic (2 pads, 2 links) type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev2 pad0: Sink [fmt:SGRBG8_1X8/1920x1080 field:none] <- "amba_pl@0:axis_switch_csi_axis_":1 [ENABLED] pad1: Source [fmt:RBG888_1X24/1920x1080 field:none] -> "a0180000.v_gamma_lut":0 [ENABLED] - entity 29: a01c0000.v_demosaic (2 pads, 2 links) type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev3 pad0: Sink [fmt:SGRBG8_1X8/1920x1080 field:none] <- "amba_pl@0:axis_switch_csi_axis_":2 [ENABLED] pad1: Source [fmt:RBG888_1X24/1920x1080 field:none] -> "a01b0000.v_gamma_lut":0 [ENABLED] - entity 32: a01f0000.v_demosaic (2 pads, 2 links) type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev4 pad0: Sink [fmt:SGRBG8_1X8/1920x1080 field:none] <- "amba_pl@0:axis_switch_csi_axis_":3 [ENABLED] pad1: Source [fmt:RBG888_1X24/1920x1080 field:none] -> "a01e0000.v_gamma_lut":0 [ENABLED] - entity 35: a0220000.v_demosaic (2 pads, 2 links) type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev5 pad0: Sink [fmt:SGRBG8_1X8/1920x1080 field:none] <- "amba_pl@0:axis_switch_csi_axis_":4 [ENABLED] pad1: Source [fmt:RBG888_1X24/1920x1080 field:none] -> "a0210000.v_gamma_lut":0 [ENABLED] - entity 38: a0180000.v_gamma_lut (2 pads, 2 links) type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev6 pad0: Sink [fmt:RBG888_1X24/1920x1080 field:none] <- "a0190000.v_demosaic":1 [ENABLED] pad1: Source [fmt:RBG888_1X24/1920x1080 field:none] -> "a0010000.v_proc_ss":0 [ENABLED] - entity 41: a01b0000.v_gamma_lut (2 pads, 2 links) type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev7 pad0: Sink [fmt:RBG888_1X24/1920x1080 field:none] <- "a01c0000.v_demosaic":1 [ENABLED] pad1: Source [fmt:RBG888_1X24/1920x1080 field:none] -> "a0020000.v_proc_ss":0 [ENABLED] - entity 44: a01e0000.v_gamma_lut (2 pads, 2 links) type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev8 pad0: Sink [fmt:RBG888_1X24/1920x1080 field:none] <- "a01f0000.v_demosaic":1 [ENABLED] pad1: Source [fmt:RBG888_1X24/1920x1080 field:none] -> "a0030000.v_proc_ss":0 [ENABLED] - entity 47: a0210000.v_gamma_lut (2 pads, 2 links) type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev9 pad0: Sink [fmt:RBG888_1X24/1920x1080 field:none] <- "a0220000.v_demosaic":1 [ENABLED] pad1: Source [fmt:RBG888_1X24/1920x1080 field:none] -> "a0100000.v_proc_ss":0 [ENABLED] - entity 50: a0010000.v_proc_ss (2 pads, 2 links) type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev10 pad0: Sink [fmt:RBG888_1X24/1920x1080 field:none] <- "a0180000.v_gamma_lut":1 [ENABLED] pad1: Source [fmt:RBG888_1X24/1920x1080 field:none] -> "a0040000.v_proc_ss":0 [ENABLED] - entity 53: a0020000.v_proc_ss (2 pads, 2 links) type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev11 pad0: Sink [fmt:RBG888_1X24/1920x1080 field:none] <- "a01b0000.v_gamma_lut":1 [ENABLED] pad1: Source [fmt:RBG888_1X24/1920x1080 field:none] -> "a0080000.v_proc_ss":0 [ENABLED] - entity 56: a0030000.v_proc_ss (2 pads, 2 links) type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev12 pad0: Sink [fmt:RBG888_1X24/1920x1080 field:none] <- "a01e0000.v_gamma_lut":1 [ENABLED] pad1: Source [fmt:RBG888_1X24/1920x1080 field:none] -> "a00c0000.v_proc_ss":0 [ENABLED] - entity 59: a0100000.v_proc_ss (2 pads, 2 links) type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev13 pad0: Sink [fmt:RBG888_1X24/1920x1080 field:none] <- "a0210000.v_gamma_lut":1 [ENABLED] pad1: Source [fmt:RBG888_1X24/1920x1080 field:none] -> "a0140000.v_proc_ss":0 [ENABLED] - entity 62: a0040000.v_proc_ss (2 pads, 2 links) type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev14 pad0: Sink [fmt:RBG888_1X24/1920x1080 field:none] <- "a0010000.v_proc_ss":1 [ENABLED] pad1: Source [fmt:VYYUYY8_1X24/1920x1080 field:none] -> "vcap_csi_p0_scalar_0 output 0":0 [ENABLED] - entity 65: a0080000.v_proc_ss (2 pads, 2 links) type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev15 pad0: Sink [fmt:RBG888_1X24/1920x1080 field:none] <- "a0020000.v_proc_ss":1 [ENABLED] pad1: Source [fmt:VYYUYY8_1X24/1920x1080 field:none] -> "vcap_csi_p0_scalar_0 output 1":0 [ENABLED] - entity 68: a00c0000.v_proc_ss (2 pads, 2 links) type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev16 pad0: Sink [fmt:RBG888_1X24/1920x1080 field:none] <- "a0030000.v_proc_ss":1 [ENABLED] pad1: Source [fmt:VYYUYY8_1X24/1920x1080 field:none] -> "vcap_csi_p0_scalar_0 output 2":0 [ENABLED] - entity 71: a0140000.v_proc_ss (2 pads, 2 links) type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev17 pad0: Sink [fmt:RBG888_1X24/1920x1080 field:none] <- "a0100000.v_proc_ss":1 [ENABLED] pad1: Source [fmt:VYYUYY8_1X24/1920x1080 field:none] -> "vcap_csi_p0_scalar_0 output 3":0 [ENABLED] - entity 74: max9286 23-0048 (5 pads, 5 links) type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev18 pad0: Sink [fmt:SGRBG8_1X8/1920x1080 field:none] <- "mars 32-0041":0 [ENABLED,IMMUTABLE] pad1: Sink [fmt:SGRBG8_1X8/1920x1080 field:none] <- "mars 33-0042":0 [ENABLED,IMMUTABLE] pad2: Sink [fmt:SGRBG8_1X8/1920x1080 field:none] <- "mars 34-0043":0 [ENABLED,IMMUTABLE] pad3: Sink [fmt:SGRBG8_1X8/1920x1080 field:none] <- "mars 35-0044":0 [ENABLED,IMMUTABLE] pad4: Source [fmt:SGRBG8_1X8/1920x1080 field:none] -> "a0000000.mipi_csi2_rx_subsystem":0 [ENABLED] - entity 80: mars 32-0041 (1 pad, 1 link) type V4L2 subdev subtype Unknown flags 20001 device node name /dev/v4l-subdev19 pad0: Source [fmt:SGRBG8_1X8/1920x1080 field:none] -> "max9286 23-0048":0 [ENABLED,IMMUTABLE] - entity 84: mars 33-0042 (1 pad, 1 link) type V4L2 subdev subtype Unknown flags 20001 device node name /dev/v4l-subdev20 pad0: Source [fmt:SGRBG8_1X8/1920x1080 field:none] -> "max9286 23-0048":1 [ENABLED,IMMUTABLE] - entity 88: mars 34-0043 (1 pad, 1 link) type V4L2 subdev subtype Unknown flags 20001 device node name /dev/v4l-subdev21 pad0: Source [fmt:SGRBG8_1X8/1920x1080 field:none] -> "max9286 23-0048":2 [ENABLED,IMMUTABLE] - entity 92: mars 35-0044 (1 pad, 1 link) type V4L2 subdev subtype Unknown flags 20001 device node name /dev/v4l-subdev22 pad0: Source [fmt:SGRBG8_1X8/1920x1080 field:none] -> "max9286 23-0048":3 [ENABLED,IMMUTABLE] root@vcu_roi_trd-2021_1:/media/card#

When the MIPI CSI-2 Rx source is not connected, it will not show any media node.