Table of Contents maxLevel 1
Table of Contents maxLevel 1
Introduction
This page gives an overview of Xilinx Multi-Scaler driver, which is available as part of the Xilinx Linux distribution.
...
- The driver is tested with YUY2, NV16, NV12, RGB, GRAY8, UYVY and BGR Gstreamer formats only.
- Please check section below on Stride and height Height align issue. Please check here.
Kernel Configuration
The following config options should be enabled in order to build Xilinx Multi-Scaler driver
CONFIG_VIDEO_XILINX
CONFIG_VIDEO_DEV
...
CONFIG_VIDEO_XILINX_MULTISCALER
To check if the options are enabled run the following:
Code Block | ||
---|---|---|
| ||
petalinux-config -c kernel |
When the GUI opens, you can use '/' to search for each config.
The driver is available at,
https://github.com/Xilinx/linux-xlnx/blob/master/drivers/media/platform/xilinx/xilinx-multi-scaler.c
Device Tree Binding
The device tree node will be automatically generated, if the core is configured in the HW design, using the Device Tree BSP.
Steps to generate device-tree is documented here,
http://www.wiki.xilinx.com/Build+Device+Tree+Blob
And a sample binding is shown below and the description of DT property is documented here
Code Block | ||
---|---|---|
| ||
v_multi_scaler_1: v_multi_scaler@a0000000 { clock-names = "ap_clk"; clocks = <&misc_clk_0>; compatible = "xlnx,v-multi-scaler-1.0", "xlnx,v-multi-scaler-v1.0"; interrupt-names = "interrupt"; interrupt-parent = <&gic>; interrupts = <0 89 4>; reg = <0x0 0xa0000000 0x0 0x20000>; reset-gpios = <&gpio 78 1>; xlnx,dma-addr-width = <0x20>; xlnx,max-chan = <8>; xlnx,max-height = <2160>; xlnx,max-width = <3840>; xlnx,num-taps = <12>; xlnx,pixels-per-clock = /bits/ 8 <2>; xlnx,vid-formats = "xrgb8888", "bgr888", "xbgr8888", "xbgr2101010", "uyvy", "y8", "y10", "vuy888", "xvuy8888", "yuvx2101010", "yuyv", "nv12", "nv16", "xv20", "xv15"; }; |
...
A reference design for testing is as below
Multi-Scaler driver creates a video device per channel, which can be visual in /dev directory.
...
Multi-Scaler is tested with the generic gstreamer v4l2videotransform plugin which maps device node to v4l2videoNconvert elements. Where N is the number of device node.
Below is the steps to get the proper gstreamer element for the Multi-Scaler.
Find the Multi-Scaler devices
Gstreamer device name for Multi-Scaler is xm2msc (Xilinx memory to memory video multi-scaler).
In below example the IP is configured for 8 channels.
Code Block | ||||
---|---|---|---|---|
| ||||
# gst-inspect-1.0 -a | grep xm2msc
v4l2video0convert: String. Default: "xm2msc"
v4l2video1convert: String. Default: "xm2msc"
v4l2video2convert: String. Default: "xm2msc"
v4l2video3convert: String. Default: "xm2msc"
v4l2video4convert: String. Default: "xm2msc"
v4l2video5convert: String. Default: "xm2msc"
v4l2video6convert: String. Default: "xm2msc"
v4l2video7convert: String. Default: "xm2msc" |
N-N Use Case:
...
N-N Use Case:
Multi-Scaler is tested with the generic gstreamer v4l2videotransform elements which is part of gst-plugins-good. So no extra library required to run N:N use case.
In below image, it is represented a basic test scenario where multiple v4l2videoNconvert elements are launched as different process opening different node of Multi-Scale. The input for the v4l2videoNconvert is frames of different format and different resolutions and the output is given to filesink which stores the scaled and converted frames in a file.
Commands to test
Below command is an example using Gstreamer. You can open Multiple channels by running v4l2videoNconvert process in parallel.
**Note Gstreamer plugins may change, please refer to Gstreamer documentation for the latest information on using Gstreamer plugins
Code Block | ||||
---|---|---|---|---|
| ||||
gst-launch-1.0 videotestsrc num-buffers=10 ! video/x-raw, width=1280, height=720, format=RGB ! v4l2convert capture-io-mode=4 output-io-mode=4 ! video/x-raw, width=640, height=480, format=RGB ! filesink location=tmp0.rgb |
1 to N Use Case:
Multi-Scaler's 1 to N use case is tested with the xlnxabrscaler plugin.
In below image, it is represented a basic test scenario where multiple v4l2videoNconvert elements are launched as different process opening single input given to xlnxabrscaler elements which opens different node of Multi-ScaleScaler. The input for the v4l2videoNconvert is frames of different format and different resolutions and the output is given to filesink which stores the scaled and converted frames in a filefiles.
Commands to test
Below command is used to test Multi-Scaler with Gstreamer. You can open Multiple channels by running v4l2videoNconvert process in parallel.
Code Block | ||||
---|---|---|---|---|
| ||||
gst-launch-1.0 videotestsrc num-buffers=10 ! video/x-raw, width=1280, height=720, format=RGB ! v4l2video0convert capture-io-mode=4 output-io-mode=4 ! video/x-raw, width=640, height=480, format=RGB ! filesink location=tmp0.rgb |
1 to N Use Case:
Multi-Scaler's 1 to N use case is tested with the xlnxabrscaler plugin.
In below image, it is represented a basic test scenario where single input given to xlnxabrscaler elements which opens different node of Multi-Scaler. The output is given to filesink which stores the scaled and converted frames in files.
Commands to test
For 1 to N Use Case, run commands mention in debugging section.
Mapping table between Gstreamer string and Xilinx Multi-Scaler Supported Formats
Replace the format string in above command with below Gstreamer strings, to test different supported formats
...
Multi-Scaler
Video Format
Prefix with XILINX_M2MSC_FMT_
...
Device-tree
strings
...
V4L2
Format
Prefix with
V4L2_PIX_FMT_
...
Gstreamer
String
...
YUV Player
Format
...
RGBX8
...
Debugging
Enable Dynamic Debugging in kernel.
Run below commands to enable logs on console
Code Block | ||||
---|---|---|---|---|
| ||||
echo 1 > /sys/module/v4l2_mem2mem/parameters/debug
echo 8 > /sys/module/videobuf2_core/parameters/debug
echo 8 > /sys/module/videobuf2_v4l2/parameters/debug
echo 8 > /proc/sys/kernel/printk |
Use Cases
...
For 1 to N Use Case, run commands mention in debugging section.
Mapping table between Gstreamer string and Xilinx Multi-Scaler Supported Formats
Replace the format string in above command with below Gstreamer strings, to test different supported formats
ID | Multi-Scaler Video Format
| Device-tree strings | V4L2 Format
| Gstreamer String | YUV Player Format |
---|---|---|---|---|---|
10 | RGBX8 | xbgr8888 | BGRX32 | ||
11 | YUVX8 | xvuy8888 | XVUY32 | ||
12 | YUYV8 | yuyv | YUYV | YUY2 | YUYV |
15 | RGBX10 | xbgr2101010 | XBGR30 | ||
16 | YUVX10 | yuvx2101010 | XVUY10 | ||
18 | Y_UV8 | nv16 | NV16M | NV16 | NV16 |
19 | Y_UV8_420 | nv12 | NV12M | NV12 | NV12 |
20 | RGB8 | bgr888 | RGB24 | RGB | RGB24 |
21 | YUV8 | vuy888 | VUY24 | ||
22 | Y_UV10 | xv20 | XV20M | XV20 | |
23 | Y_UV10_420 | xv15 | XV15M | XV15 | |
24 | Y8 | y8 | GREY | GRAY8 | Y |
25 | Y10 | y10 | Y10 | Y10 | |
27 | BGRX8 | xrgb8888 | XBGR32 | BGRx | BGR32 |
28 | UYVY8 | uyvy | UYVY | UYVY | UYVY |
29 | BGR8 | rgb888 | BGR24 | BGR | BGR24 |
Debugging
Enable Dynamic Debugging in kernel.
Run below commands to enable logs on console
Code Block | ||||
---|---|---|---|---|
| ||||
echo 1 > /sys/module/v4l2_mem2mem/parameters/debug
echo 8 > /sys/module/videobuf2_core/parameters/debug
echo 8 > /sys/module/videobuf2_v4l2/parameters/debug
echo 8 > /proc/sys/kernel/printk |
Use Cases
1-1 Scaler
Code Block | ||||
---|---|---|---|---|
| ||||
gst-launch-1.0 videotestsrc num-buffers=10 ! video/x-raw, width=1280, height=720, format=RGB ! v4l2convert capture-io-mode=4 output-io-mode=4 ! video/x-raw, width=640, height=480, format=RGB ! filesink location=640X480.rgb |
N-N Scaler
Code Block | ||||
---|---|---|---|---|
| ||||
#!/bin/bash gst-launch-1.0 videotestsrc num-buffers=103 ! video/x-raw, width=1280, height=720, format=RGB ! v4l2video0convertv4l2video1convert capture-io-mode=4 output-io-mode=4 ! video/x-raw, width=640, height=480, format=RGB ! filesink location=640X480.rgb |
N-N Scaler
Code Block | ||||
---|---|---|---|---|
| ||||
#!/bin/bashtmp1.rgb & sleep 3 gst-launch-1.0 videotestsrc num-buffers=3 ! video/x-raw, width=1280, height=720, format=RGB ! v4l2video1convertv4l2video2convert capture-io-mode=4 output-io-mode=4 ! video/x-raw, width=640, height=480, format=RGB ! filesink location=tmp1tmp2.rgb & sleep 3 gst-launch-1.0 videotestsrc num-buffers=3 ! video/x-raw, width=1280, height=720, format=RGB ! v4l2video2convertv4l2video3convert capture-io-mode=4 output-io-mode=4 ! video/x-raw, width=640, height=480, format=RGB ! filesink location=tmp2tmp3.rgb & sleep 3 gst-launch-1.0 videotestsrc num-buffers=3 ! video/x-raw, width=1280, height=720, format=RGB ! v4l2video3convertv4l2video4convert capture-io-mode=4 output-io-mode=4 ! video/x-raw, width=640, height=480, format=RGB ! filesink location=tmp3tmp4.rgb & sleep 3 gst-launch-1.0 videotestsrc num-buffers=3 ! video/x-raw, width=1280, height=720, format=RGB ! v4l2video4convertv4l2video5convert capture-io-mode=4 output-io-mode=4 ! video/x-raw, width=640, height=480, format=RGB ! filesink location=tmp4tmp5.rgb & sleep 3 gst-launch-1.0 videotestsrc num-buffers=3 ! video/x-raw, width=1280, height=720, format=RGB ! v4l2video5convertv4l2video6convert capture-io-mode=4 output-io-mode=4 ! video/x-raw, width=640, height=480, format=RGB ! filesink location=tmp5tmp6.rgb & sleep 3 gst-launch-1.0 videotestsrc num-buffers=3 ! video/x-raw, width=1280, height=720, format=RGB ! v4l2video6convertv4l2video7convert capture-io-mode=4 output-io-mode=4 ! video/x-raw, width=640, height=480, format=RGB ! filesink location=tmp6tmp7.rgb & sleep 3 gst-launch-1.0 videotestsrc num-buffers=310 ! video/x-raw, width=1280, height=720, format=RGB ! v4l2video7convertv4l2convert capture-io-mode=4 output-io-mode=4 ! video/x-raw, width=640, height=480, format=RGB ! filesink location=tmp7tmp0.rgb & sleep 3 |
1-N Scaler
Code Block | ||||
---|---|---|---|---|
| ||||
#Create input file for abrscaler gst-launch-1.0 videotestsrc videotestsrc num-buffers=10 ! video/x-raw, width=1280, height=720, format=RGB ! v4l2video0convert v4l2convert capture-io-mode=4 output-io-mode=4 ! video/x-raw, width=6401920, height=4801080, format=RGB ! filesink location=tmp0.rgb |
1-N Scaler
Code Block | ||||
---|---|---|---|---|
| ||||
#Create input file for abrscaler testsrc_rgb_time_10frames.rgb #Run ABR Test gst-launch-1.0 videotestsrc filesrc location=testsrc_rgb_time_10frames.rgb num-buffers=10 blocksize=6220800 do-timestamp=true ! "video/x-raw, width=12801920, height=7201080, format=RGB ! v4l2video0convert capture-io, framerate=30/1, interlace-mode=4 output-io-mode=4 ! progressive, colorimetry=sRGB" ! xlnxabrscaler device="/dev/video0" name=sc \ sc.src_0 ! "video/x-raw, width=19201600, height=1080900, format=RGB" ! queue ! filesink location=testsrc1600x900_rgb_time_10frames50frames.rgb \ sc.src_1 #Run ABR Test gst-launch-1.0 filesrc location=testsrc_rgb_time_10frames.rgb num-buffers=10 blocksize=6220800 do-timestamp=true ! "video/x-raw, width=1280, height=720, format=RGB" ! queue ! filesink location=1280x720_rgb_50frames.rgb \ sc.src_2 ! "video/x-raw, width=1920800, height=1080600, format=RGB, framerate=30/1, interlace-mode=progressive, colorimetry=sRGB" ! xlnxabrscaler device="/dev/video0" name=sc " ! queue ! filesink location=800x600_rgb_50frames.rgb \ sc.src_03 ! "video/x-raw, width=1600832, height=900480, format=RGB" ! queue ! filesink location=1600x900832x480_rgb_50frames.rgb \ sc.src_14 ! "video/x-raw, width=1280640, height=720480, format=RGB" ! queue ! filesink location=1280x720640x480_rgb_50frames.rgb \ sc.src_25 ! "video/x-raw, width=800480, height=600320, format=RGB" ! queue ! filesink location=800x600480x320_rgb_50frames.rgb \ sc.src_36 ! "video/x-raw, width=832320, height=480240, format=RGB" ! queue ! filesink location=832x480320x240_rgb_50frames.rgb \ sc.src_47 ! "video/x-raw, width=640176, height=480144, format=RGB" ! queue ! filesink location=640x480176x144_rgb_50frames.rgb \ sc.src_5 ! "video/x-raw, width=480, height=320, format=RGB" ! queue ! filesink location=480x320_rgb_50frames.rgb \ sc.src_6 ! "video/x-raw, width=320, height=240, format=RGB" ! queue ! filesink location=320x240_rgb_50frames.rgb \ sc.src_7 ! "video/x-raw, width=176, height=144, format=RGB" ! queue ! filesink location=176x144_rgb_50frames.rgb |
Stride and Height Alignment
Multiple IPs required aligned stride and height to work properly, but till now there is no way to share this information with Multi-Scaler plugin and driver.
Multi-Scaler driver has implemented a mechanism to provide stride and height align values per channel for both output and capture pads, at run-time. This is temporary fix. Once the stride and height alignment support added to plugin, this change will be reverted.
...
Stride and Height Alignment
Multiple IPs required aligned stride and height to work properly, but till now there is no way to share this information with Multi-Scaler plugin and driver.
Multi-Scaler driver has implemented a mechanism to provide stride and height align values per channel for both output and capture pads, at run-time. This is temporary fix. Once the stride and height alignment support added to plugin, this change will be reverted.
Example 1:
Filesrc -> Decoder -> Multi-Scaler -> filesink
As decoder require 256 align byteperline and 64 align lines per frame, so the setting before running gstreamer pipe are :
Code Block | ||||
---|---|---|---|---|
| ||||
echo 256 > /sys/modules/xilinx-multi-scaler/parameters/output_stride_align
echo 64 > /sys/modules/xilinx-multi-scaler/parameters/output_height_align
echo 1 > /sys/modules/xilinx-multi-scaler/parameters/capture_stride_align
echo 1 > /sys/modules/xilinx-multi-scaler/parameters/capture_height_align |
And then run :
Code Block | ||||
---|---|---|---|---|
| ||||
gst-launch-1.0 filesrc location=./1080p_h264.mp4 ! qtdemux ! h264parse ! omxh264dec ! v4l2video1convert capture-io-mode=4 output-io-mode=5 ! video/x-raw, width=1280, height=720, format=BGR ! filesink |
Example 2:
Filesrc -> Decoder -> Multi-Scaler -> filesink
As decoder require 256 align byteperline and 64 align lines per frame, so the setting before running gstreamer pipe are :
...
DP kmssink
Code Block | ||||
---|---|---|---|---|
| ||||
echo 256 > /sys/modules/xilinx-multi-scaler/parameters/output_stride_align echo 64 > /sys/modules/xilinx-multi-scaler/parameters/output_height_align echo 1256 > /sys/modules/xilinx-multi-scaler/parameters/capture_stride_align echo 1 > /sys/modules/xilinx-multi-scaler/parameters/capture_height_align |
And then the run:
Code Block | ||||
---|---|---|---|---|
| ||||
gst-launch-1.0 filesrc location=./1080p_h264.mp4 ! qtdemux ! h264parse ! omxh264dec ! v4l2video1convert capture-io-mode=4 output-io-mode=5 ! video/x-raw, width=1280, height=720, format=BGR ! queue max-size-bytes=0 ! filesink |
Example 2:
...
kmssink bus-id="fd4a0000.zynqmp-display" fullscreen-overlay=1 sync=false |
Example 3:
videotestsrc -> Multi-Scaler -> DP kmssinkfilesink
Code Block | ||||
---|---|---|---|---|
| ||||
echo 2561 > /sys/modules/xilinx-multi-scaler/parameters/output_stride_align echo 641 > /sys/modules/xilinx-multi-scaler/parameters/output_height_align echo 2561 > /sys/modules/xilinx-multi-scaler/parameters/capture_stride_align echo 1 > /sys/modules/xilinx-multi-scaler/parameters/capture_height_align |
...
Example 4:
videotestsrc -> Multi-Scaler0 -> filesink
Filesrc -> Decoder -> Multi-Scaler1 -> DP kmssink
Code Block | ||||
---|---|---|---|---|
| ||||
gst-launch-1.0 filesrc location=./1080p_h264.mp4 ! qtdemux ! h264parse ! omxh264dec ! v4l2video1convert capture-io-mode=4 output-io-mode=5 ! video/x-raw, width=1280, height=720, format=BGR ! queue max-size-bytes=0 ! kmssink bus-id="fd4a0000.zynqmp-display" fullscreen-overlay=1 sync=false |
Example 3:
videotestsrc -> Multi-Scaler -> filesink
Code Block | ||||
---|---|---|---|---|
| ||||
echo 1 > /sys/modules/xilinx-multi-scaler/parameters/output_stride_align
echo 1 > /sys/modules/xilinx-multi-scaler/parameters/output_height_align
echo 1 > /sys/modules/xilinx-multi-scaler/parameters/capture_stride_align
echo 1 > /sys/modules/xilinx-multi-scaler/parameters/capture_height_align |
Example 4:
videotestsrc -> Multi-Scaler0 -> filesink
Filesrc -> Decoder -> Multi-Scaler1 -> DP kmssink
Code Block | ||||
---|---|---|---|---|
| ||||
echo 1,256 > /sys/modules/xilinx-multi-scaler/parameters/output_stride_align
echo 1,64 > /sys/modules/xilinx-multi-scaler/parameters/output_height_align
echo 1,256 > /sys/modules/xilinx-multi-scaler/parameters/capture_stride_align
echo 1,1 > /sys/modules/xilinx-multi-scaler/parameters/capture_height_align |
Notes
Gstreamer plugins enumerate all the channel/video devices for each of gst command. So, there might be chances that starting all channels with different gst process at same time do not allow the actual process to open the device and the process dedicated for the device does not start. For example, if all channels are opened immediately in a script, all the gst process open and close all the devices for all the channels and if somehow gst process 1 opens channel 2 and at same time if process 2 open channel 2, the 2nd process do not get the channel 2 and returns, despite the process 1 do not require the channel 2 and close the channel 2 afterwards. Adding sleep after every enumeration is required to make all applications work properly.
...
echo 1,256 > /sys/modules/xilinx-multi-scaler/parameters/output_stride_align
echo 1,64 > /sys/modules/xilinx-multi-scaler/parameters/output_height_align
echo 1,256 > /sys/modules/xilinx-multi-scaler/parameters/capture_stride_align
echo 1,1 > /sys/modules/xilinx-multi-scaler/parameters/capture_height_align |
Notes
Gstreamer plugins enumerate all the channel/video devices for each of gst command. So, there might be chances that starting all channels with different gst process at same time do not allow the actual process to open the device and the process dedicated for the device does not start. For example, if all channels are opened immediately in a script, all the gst process open and close all the devices for all the channels and if somehow gst process 1 opens channel 2 and at same time if process 2 open channel 2, the 2nd process do not get the channel 2 and returns, despite the process 1 do not require the channel 2 and close the channel 2 afterwards. Adding sleep after every enumeration is required to make all applications work properly.
- The gstreamer plugins mentioned on this page are developed for Proof of Concept and testing some limited featured only. This plugins are not tested for all possible resolutions, video formats etc. Customers may need to develop their own plugins by using these plugins as example.
Change Log
2023.2
- Summary
- Removed old documentation about gst-inspect.
2023.1
- Summary
- Fix corruption for YUV422 8 and 10 bpc
- Commits
2022.2
- No changes
2022.1
- Summary
- Fix warning for unchecked return value
- Commits
2021.2
- Summary
- Change print level from error to debug
- Commits
2021.1
- No changes
2020.2
- Summary
- Add streaming capabilities
- Derive caps from video node
- Fix Coverity warnings
- Commits
- v4l: xilinx: multi-scaler: Added streaming capabilities to device
- v4l: xilinx: multi-scaler: Derive the caps from video device node
- v4l: xilinx: multi-scaler: Fix warnings for NULL_RETURNS
- v4l: xilinx: multi-scaler: Fix warnings for INCOMPATIBLE_PARAM
- v4l: xilinx: multi-scaler: Fix warnings for MIXED_ENUM_TYPE
- v4l: xilinx: multi-scaler: Fix warnings for UNUSED_VALUE
2020.1
- Summary
- Minor fixes
- Commits
2019.2
- Summary
- No changes
2019.1
- Summary:
...
2018.3
- Summary:
- Add first version of driver.
- Commits:
- 484cdb2 platform: xilinx: Add mem to mem Multi-Scaler driver (XM2MSC)