This page gives an overview of Xilinx Multi-Scaler driver, which is available as part of the Xilinx Linux distribution.
The driver is part of the V4L2 Memory to Memory framework.
The Multi-Scaler driver generates up to eight scaled output images from a single or multiple (up to eight) external video and/or graphics sources. Video scaling is the process of converting an input color image of dimensions Xin pixels by Yin lines to an output color image of dimensions Xout pixels by Yout lines. This IP also capable of doing color space conversion. Multi-Scaler IP reads the input images from the memory addresses and writes the scaled output images to the destination memory addresses.
The Multi-scaler works in a sequential way, i.e. the input of the first channel scaled first and then the 2nd input gets scaled and so no up to the maximum channels configured in HwReg_num_outs. When all channels (configured) get scaled, IP generates the interrupt and ready for next processing.
Multi-Scaler converts and scales video frames between a great variety of video formats. For more supported formats please visit IP Product guide.
Xilinx-multiscaler creates a device node per channel and can be accessed from user-space via standard V4L2 system calls. Even though the devices created by driver looks like independent nodes but as multi-scaler IP has one scaler inside and works in sequential manner these nodes/devices are not independent. For more details see limitation section.
Example: Channels 0,1,2,3 are streaming, then the user can stream on immediate next channel 4.
If user adds stream for channel 6 when channels 0,1,2,3 are running, then driver does not start 6th channel and does not return any error.
If any intermediate channel stops the streaming, the streaming continues for only channels before the stopped channel.
In this case if channel 0,1,2,3,4 are running and channel 3 stops, then the streaming for channels 0,1,2 will continue and driver stop processing 4th channel without any error indications.
A channel is not processed by the driver, if all channel with lower channel number are not queueing the buffers.
E.g. If 0,1,2,3 channels were being processed by driver and channel 1 didn't queue any buffers, then channel 2 and 3 won't be processed but channel 0 will continue.
The following config options should be enabled in order to build Xilinx Multi-Scaler driver
CONFIG_VIDEO_XILINX
CONFIG_VIDEO_DEV
CONFIG_VIDEO_V4L2
CONFIG_VIDEO_XILINX_MULTISCALER
The driver is available at,
https://github.com/Xilinx/linux-xlnx/blob/master/drivers/media/platform/xilinx/xilinx-multi-scaler.c
The device tree node will be automatically generated, if the core is configured in the HW design, using the Device Tree BSP.
Steps to generate device-tree is documented here,
http://www.wiki.xilinx.com/Build+Device+Tree+Blob
And a sample binding is shown below and the description of DT property is documented here
v_multi_scaler_1: v_multi_scaler@a0000000 { clock-names = "ap_clk"; clocks = <&misc_clk_0>; compatible = "xlnx,v-multi-scaler-1.0", "xlnx,v-multi-scaler-v1.0"; interrupt-names = "interrupt"; interrupt-parent = <&gic>; interrupts = <0 89 4>; reg = <0x0 0xa0000000 0x0 0x20000>; reset-gpios = <&gpio 78 1>; xlnx,dma-addr-width = <0x20>; xlnx,max-chan = <8>; xlnx,max-height = <2160>; xlnx,max-width = <3840>; xlnx,num-taps = <12>; xlnx,pixels-per-clock = /bits/ 8 <2>; xlnx,vid-formats = "xrgb8888", "bgr888", "xbgr8888", "xbgr2101010", "uyvy", "y8", "y10", "vuy888", "xvuy8888", "yuvx2101010", "yuyv", "nv12", "nv16", "xv20", "xv15"; }; |
A reference design for testing is as below
Multi-Scaler driver creates a video device per channel, which can be visual in /dev directory.
The device names are /dev/videoN to /dev/videoN+8 where N starts from the next available video device to the maximum number of devices the IP is configured (xlnx, max-chan in DT).
Example:
# ls /dev/ | grep video video0 video1 video2 video3 video4 video5 video6 video7 |
Multi-Scaler is tested with the generic gstreamer v4l2videotransform plugin which maps device node to v4l2videoNconvert elements. Where N is the number of device node.
Below is the steps to get the proper gstreamer element for the Multi-Scaler.
Gstreamer device name for Multi-Scaler is xm2msc (Xilinx memory to memory video multi-scaler).
In below example the IP is configured for 8 channels.
# gst-inspect-1.0 -a | grep xm2msc v4l2video0convert: String. Default: "xm2msc" v4l2video1convert: String. Default: "xm2msc" v4l2video2convert: String. Default: "xm2msc" v4l2video3convert: String. Default: "xm2msc" v4l2video4convert: String. Default: "xm2msc" v4l2video5convert: String. Default: "xm2msc" v4l2video6convert: String. Default: "xm2msc" v4l2video7convert: String. Default: "xm2msc" |
Multi-Scaler is tested with the generic gstreamer v4l2videotransform elements which is part of gst-plugins-good. So no extra library required to run N:N use case.
In below image, it is represented a basic test scenario where multiple v4l2videoNconvert elements are launched as different process opening different node of Multi-Scale. The input for the v4l2videoNconvert is frames of different format and different resolutions and the output is given to filesink which stores the scaled and converted frames in a file.
Below command is used to test Multi-Scaler with Gstreamer. You can open Multiple channels by running v4l2videoNconvert process in parallel.
gst-launch-1.0 videotestsrc num-buffers=10 ! video/x-raw, width=1280, height=720, format=RGB ! v4l2video0convert capture-io-mode=4 output-io-mode=4 ! video/x-raw, width=640, height=480, format=RGB ! filesink location=tmp0.rgb |
Multi-Scaler's 1 to N use case is tested with the xlnxabrscaler plugin.
In below image, it is represented a basic test scenario where single input given to xlnxabrscaler elements which opens different node of Multi-Scaler. The output is given to filesink which stores the scaled and converted frames in files.
For 1 to N Use Case, run commands mention in debugging section.
Replace the format string in above command with below Gstreamer strings, to test different supported formats
ID | Multi-Scaler Video Format
| Device-tree strings | V4L2 Format
| Gstreamer String | YUV Player Format |
---|---|---|---|---|---|
10 | RGBX8 | xbgr8888 | BGRX32 | ||
11 | YUVX8 | xvuy8888 | XVUY32 | ||
12 | YUYV8 | yuyv | YUYV | YUY2 | YUYV |
15 | RGBX10 | xbgr2101010 | XBGR30 | ||
16 | YUVX10 | yuvx2101010 | XVUY10 | ||
18 | Y_UV8 | nv16 | NV16M | NV16 | NV16 |
19 | Y_UV8_420 | nv12 | NV12M | NV12 | NV12 |
20 | RGB8 | bgr888 | RGB24 | RGB | RGB24 |
21 | YUV8 | vuy888 | VUY24 | ||
22 | Y_UV10 | xv20 | XV20M | XV20 | |
23 | Y_UV10_420 | xv15 | XV15M | XV15 | |
24 | Y8 | y8 | GREY | GRAY8 | Y |
25 | Y10 | y10 | Y10 | Y10 | |
27 | BGRX8 | xrgb8888 | XBGR32 | BGRx | BGR32 |
28 | UYVY8 | uyvy | UYVY | UYVY | UYVY |
29 | BGR8 | rgb888 | BGR24 | BGR | BGR24 |
Enable Dynamic Debugging in kernel.
Run below commands to enable logs on console
echo 1 > /sys/module/v4l2_mem2mem/parameters/debug echo 8 > /sys/module/videobuf2_core/parameters/debug echo 8 > /sys/module/videobuf2_v4l2/parameters/debug echo 8 > /proc/sys/kernel/printk |
gst-launch-1.0 videotestsrc num-buffers=10 ! video/x-raw, width=1280, height=720, format=RGB ! v4l2video0convert capture-io-mode=4 output-io-mode=4 ! video/x-raw, width=640, height=480, format=RGB ! filesink location=640X480.rgb |
#!/bin/bash gst-launch-1.0 videotestsrc num-buffers=3 ! video/x-raw, width=1280, height=720, format=RGB ! v4l2video1convert capture-io-mode=4 output-io-mode=4 ! video/x-raw, width=640, height=480, format=RGB ! filesink location=tmp1.rgb & sleep 3 gst-launch-1.0 videotestsrc num-buffers=3 ! video/x-raw, width=1280, height=720, format=RGB ! v4l2video2convert capture-io-mode=4 output-io-mode=4 ! video/x-raw, width=640, height=480, format=RGB ! filesink location=tmp2.rgb & sleep 3 gst-launch-1.0 videotestsrc num-buffers=3 ! video/x-raw, width=1280, height=720, format=RGB ! v4l2video3convert capture-io-mode=4 output-io-mode=4 ! video/x-raw, width=640, height=480, format=RGB ! filesink location=tmp3.rgb & sleep 3 gst-launch-1.0 videotestsrc num-buffers=3 ! video/x-raw, width=1280, height=720, format=RGB ! v4l2video4convert capture-io-mode=4 output-io-mode=4 ! video/x-raw, width=640, height=480, format=RGB ! filesink location=tmp4.rgb & sleep 3 gst-launch-1.0 videotestsrc num-buffers=3 ! video/x-raw, width=1280, height=720, format=RGB ! v4l2video5convert capture-io-mode=4 output-io-mode=4 ! video/x-raw, width=640, height=480, format=RGB ! filesink location=tmp5.rgb & sleep 3 gst-launch-1.0 videotestsrc num-buffers=3 ! video/x-raw, width=1280, height=720, format=RGB ! v4l2video6convert capture-io-mode=4 output-io-mode=4 ! video/x-raw, width=640, height=480, format=RGB ! filesink location=tmp6.rgb & sleep 3 gst-launch-1.0 videotestsrc num-buffers=3 ! video/x-raw, width=1280, height=720, format=RGB ! v4l2video7convert capture-io-mode=4 output-io-mode=4 ! video/x-raw, width=640, height=480, format=RGB ! filesink location=tmp7.rgb & sleep 3 gst-launch-1.0 videotestsrc num-buffers=10 ! video/x-raw, width=1280, height=720, format=RGB ! v4l2video0convert capture-io-mode=4 output-io-mode=4 ! video/x-raw, width=640, height=480, format=RGB ! filesink location=tmp0.rgb |
#Create input file for abrscaler gst-launch-1.0 videotestsrc num-buffers=10 ! video/x-raw, width=1280, height=720, format=RGB ! v4l2video0convert capture-io-mode=4 output-io-mode=4 ! video/x-raw, width=1920, height=1080, format=RGB ! filesink location=testsrc_rgb_time_10frames.rgb #Run ABR Test gst-launch-1.0 filesrc location=testsrc_rgb_time_10frames.rgb num-buffers=10 blocksize=6220800 do-timestamp=true ! "video/x-raw, width=1920, height=1080, format=RGB, framerate=30/1, interlace-mode=progressive, colorimetry=sRGB" ! xlnxabrscaler device="/dev/video0" name=sc \ sc.src_0 ! "video/x-raw, width=1600, height=900, format=RGB" ! queue ! filesink location=1600x900_rgb_50frames.rgb \ sc.src_1 ! "video/x-raw, width=1280, height=720, format=RGB" ! queue ! filesink location=1280x720_rgb_50frames.rgb \ sc.src_2 ! "video/x-raw, width=800, height=600, format=RGB" ! queue ! filesink location=800x600_rgb_50frames.rgb \ sc.src_3 ! "video/x-raw, width=832, height=480, format=RGB" ! queue ! filesink location=832x480_rgb_50frames.rgb \ sc.src_4 ! "video/x-raw, width=640, height=480, format=RGB" ! queue ! filesink location=640x480_rgb_50frames.rgb \ sc.src_5 ! "video/x-raw, width=480, height=320, format=RGB" ! queue ! filesink location=480x320_rgb_50frames.rgb \ sc.src_6 ! "video/x-raw, width=320, height=240, format=RGB" ! queue ! filesink location=320x240_rgb_50frames.rgb \ sc.src_7 ! "video/x-raw, width=176, height=144, format=RGB" ! queue ! filesink location=176x144_rgb_50frames.rgb |
Multiple IPs required aligned stride and height to work properly, but till now there is no way to share this information with Multi-Scaler plugin and driver.
Multi-Scaler driver has implemented a mechanism to provide stride and height align values per channel for both output and capture pads, at run-time. This is temporary fix. Once the stride and height alignment support added to plugin, this change will be reverted.
Filesrc -> Decoder -> Multi-Scaler -> filesink
As decoder require 256 align byteperline and 64 align lines per frame, so the setting before running gstreamer pipe are :
echo 256 > /sys/modules/xilinx-multi-scaler/parameters/output_stride_align echo 64 > /sys/modules/xilinx-multi-scaler/parameters/output_height_align echo 1 > /sys/modules/xilinx-multi-scaler/parameters/capture_stride_align echo 1 > /sys/modules/xilinx-multi-scaler/parameters/capture_height_align |
And then run :
gst-launch-1.0 filesrc location=./1080p_h264.mp4 ! qtdemux ! h264parse ! omxh264dec ! v4l2video1convert capture-io-mode=4 output-io-mode=5 ! video/x-raw, width=1280, height=720, format=BGR ! filesink |
Filesrc -> Decoder -> Multi-Scaler -> DP kmssink
echo 256 > /sys/modules/xilinx-multi-scaler/parameters/output_stride_align echo 64 > /sys/modules/xilinx-multi-scaler/parameters/output_height_align echo 256 > /sys/modules/xilinx-multi-scaler/parameters/capture_stride_align echo 1 > /sys/modules/xilinx-multi-scaler/parameters/capture_height_align |
And the run:
gst-launch-1.0 filesrc location=./1080p_h264.mp4 ! qtdemux ! h264parse ! omxh264dec ! v4l2video1convert capture-io-mode=4 output-io-mode=5 ! video/x-raw, width=1280, height=720, format=BGR ! queue max-size-bytes=0 ! kmssink bus-id="fd4a0000.zynqmp-display" fullscreen-overlay=1 sync=false |
videotestsrc -> Multi-Scaler -> filesink
echo 1 > /sys/modules/xilinx-multi-scaler/parameters/output_stride_align echo 1 > /sys/modules/xilinx-multi-scaler/parameters/output_height_align echo 1 > /sys/modules/xilinx-multi-scaler/parameters/capture_stride_align echo 1 > /sys/modules/xilinx-multi-scaler/parameters/capture_height_align |
videotestsrc -> Multi-Scaler0 -> filesink
Filesrc -> Decoder -> Multi-Scaler1 -> DP kmssink
echo 1,256 > /sys/modules/xilinx-multi-scaler/parameters/output_stride_align echo 1,64 > /sys/modules/xilinx-multi-scaler/parameters/output_height_align echo 1,256 > /sys/modules/xilinx-multi-scaler/parameters/capture_stride_align echo 1,1 > /sys/modules/xilinx-multi-scaler/parameters/capture_height_align |
Gstreamer plugins enumerate all the channel/video devices for each of gst command. So, there might be chances that starting all channels with different gst process at same time do not allow the actual process to open the device and the process dedicated for the device does not start. For example, if all channels are opened immediately in a script, all the gst process open and close all the devices for all the channels and if somehow gst process 1 opens channel 2 and at same time if process 2 open channel 2, the 2nd process do not get the channel 2 and returns, despite the process 1 do not require the channel 2 and close the channel 2 afterwards. Adding sleep after every enumeration is required to make all applications work properly.