1 Introduction
2 HW IP features
3 Missing Features / Known Issues / Limitations in Driver
4 Kernel Configuration
5 Device Tree Binding
6 Test Procedure
7 Notes
8 Change Log
9 Related Links

Introduction

This page gives an overview of Xilinx Multi-Scaler driver, which is available as part of the Xilinx Linux distribution.

The driver is part of the V4L2 Memory to Memory framework.

The Multi-Scaler driver generates up to eight scaled output images from a single or multiple (up to eight) external video and/or graphics sources. Video scaling is the process of converting an input color image of dimensions Xin pixels by Yin lines to an output color image of dimensions Xout pixels by Yout lines. This IP also capable of doing color space conversion. Multi-Scaler IP reads the input images from the memory addresses and writes the scaled output images to the destination memory addresses.

The Multi-scaler works in a sequential way, i.e. the input of the first channel scaled first and then the 2^nd input gets scaled and so no up to the maximum channels configured in HwReg_num_outs. When all channels (configured) get scaled, IP generates the interrupt and ready for next processing.

Multi-Scaler converts and scales video frames between a great variety of video formats. For more supported formats please visit IP Product guide.

Xilinx-multiscaler creates a device node per channel and can be accessed from user-space via standard V4L2 system calls. Even though the devices created by driver looks like independent nodes but as multi-scaler IP has one scaler inside and works in sequential manner these nodes/devices are not independent. For more details see limitation section.

HW IP features

Memory mapped AXI4 Interface
Supports maximum 8 outputs
Supports spatial resolutions from 64 × 64 up to 7680 × 4320
Supports one, two, or four pixel-width
Supports RGB, YUV 444, YUV 422, YUV 420
Supports 8-bit and 10-bit per color component on memory interface
Supports semi-planar memory formats next to packed memory formats
Dynamically configurable source and destination buffer addresses
Supports 6, 8, 10, and 12 taps in both H and V domains
Supports 64 phases
Supports 8K 30 fps depending on the device family

Missing Features / Known Issues / Limitations in Driver

Tested 64 × 64 up to 1920 × 1080.
10-bit per color component is not tested.
As hardware processes all channels in sequential order, the driver has limitation to add only the next immediate channel.

Example: Channels 0,1,2,3 are streaming, then the user can stream on immediate next channel 4.

If user adds stream for channel 6 when channels 0,1,2,3 are running, then driver does not start 6th channel and does not return any error.

If any intermediate channel stops the streaming, the streaming continues for only channels before the stopped channel.

In this case if channel 0,1,2,3,4 are running and channel 3 stops, then the streaming for channels 0,1,2 will continue and driver stop processing 4th channel without any error indications.

A channel is not processed by the driver, if all channel with lower channel number are not queueing the buffers.

E.g. If 0,1,2,3 channels were being processed by driver and channel 1 didn't queue any buffers, then channel 2 and 3 won't be processed but channel 0 will continue.

The driver is tested with YUY2, NV16, NV12, RGB, GRAY8, UYVY and BGR Gstreamer formats only.
Please check section below on Stride and Height align.

Kernel Configuration

The following config options should be enabled in order to build Xilinx Multi-Scaler driver

CONFIG_VIDEO_XILINX

CONFIG_VIDEO_DEV

CONFIG_VIDEO_V4L2

CONFIG_VIDEO_XILINX_MULTISCALER

To check if the options are enabled run the following:

petalinux-config -c kernel

When the GUI opens, you can use '/' to search for each config.

The driver is available at,
https://github.com/Xilinx/linux-xlnx/blob/master/drivers/media/platform/xilinx/xilinx-multi-scaler.c

Device Tree Binding

The device tree node will be automatically generated, if the core is configured in the HW design, using the Device Tree BSP.

Steps to generate device-tree is documented here,
http://www.wiki.xilinx.com/Build+Device+Tree+Blob

And a sample binding is shown below and the description of DT property is documented here

v_multi_scaler_1: v_multi_scaler@a0000000 {
          clock-names = "ap_clk";
          clocks = <&misc_clk_0>;
          compatible = "xlnx,v-multi-scaler-1.0", "xlnx,v-multi-scaler-v1.0";
          interrupt-names = "interrupt";
          interrupt-parent = <&gic>;
          interrupts = <0 89 4>;
          reg = <0x0 0xa0000000 0x0 0x20000>;
          reset-gpios = <&gpio 78 1>;
          xlnx,dma-addr-width = <0x20>;
          xlnx,max-chan = <8>;
          xlnx,max-height = <2160>;
          xlnx,max-width = <3840>;
          xlnx,num-taps = <12>;
          xlnx,pixels-per-clock = /bits/ 8 <2>;
          xlnx,vid-formats = "xrgb8888", "bgr888", "xbgr8888", "xbgr2101010", "uyvy", "y8", "y10", "vuy888", "xvuy8888", "yuvx2101010", "yuyv", "nv12", "nv16", "xv20", "xv15";
};

Test Procedure

A reference design for testing is as below

Multi-Scaler driver creates a video device per channel, which can be visual in /dev directory.

The device names are /dev/videoN to /dev/videoN+8 where N starts from the next available video device to the maximum number of devices the IP is configured (xlnx, max-chan in DT).

Example:

# ls /dev/ | grep video
video0
video1
video2
video3
video4
video5
video6
video7

Multi-Scaler is tested with the generic gstreamer v4l2videotransform plugin which maps device node to v4l2videoNconvert elements. Where N is the number of device node.

N-N Use Case:

Multi-Scaler is tested with the generic gstreamer v4l2videotransform elements which is part of gst-plugins-good. So no extra library required to run N:N use case.

In below image, it is represented a basic test scenario where multiple v4l2videoNconvert elements are launched as different process opening different node of Multi-Scale. The input for the v4l2videoNconvert is frames of different format and different resolutions and the output is given to filesink which stores the scaled and converted frames in a file.

Commands to test

Below command is an example using Gstreamer. You can open Multiple channels by running v4l2videoNconvert process in parallel.
**Note Gstreamer plugins may change, please refer to Gstreamer documentation for the latest information on using Gstreamer plugins

gst-launch-1.0 videotestsrc num-buffers=10 ! video/x-raw, width=1280, height=720, format=RGB ! v4l2convert capture-io-mode=4 output-io-mode=4 ! video/x-raw, width=640, height=480, format=RGB ! filesink location=tmp0.rgb

1 to N Use Case:

Multi-Scaler's 1 to N use case is tested with the xlnxabrscaler plugin.

In below image, it is represented a basic test scenario where single input given to xlnxabrscaler elements which opens different node of Multi-Scaler. The output is given to filesink which stores the scaled and converted frames in files.

Commands to test

For 1 to N Use Case, run commands mention in debugging section.

Mapping table between Gstreamer string and Xilinx Multi-Scaler Supported Formats

Replace the format string in above command with below Gstreamer strings, to test different supported formats

ID	Multi-Scaler Video Format `Prefix with XILINX_M2MSC_FMT_`	Device-tree strings	V4L2 Format `Prefix with` `V4L2_PIX_FMT_`	Gstreamer String	YUV Player Format

ID	Multi-Scaler Video Format `Prefix with XILINX_M2MSC_FMT_`	Device-tree strings	V4L2 Format `Prefix with` `V4L2_PIX_FMT_`	Gstreamer String	YUV Player Format
10	RGBX8	xbgr8888	BGRX32
11	YUVX8	xvuy8888	XVUY32
12	YUYV8	yuyv	YUYV	YUY2	YUYV
15	RGBX10	xbgr2101010	XBGR30
16	YUVX10	yuvx2101010	XVUY10
18	Y_UV8	nv16	NV16M	NV16	NV16
19	Y_UV8_420	nv12	NV12M	NV12	NV12
20	RGB8	bgr888	RGB24	RGB	RGB24
21	YUV8	vuy888	VUY24
22	Y_UV10	xv20	XV20M		XV20
23	Y_UV10_420	xv15	XV15M		XV15
24	Y8	y8	GREY	GRAY8	Y
25	Y10	y10	Y10		Y10
27	BGRX8	xrgb8888	XBGR32	BGRx	BGR32
28	UYVY8	uyvy	UYVY	UYVY	UYVY
29	BGR8	rgb888	BGR24	BGR	BGR24

Debugging

Enable Dynamic Debugging in kernel.

Run below commands to enable logs on console

echo 1 > /sys/module/v4l2_mem2mem/parameters/debug
echo 8 > /sys/module/videobuf2_core/parameters/debug
echo 8 > /sys/module/videobuf2_v4l2/parameters/debug
echo 8 > /proc/sys/kernel/printk

Use Cases

1-1 Scaler

gst-launch-1.0 videotestsrc num-buffers=10 ! video/x-raw, width=1280, height=720, format=RGB ! v4l2convert capture-io-mode=4 output-io-mode=4 ! video/x-raw, width=640, height=480, format=RGB ! filesink location=640X480.rgb

N-N Scaler

#!/bin/bash
gst-launch-1.0 videotestsrc num-buffers=3 ! video/x-raw, width=1280, height=720, format=RGB ! v4l2video1convert capture-io-mode=4 output-io-mode=4 ! video/x-raw, width=640, height=480, format=RGB ! filesink location=tmp1.rgb &

sleep 3

gst-launch-1.0 videotestsrc num-buffers=3 ! video/x-raw, width=1280, height=720, format=RGB ! v4l2video2convert capture-io-mode=4 output-io-mode=4 ! video/x-raw, width=640, height=480, format=RGB ! filesink location=tmp2.rgb &

sleep 3

gst-launch-1.0 videotestsrc num-buffers=3 ! video/x-raw, width=1280, height=720, format=RGB ! v4l2video3convert capture-io-mode=4 output-io-mode=4 ! video/x-raw, 
width=640, height=480, format=RGB ! filesink location=tmp3.rgb &

sleep 3

gst-launch-1.0 videotestsrc num-buffers=3 ! video/x-raw, width=1280, height=720, format=RGB ! v4l2video4convert capture-io-mode=4 output-io-mode=4 ! video/x-raw, width=640, height=480, format=RGB ! filesink location=tmp4.rgb &

sleep 3

gst-launch-1.0 videotestsrc num-buffers=3 ! video/x-raw, width=1280, height=720, format=RGB ! v4l2video5convert capture-io-mode=4 output-io-mode=4 ! video/x-raw, 
width=640, height=480, format=RGB ! filesink location=tmp5.rgb &

sleep 3

gst-launch-1.0 videotestsrc num-buffers=3 ! video/x-raw, width=1280, height=720, format=RGB ! v4l2video6convert capture-io-mode=4 output-io-mode=4 ! video/x-raw, width=640, height=480, format=RGB ! filesink location=tmp6.rgb &

sleep 3

gst-launch-1.0 videotestsrc num-buffers=3 ! video/x-raw, width=1280, height=720, format=RGB ! v4l2video7convert capture-io-mode=4 output-io-mode=4 ! video/x-raw, 
width=640, height=480, format=RGB ! filesink location=tmp7.rgb &

sleep 3

gst-launch-1.0  videotestsrc num-buffers=10 ! video/x-raw, width=1280, height=720, format=RGB ! v4l2convert capture-io-mode=4 output-io-mode=4 ! video/x-raw, width=640, height=480, format=RGB ! filesink location=tmp0.rgb

1-N Scaler

#Create input file for abrscaler
gst-launch-1.0 videotestsrc 
num-buffers=10 ! video/x-raw, width=1280, height=720, format=RGB ! 
v4l2convert capture-io-mode=4 output-io-mode=4 ! video/x-raw, 
width=1920, height=1080, format=RGB ! filesink 
location=testsrc_rgb_time_10frames.rgb


#Run ABR Test
gst-launch-1.0 filesrc location=testsrc_rgb_time_10frames.rgb num-buffers=10 blocksize=6220800 do-timestamp=true ! "video/x-raw, width=1920, height=1080, format=RGB, framerate=30/1, interlace-mode=progressive, colorimetry=sRGB" ! xlnxabrscaler device="/dev/video0" name=sc \
sc.src_0 ! "video/x-raw, width=1600, height=900, format=RGB" ! queue ! filesink location=1600x900_rgb_50frames.rgb \
sc.src_1 ! "video/x-raw, width=1280, height=720, format=RGB" ! queue ! filesink location=1280x720_rgb_50frames.rgb \
sc.src_2 ! "video/x-raw, width=800, height=600, format=RGB" ! queue ! filesink location=800x600_rgb_50frames.rgb \
sc.src_3 ! "video/x-raw, width=832, height=480, format=RGB" ! queue ! filesink location=832x480_rgb_50frames.rgb \
sc.src_4 ! "video/x-raw, width=640, height=480, format=RGB" ! queue ! filesink location=640x480_rgb_50frames.rgb \
sc.src_5 ! "video/x-raw, width=480, height=320, format=RGB" ! queue ! filesink location=480x320_rgb_50frames.rgb \
sc.src_6 ! "video/x-raw, width=320, height=240, format=RGB" ! queue ! filesink location=320x240_rgb_50frames.rgb \
sc.src_7 ! "video/x-raw, width=176, height=144, format=RGB" ! queue ! filesink location=176x144_rgb_50frames.rgb

Stride and Height Alignment

Multiple IPs required aligned stride and height to work properly, but till now there is no way to share this information with Multi-Scaler plugin and driver.

Multi-Scaler driver has implemented a mechanism to provide stride and height align values per channel for both output and capture pads, at run-time. This is temporary fix. Once the stride and height alignment support added to plugin, this change will be reverted.

Example 1:

Filesrc -> Decoder -> Multi-Scaler -> filesink

As decoder require 256 align byteperline and 64 align lines per frame, so the setting before running gstreamer pipe are :

echo 256 > /sys/modules/xilinx-multi-scaler/parameters/output_stride_align
echo 64 > /sys/modules/xilinx-multi-scaler/parameters/output_height_align
echo 1 > /sys/modules/xilinx-multi-scaler/parameters/capture_stride_align
echo 1 > /sys/modules/xilinx-multi-scaler/parameters/capture_height_align

And then run :

gst-launch-1.0  filesrc location=./1080p_h264.mp4 ! qtdemux ! h264parse ! omxh264dec ! v4l2video1convert capture-io-mode=4 output-io-mode=5 ! video/x-raw, width=1280, height=720, format=BGR ! filesink

Example 2:

Filesrc -> Decoder -> Multi-Scaler -> DP kmssink

echo 256 > /sys/modules/xilinx-multi-scaler/parameters/output_stride_align
echo 64 > /sys/modules/xilinx-multi-scaler/parameters/output_height_align
echo 256 > /sys/modules/xilinx-multi-scaler/parameters/capture_stride_align
echo 1 > /sys/modules/xilinx-multi-scaler/parameters/capture_height_align

And the run:

gst-launch-1.0  filesrc location=./1080p_h264.mp4 ! qtdemux ! h264parse ! omxh264dec ! v4l2video1convert capture-io-mode=4 output-io-mode=5 ! video/x-raw, width=1280, height=720, format=BGR ! queue max-size-bytes=0 ! kmssink bus-id="fd4a0000.zynqmp-display" fullscreen-overlay=1 sync=false

Example 3:

videotestsrc -> Multi-Scaler -> filesink

echo 1 > /sys/modules/xilinx-multi-scaler/parameters/output_stride_align
echo 1 > /sys/modules/xilinx-multi-scaler/parameters/output_height_align
echo 1 > /sys/modules/xilinx-multi-scaler/parameters/capture_stride_align
echo 1 > /sys/modules/xilinx-multi-scaler/parameters/capture_height_align

Example 4:

videotestsrc -> Multi-Scaler0 -> filesink

Filesrc -> Decoder -> Multi-Scaler1 -> DP kmssink

echo 1,256 > /sys/modules/xilinx-multi-scaler/parameters/output_stride_align
echo 1,64 > /sys/modules/xilinx-multi-scaler/parameters/output_height_align
echo 1,256 > /sys/modules/xilinx-multi-scaler/parameters/capture_stride_align
echo 1,1 > /sys/modules/xilinx-multi-scaler/parameters/capture_height_align

Notes

Gstreamer plugins enumerate all the channel/video devices for each of gst command. So, there might be chances that starting all channels with different gst process at same time do not allow the actual process to open the device and the process dedicated for the device does not start. For example, if all channels are opened immediately in a script, all the gst process open and close all the devices for all the channels and if somehow gst process 1 opens channel 2 and at same time if process 2 open channel 2, the 2^nd process do not get the channel 2 and returns, despite the process 1 do not require the channel 2 and close the channel 2 afterwards. Adding sleep after every enumeration is required to make all applications work properly.
The gstreamer plugins mentioned on this page are developed for Proof of Concept and testing some limited featured only. This plugins are not tested for all possible resolutions, video formats etc. Customers may need to develop their own plugins by using these plugins as example.

Change Log

2025.1

No change

2024.2

No change

2024.1

No change

2023.2

Summary
- Removed old documentation about gst-inspect.

2023.1

Summary
- Fix corruption for YUV422 8 and 10 bpc
Commits
- v4l: xilinx: multi-scaler:fix corruption for YUV422 8 and 10 bpc

2022.2

No changes

2022.1

Summary
- Fix warning for unchecked return value
Commits
- v4l: xilinx: multi-scaler: Fix warnings for unchecked return value

2021.2

Summary
- Change print level from error to debug
Commits
- v4l: xilinx: multi-scaler: change print level from error to debug

2021.1

No changes

2020.2

Summary
- Add streaming capabilities
- Derive caps from video node
- Fix Coverity warnings
Commits

2020.1

Summary
- Minor fixes
Commits
- v4l: xilinx: multi-scaler: fix a printk format specifier
- v4l: xilinx: multi-scaler: suppress info prints in vidioc_try_fmt API

2019.2

Summary
- No changes

2019.1

Summary:

- Support VB2_USERPTR IO mode
- Added support for Pixels per clock
- Add support for formats having contiguous buffers for two planes. (NV12, NV16, XV20, XV15)
- Clock framework support
- Hack support for stride and height alignment required for VCU decoder
Commits:
- 144e268 v4l: xilinx: multi-scaler: Set channel parameters to default values
- 54b31ac v4l: xilinx: multi-scaler: Fix precedence error
- 1fa95a9 media: xilinx: multi-scaler: Support stride and height alignment
- 0dce6af media: xilinx: multi-scaler: Add clock framework support
- f280f6a media: xilinx: multi-scaler: Add support for formats having contiguous buffers for two planes.
- b30b1e media: xilinx: multi-scaler: Align width to pixels per clock
- 68bbe2 media: xilinx: multi-scaler: Add extra padding bytes after each row of pixels
- fc81159 media: xilinx: multi-scaler: Replace dev_info with dev_dbg if all jobs are not ready
- 2e2ece media: xilinx: multi-scaler: support VB2_USERPTR IO mode

2018.3

Summary:
- Add first version of driver.
Commits:
- 484cdb2 platform: xilinx: Add mem to mem Multi-Scaler driver (XM2MSC)

Multi-Scaler Linux Driver

Introduction

HW IP features

Missing Features / Known Issues / Limitations in Driver

Kernel Configuration

Device Tree Binding

Test Procedure

N-N Use Case:

Commands to test

1 to N Use Case:

Commands to test

Mapping table between Gstreamer string and Xilinx Multi-Scaler Supported Formats

Debugging

Use Cases

1-1 Scaler

N-N Scaler

1-N Scaler

Stride and Height Alignment

Example 1:

Example 2:

Example 3:

Example 4:

Notes

Change Log

2025.1

2024.2

2024.1

2023.2

2023.1

2022.2

2022.1

2021.2

2021.1

2020.2

2020.1

2019.2

2019.1

2018.3

Related Links