Multi-Scaler Linux Driver


Introduction

This page gives an overview of Xilinx Multi-Scaler driver, which is available as part of the Xilinx Linux distribution.

The driver is part of the V4L2 Memory to Memory framework.

The Multi-Scaler driver generates up to eight scaled output images from a single or multiple (up to eight) external video and/or graphics sources. Video scaling is the process of converting an input color image of dimensions Xin pixels by Yin lines to an output color image of dimensions Xout pixels by Yout lines. This IP also capable of doing color space conversion. Multi-Scaler IP reads the input images from the memory addresses and writes the scaled output images to the destination memory addresses.

The Multi-scaler works in a sequential way, i.e. the input of the first channel scaled first and then the 2nd input gets scaled and so no up to the maximum channels configured in HwReg_num_outs. When all channels (configured) get scaled, IP generates the interrupt and ready for next processing.

Multi-Scaler converts and scales video frames between a great variety of video formats. For more supported formats please visit IP Product guide.

Xilinx-multiscaler creates a device node per channel and can be accessed from user-space via standard V4L2 system calls. Even though the devices created by driver looks like independent nodes but as multi-scaler IP has one scaler inside and works in sequential manner these nodes/devices are not independent. For more details see limitation section. 

HW IP features

  • Memory mapped AXI4 Interface
  • Supports maximum 8 outputs
  • Supports spatial resolutions from 64 × 64 up to 7680 × 4320
  • Supports one, two, or four pixel-width
  • Supports RGB, YUV 444, YUV 422, YUV 420
  • Supports 8-bit and 10-bit per color component on memory interface
  • Supports semi-planar memory formats next to packed memory formats
  • Dynamically configurable source and destination buffer addresses
  • Supports 6, 8, 10, and 12 taps in both H and V domains
  • Supports 64 phases
  • Supports 8K 30 fps depending on the device family

Missing Features / Known Issues / Limitations in Driver

  • Tested 64 × 64 up to 1920 × 1080.
  • 10-bit per color component is not tested.
  • As hardware processes all channels in sequential order, the driver has limitation to add only the next immediate channel. 

Example: Channels 0,1,2,3 are streaming, then the user can stream on immediate next channel 4.

If user adds stream for channel 6 when channels 0,1,2,3 are running, then driver does not start 6th channel and does not return any error.

If any intermediate channel stops the streaming, the streaming continues for only channels before the stopped channel.

In this case if channel 0,1,2,3,4 are running and channel 3 stops, then the streaming for channels 0,1,2 will continue and driver stop processing 4th channel without any error indications.

A channel is not processed by the driver, if all channel with lower channel number are not queueing the buffers.

E.g. If 0,1,2,3 channels were being processed by driver and channel 1 didn't queue any buffers, then channel 2 and 3 won't be processed but channel 0 will continue.

  • The driver is tested with YUY2, NV16, NV12, RGB, GRAY8, UYVY and BGR Gstreamer formats only.  
  • Please check section below on Stride and Height align.


Kernel Configuration

The following config options should be enabled in order to build Xilinx Multi-Scaler driver

CONFIG_VIDEO_XILINX

CONFIG_VIDEO_DEV

CONFIG_VIDEO_V4L2

CONFIG_VIDEO_XILINX_MULTISCALER


To check if the options are enabled run the following:

petalinux-config -c kernel

When the GUI opens, you can use '/' to search for each config.


The driver is available at,
https://github.com/Xilinx/linux-xlnx/blob/master/drivers/media/platform/xilinx/xilinx-multi-scaler.c

Device Tree Binding

The device tree node will be automatically generated, if the core is configured in the HW design, using the Device Tree BSP.

Steps to generate device-tree is documented here,
http://www.wiki.xilinx.com/Build+Device+Tree+Blob

And a sample binding is shown below and the description of DT property is documented here


v_multi_scaler_1: v_multi_scaler@a0000000 {
          clock-names = "ap_clk";
          clocks = <&misc_clk_0>;
          compatible = "xlnx,v-multi-scaler-1.0", "xlnx,v-multi-scaler-v1.0";
          interrupt-names = "interrupt";
          interrupt-parent = <&gic>;
          interrupts = <0 89 4>;
          reg = <0x0 0xa0000000 0x0 0x20000>;
          reset-gpios = <&gpio 78 1>;
          xlnx,dma-addr-width = <0x20>;
          xlnx,max-chan = <8>;
          xlnx,max-height = <2160>;
          xlnx,max-width = <3840>;
          xlnx,num-taps = <12>;
          xlnx,pixels-per-clock = /bits/ 8 <2>;
          xlnx,vid-formats = "xrgb8888", "bgr888", "xbgr8888", "xbgr2101010", "uyvy", "y8", "y10", "vuy888", "xvuy8888", "yuvx2101010", "yuyv", "nv12", "nv16", "xv20", "xv15";
};                                

Test Procedure

A reference design for testing is as below

Multi-Scaler driver creates a video device per channel, which can be visual in /dev directory.

The device names are /dev/videoN to /dev/videoN+8 where N starts from the next available video device to the maximum number of devices the IP is configured (xlnx, max-chan in DT).   

Example:

# ls /dev/ | grep video
video0
video1
video2
video3
video4
video5
video6
video7


Multi-Scaler is tested with the generic gstreamer v4l2videotransform plugin which maps device node to v4l2videoNconvert elements. Where N is the number of device node. 

N-N Use Case:

Multi-Scaler is tested with the generic gstreamer v4l2videotransform elements which is part of gst-plugins-good. So no extra library required to run N:N use case.

In below image, it is represented a basic test scenario where multiple v4l2videoNconvert elements are launched as different process opening different node of Multi-Scale. The input for the v4l2videoNconvert is frames of different format and different resolutions and the output is given to filesink which stores the scaled and converted frames in a file.    



Commands to test

Below command is an example using Gstreamer. You can open Multiple channels by running v4l2videoNconvert process in parallel.
**Note Gstreamer plugins may change, please refer to Gstreamer documentation for the latest information on using Gstreamer plugins

gst-launch-1.0 videotestsrc num-buffers=10 ! video/x-raw, width=1280, height=720, format=RGB ! v4l2convert capture-io-mode=4 output-io-mode=4 ! video/x-raw, width=640, height=480, format=RGB ! filesink location=tmp0.rgb


1 to N Use Case:

Multi-Scaler's 1 to N use case  is tested with the xlnxabrscaler plugin. 

In below image, it is represented a basic test scenario where single input given to xlnxabrscaler elements which opens different node of Multi-Scaler. The output is given to filesink which stores the scaled and converted frames in files.    


Commands to test

For 1 to N Use Case, run commands mention in debugging section.

Mapping table between Gstreamer string and Xilinx Multi-Scaler Supported Formats

Replace the format string in above command with below Gstreamer strings, to test different supported formats 


ID

Multi-Scaler

Video Format

Prefix with XILINX_M2MSC_FMT_

Device-tree

strings

V4L2

Format

Prefix with

V4L2_PIX_FMT_

Gstreamer

String

YUV Player

Format

10

RGBX8

xbgr8888BGRX32

11YUVX8xvuy8888XVUY32

12YUYV8yuyvYUYVYUY2YUYV
15RGBX10xbgr2101010XBGR30

16YUVX10yuvx2101010XVUY10

18Y_UV8nv16NV16MNV16NV16
19Y_UV8_420nv12NV12MNV12NV12
20RGB8bgr888RGB24RGBRGB24
21YUV8vuy888VUY24

22Y_UV10xv20XV20M
XV20
23Y_UV10_420xv15XV15M
XV15
24Y8y8GREYGRAY8Y
25Y10y10Y10
Y10
27BGRX8xrgb8888XBGR32BGRxBGR32
28UYVY8uyvyUYVYUYVYUYVY
29BGR8rgb888BGR24BGRBGR24

Debugging

Enable Dynamic Debugging in kernel.

Run below commands to enable logs on console

echo 1 > /sys/module/v4l2_mem2mem/parameters/debug
echo 8 > /sys/module/videobuf2_core/parameters/debug
echo 8 > /sys/module/videobuf2_v4l2/parameters/debug
echo 8 > /proc/sys/kernel/printk

Use Cases

1-1 Scaler

gst-launch-1.0 videotestsrc num-buffers=10 ! video/x-raw, width=1280, height=720, format=RGB ! v4l2convert capture-io-mode=4 output-io-mode=4 ! video/x-raw, width=640, height=480, format=RGB ! filesink location=640X480.rgb

N-N Scaler