Video Framebuffer Read
*This page contains info for Frame buffer Write IP too as there is a single driver for both Frame buffer Read and Write IP.
Table of Contents
Introduction
Interfacing with the Video Framebuffer Driver from DMA Clients
The general steps for preparing DMA to write to a specific memory buffer:
- Using the Video Framebuffer API, configure the DMA device with the expected memory format for write
- Prepare an interleaved template describing the buffer location (note: see section DMA Interleaved Template Requirements below for more details)
- Pass the interleaved template to the DMA device using the Linux DMA Engine interface
- With the DMA descriptor which is returned from step 3, add a callback and then submit to the DMA device via the DMA Engine interface
- Start the DMA write operation
- Terminate DMA write operation when frame processing deemed complete by client
/* Abstract V4L2 Client Code Example */ struct dma_chan *frmbuf_dma = to_frmbuf_dma_chan(xdev); struct dma_interleaved_template dma_tmplt; dma_addr_t addr = vb2_dma_contig_plane_dma_addr(vb2_buffer_ptr, 0); u32 flags = DMA_PREP_INTERRUPT | DMA_CTRL_ACK; /* Step 1 - Configure the dma channel to write out packed RGB */ xilinx_xdma_v4l2_config(frmbuf_dma, V4L2_PIX_FMT_RGB24); /* Step 2 - Describe the buffer attributes for a 1080p frame */ dma_tmplt.dir = DMA_DEV_TO_MEM; /* DMA_MEM_TO_DEV */ dma_tmplt.src_sgl = false; dma_tmplt.dst_sgl = true; dma_tmplt.dst_start = addr; dma_tmplt.frame_size = 1; /* single plane pixel format */ dma_tmplt.numf = 1080; /* 1920x1080 frame */ dma_tmplt.sgl[0].size = 5760; /* 3 bytes/pixel x 1920 pixels */ dma_tmplt.sgl[0].icg = 0; /* Step 3 - Submit the buffer description to the dma channel */ desc = dmaengine_prep_interleaved_dma(frmbuf_dma, &&dma_tmplt, flags); desc->callback = dma_complete; desc->callback_param = buf; /* Step 4 - Submit the returned and updated descriptor to the dma channel */ dmaengine_submit(desc); /* Step 5 - Start dma to memory operation */ dma_async_issue_pending(frmbuf_dma); /* Step 6 - Halt DMA when required frame processing completed */ dmaengine_terminate_all(frmbuf_dma);
DMA Interleaved Template Requirements
The Video Framebuffer IP supports two dma address pointers for semi-planar formats: one for luma and one for chroma. As such, data for the two planes need not be strictly contiguous which permits for alignment of plane data within a larger buffer. However, all frame data (luma and chroma) must be contained within a single, larger contiguous frame buffer and luma plane data should be arranged to come before chroma data within this frame buffer space. Note that this is not a limitation imposed by the IP but by the driver at this moment. When preparing a struct dma_interleaved_template instance to describe a semi-planar format, the following members must be filled out as follows:From linux/dmaengine.h:
struct dma_interleaved_template:
dst_start = <physical address from which to start reading frame data (any offsets should be added to this value)>src_sgl = false
dst_sgl = true
numf = <height of frame in pixels; height of luma frame for semi-planar formats>
frame_size = < 1 or 2 depending on whether this is describing a packed or semi-planar format>
sgl = <see struct data_chunk below>
struct data_chunk:
sgl[0].size = <number of bytes devoted to image data for a row>sgl[0].icg = < number of non-data bytes within a row of image data; padding>
sgl[0].dst_sgl = <the offset in bytes between the end of luma frame data to the start of chroma plane data; only needed for semi-planar formats>
Below is a code example for semi-planar YUV 422 (i.e. NV16) demonstrating how steps 1 and 2 of the above code snippet change in such a case:
/* Step 1 - Configure the dma channel to write out semi-planar YUV 422 */ xilinx_xdma_v4l2_config(frmbuf_dma, V4L2_PIX_FMT_NV16M); /* use xilinx_xdma_drm_config with DRM_FORMAT_NV16 */ /* Step 2 - Describe the buffer attributes for a 1080p frame */ dma_tmplt.dir = DMA_DEV_TO_MEM; /* use DMA_MEM_TO_DEV for Framebuffer Read */ dma_tmplt.src_sgl = false; dma_tmplt.dst_sgl = true; dma_tmplt.dst_start = luma_addr; dma_tmplt.frame_size = 2; /* two plane pixel format */ dma_tmplt.numf = 1080; /* height of luma frame */ dma_tmplt.sgl[0].size = 1920; /* 1 byte/pixel x 1920 pixels for Y plane */ dma_tmplt.sgl[0].icg = 0; frame_height = dma_tmplt.numf; stride = dma_tmplt.sgl[0].size + dma_tmplt.sgl[0].icg; dma_tmplt.sql[0].dst_icg = chroma_addr - luma_addr - (frame_height * stride);
Driver Operation
The Framebuffer driver manages buffer descriptors in software keeping them in one of four possible states in the following order:
- pending
- staged
- active
- done
When a DMA client calls dma_commit(), the buffer descriptor is placed in the driver’s “pending” queue. Multiple buffers can be queued in this manner by the DMA client before proceeding to the next step (see step 4 of Interfacing with the Video Framebuffer Driver from DMA Clients).
When dma_async_issue_pending() is called (step 5 in the client code sample above), the driver begins processing all queued buffers on the “pending” list. A buffer is picked from the pending list and then stored as “staged”. At this moment, driver programs the registers with data provided within the “staged” buffer descriptor. During normal processing (i.e. all frames except the first frame*), these values will not become active until the currently processed frame completes. As such, there is a one-frame delay between programming and the actual writing data to memory. Hence the term “staged” to describe this part of the buffer lifecycle.
When the currently active frame completed, the buffer descriptor is classified as “active” in the driver. At this point, a new descriptor is picked from the pending list and this new buffer is marked as “staged” with its values programmed into the IP registers as described earlier. The buffer marked “active” represents the data currently being written to memory. Other than being held in the “active” state, no other action is taken with the buffer
When the active frame completes, it is moved to the “done” list. The driver utilizes a tasklet which is called at the end of the frame interrupt handler. The tasklet will process any buffer descriptors on the done list by removing them from the list and calling any callback the client has linked to the descriptor.
This completes the life cycle of a buffer descriptor. As can be seen, with four possible states, it is best to allocate at least four buffers to maintain consistent frame processing. Fewer buffers will result in gaps within the pipeline and result in frame data within a given buffer being overwritten one or more times (depending on how few buffers are queued and the number of resulting gaps in the driver’s buffer pipeline).
Buffer Alignment
- Note: normally, registers programmed while the IP is running will not take effect until the next frame. The very first frame, however, is an exception: the IP is not yet running and, as such, the values take effect immediately. Nevertheless, there is no additional special treatment given the first frame buffer. As such, it will be written to, in effect, twice.
IP/Driver Features
IP features | 2018.1 | 2018.2 | 2018.3 | 2019.1 | 2019.2 | 2020.1 | 2020.2 | 2021.1 | 2021.2 | 2022.1 | 2022.2 | 2022.3 |
---|---|---|---|---|---|---|---|---|---|---|---|---|
IP version | 2.0 | 2.0 | 2.1 | 2.1 | 2.2 | 2.3 | 2.4 | |||||
Streaming Video Formats supported | RGB, RGBA, YUV 4:4:4, YUVA 4:4:4, YUV 4:2:2, YUV 4:2:0 | |||||||||||
Color Formats supported | Video Formats with per Pixel Alpha (valid only for Framebuffer Read) RGBA8, BGRA8, YUVA8 Support for 8 bit Video Formats RGBX8,RGB8, BGRX8,BGR8,YUVX8,YUV8,YUYV8,UYVY8,Y_UV8,Y_UV8_420,Y8 Support for 10 bit Video Formats RGBX10, YUVX10, Y_UV10,Y_UV10_420,Y10 | Video Formats with per Pixel Alpha (valid only for Framebuffer Read) RGBA8, BGRA8, YUVA8 Support for 8 bit Video Formats RGBX8,RGB8, BGRX8,BGR8,YUVX8,YUV8,YUYV8,UYVY8,Y_UV8,Y_UV8_420,Y8 Support for 10 bit Video Formats RGBX10, YUVX10, Y_UV10,Y_UV10_420,Y10 Support for 12 bit Video Formats RGBX12, YUVX12, Y_UV12, Y_UV12_420, Y12 Driver supports only RGBX12. Support for 16 bit Video Formats RGB16, YUV16, Y_UV16, Y_UV16_420, Y16. Driver supports only RGB16 | Added support for Y_U_V8 3 planar video format. | Added support for Y_U_V10 3 planar video format. | ||||||||
Supports progressive and interlaced video | IP supported both progressive and interlaced Driver only supported progressive | IP and Driver both support progressive and interlaced video | ||||||||||
Maximum and Minimum spatial resolution | Max 8192x4320 Min 64 x 64 | Max 10328 x 7760 (Driver tested for standard resolutions up to 8K only) Min 64 x 64 | Max 15360 x 8640 Min 64 x 64 | |||||||||
Pixels per clock | 1,2,4 ppc | IP supports 1,2,4 and 8 ppc Driver doesn't support 8 ppc | Driver supports 1, 2, 4 and 8 ppc |
Missing Features / Known Issues / Limitations in Driver
- Tested for standard resolutions up to 8K
- YUV 12 and 16 bpc color formats support added in IP in 2019.1 but are not supported in driver in 2019.1.
- The first buffer returned by the driver will contain the second frame contents, after this the driver will correctly give the data
- When DMA operations are initiated by a client, the hardware is placed into "autorestart" mode. When the last buffer has been returned to the client as "completed", if the client does not supply a new write/read buffer location or fails to halt the driver, then the last buffer location written to will continue to be utilized by the driver. In effect, the driver will "spin" on the last location programmed.
Kernel Configuration
The driver must be enabled in the kernel by selecting option CONFIG_XILINX_FRMBUFDevice Tree Binding
Complete documentation on the device tree requirements may be found in the Linux source located at xilinx_frmbuf.txtTesting Procedure
To ensure the Framebuffer Write IP Linux driver has been configured to work properly, a suitable test design will require an input source (i.e. HDMI Rx) connected to the Framebuffer.
Once properly configured, the design can be tested via the tool known as "yavta". yavta may be found here.
To run yavta, data must be streaming into your media pipeline. To verify the status of your media pipleline, run the tool known as "media-ctl":
root@hdmi_proj:~# media-ctl -p Media controller API version 0.1.0 Media device information ------------------------ driver xilinx-video model Xilinx Video Composite Device serial bus info hw revision 0x0 driver version 0.0.0 Device topology - entity 1: vcap_hdmi output 0 (1 pad, 1 link) type Node subtype V4L flags 0 device node name /dev/video0 pad0: Sink <- "a0000000.v_hdmi_rx_ss":0 [ENABLED] - entity 5: a0000000.v_hdmi_rx_ss (1 pad, 1 link) type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev0 pad0: Source [fmt:UYVY/1920x1080 field:none] [dv.caps:BT.656/1120 min:0x0@25000000 max:4096x2160@297000000 stds:CEA-861,DMT,CVT,GTF caps:progressive,reduced-blanking,custom] [dv.detect:BT.656/1120 1920x1080p60 (2200x1125) stds:CEA-861 flags:CE-video] -> "vcap_hdmi output 0":0 [ENABLED]
root@hdmi_proj:~# yavta -c10 -f YUYV -s 1920x1080 --skip 7 -F /dev/video0 && [2] 2362 Device /dev/video0 opened. Device `vcap_hdmi output 0' on `platform:vcap_hdmi:0' is a video output (without mplanes) device. Video format set: YUYV (56595559) 1920x1080 field n[ 1393.139514] one, 1 planes: * Stride 3840, buffer size 4147200 <snip> [ 1393.747654] xhdmi_s_stream enable = 0 Captured 10 frames in 0.289203 seconds (34.577689 fps, 0.000000 B/s). 8 buffers released. [2]- Done yavta -c10 -f YUYV -s 1920x1080 --skip 7 -F /dev/video0 root@hdmi_proj:~# ls frame-000007.bin frame-000008.bin frame-000009.bin
Additionally, run modetest to change the output resolution with the -v argument which will result in page flipping on the primary plane
root@mixer_proj:~# modetest -M xilinx_drm_mixer -s 37:640x480@BG24 -v setting mode 640x480-75Hz@BG24 on connectors 37, crtc 35 select timed out or error (ret 0) freq: 7.20Hz freq: 15.00Hz freq: 15.00Hz freq: 15.00Hz freq: 15.00Hz freq: 15.00Hz
The output frequency reported should be approximately 1/4 that of the current refresh rate. This is because modetest only creates a single framebuffer and the Video Framebuffer driver requires four (4) buffers for optimal operation.
Boards Supported
- ZCU102
- ZCU106
Known Issues
- AR68764 - LogiCORE Video Frame Buffer Read - Release Notes and Known Issues for the Vivado 2017.1 tool and later versions
Change Log
2023.1
- Summary
- No changes
2022.2
- Summary
- Add support for 3 planar YUV 444 10 bits per component
2022.1
- Summary
- No changes
2021.2
- Summary
- Add support for 3 planar YUV 444 8 bits per component
- Add support for 15360 x 8640 resolution
- Fix code alignment
- Update the SPDX license header
- Fix kernel doc warning
- Add support for field id error detection
- Correct the ADDR3 register offset
- Commits
- dmaengine: xilinx: frmbuf: Add support for 3 planar YUV444 8bpc
- dmaengine: xilinx: frmbuf: Add support for max 15360x8640 resolution
- dmaengine: xilinx: frmbuf: Fix code alignment to match paranthesis
- dmaengine: xilinx: frmbuf: Update to SPDX license header
- dmaengine: xilinx: frmbuf: Fix kernel doc warning
- dmaengine: xilinx: frmbuf: Add support for field id error detection
- engine: xilinx: frmbuf: Correct ADDR3 register offset to support 3 pl…
2021.1
- Summary
- Export pixels per clock as width alignment for dma clients
- Commits
2020.2
- Summary
- Correctly free up resources while removing dma
- Fix Y10 v4l2 pixel format
- Fix for Coverity warnins
- Use AP_DONE instead of AP_READY
- Commits
2020.1
- Summary
- Add support 8 ppc
- Commits
- bb91ad8 dmaengine: xilinx: frmbuf: Add support for 8 ppc
2019.2
- Summary
- Add support for low latency capture
- Commits
2019.1
- Summary
- Make max-width, max-height dt properties mandatory
- Support 12,16 bpc RGB color formats
- Support early callback using bitmask
- Call the call back registered just before programming the descriptor in EARLY_CALLBACK_LOW_LATENCY mode.
- Add clock framework support
- Commits
09e6f94 dmaengine: xilinx: frmbuf: Make max-width and max-height mandatory properties
04fc254 dmaengine: xilinx: frmbuf: Add support for 12 and 16 bpc RGB formats
6ce7c27 dma: xilinx: Release buffers before DMA transfer
d0489cb dma: xilinx: Support early callback modes
1d2bb41 dma: xilinx: Add clock framework support to Framebuffer driver
2018.3
- Summary
- Add default mode for non streaming use case which has only states.
- Add early callback support
- Mark previous active descriptor as complete in irq handler
- Add check for max-width and height
- Commits
2018.2
- Summary
- Flush FIFO on halt.
- Get DMA alignment from pixels per clock or dma-align device tree properties
- Add interlaced support
- Commits
ecc3314 dma: xilinx: Add fid property for interlaced support in framebuffer
699b419 dma: xilinx: Flush the framebuffer FIFO on halt
f805b07 dma: xilinx: Get DMA alignment from device tree for Framebuffer
16c2f51 dma: xilinx: Add alpha formats support for Framebuffer Read
705bac9 dma: xilinx: Add interlaced support to Xilinx Framebuffer driver
2018.1
- Summary
- Add XV15, XV20, BGR8, RGBA8, YUVA8, RGBX10, YUVX10, Y_UV10, Y_UV10_420, Y10, BGRA8, BGRX8 and UYVY8 color formats
- Fix V4L2 PIX FORMAT for BGRX8
- Commits
2017.4
- Summary
- No changes
2017.3
- Summary
- Add initial driver version
- Commits
f618d3b dma: xilinx: Add private API to permit retrieval of supported mem formats
7858ca5 dma: xilinx: Bug fix to ensure GPIO is reset between DMA operations
e506e84 Bug fix to ensure only video formats enabled in IP are in driver
49026f2 dma: xilinx: Update to Framebuffer Driver to support dual addr pointers
166f3ac dma: xilinx: New driver for Video Framebuffer IP
Related Links
© Copyright 2019 - 2022 Xilinx Inc. Privacy Policy