/
Versal AI Edge Series Gen 2 ISP Linux Driver and ISP Firmware

Versal AI Edge Series Gen 2 ISP Linux Driver and ISP Firmware

1. Versal AI Edge Series Gen 2 - ISP Software Architecture 

ISP is a hard block in a Multimedia Distributed (MMD) tile of the AMD VersalTM AI Edge Series Gen 2 SoC. The ISP software stack operates across both the Real-Time Processing Unit (RPU) and Application Processing Unit (APU) in a Split Architecture mode. The ISP software driver within the APU is responsible for extracting the ISP hardware configuration derived from the Device Tree Generator (DTG). These configurations are then transmitted to the RPU using Inter Process Communication (IPC) for the purpose of configuring the ISP instance.

The following diagram describes the structure of this Split Architecture.

ISP Core Stack.png

 

The APU software encompasses the Linux operating system, configured to boot ARM Cortex® A78 cores. It includes essential drivers such as the remote proc driver, facilitating firmware loading onto the ARM Cortex R52 core-6 which is configured in lock-step mode. Additionally, an ISP driver extracts ISP configuration details from the device tree, registers itself within the V4L2 framework, and communicates with the RPU firmware through an IPC driver.

The APU operates by parsing an input JSON file that contains comprehensive information regarding active ISPs, their operational modes (for example, streaming, MCM), buffer quantities, input/output formats, and respective resolutions. Additionally, it manages the enabling of ISP filters and their configuration parameters. All of these crucial parameters are structured and transmitted via the mailbox to the RPU for execution.

The RPU firmware is the software running on the SRPU used for configuring the ISP instance, executing the 2A algorithm, configuring the sensor, IPC driver and other platform-related drivers such as I2C, IPI, and TTC.

ISP End to End Pipelines:

The ISP E2E pipeline contains a camera sensor, MIPI CSI 2 D-PHY, ISP (Processing IP), Video mixer (HLS IP) and HDMI 2.1 TX.

Non-MCM Use Case: In a Non-MCM use case, each ISP instance receives live data from the specific sensor.

image-20250604-152315.png

MCM Use Case: In an MCM use case, each ISP instance can handle a maximum of 4 live input streams.

image-20250604-152424.png

2. Steps to Generate ISP Example Design and Build Software Components

Generate ISP Example Design: Begin by generating the ISP Vivado example design, according to the steps outlined in the PG432 ISP Product Guide.

Generate Yocto-based Linux Artifacts: Follow the instructions provided in the Linux section to Create Yocto-based Linux Artifacts. In case MIPI is connected to the FPD domain in the Programmable Logic (PL) design, refer to this MIPI Stability AR for guidance on addressing the MIPI stability issue.
To address the CDO clock issues in example designs (Non-MCM LIMO/LILO with TILE0), apply the patch from this Answer Record.

The firmware currently uses VC ID 0 for sensors 2, 5, and 6, and VC ID 1 for sensors 3 and 7. To fix any mismatch, either update the design (like the IBA VC ID setting in the ISP config wizard) to match the firmware, or change the sensor library to match the current design.

ISP Firmware Preparation: You can either use the pre-built ISP firmware binary from the generated root filesystem (rootfs) or generate the ISP firmware by following the steps detailed on the ISP Firmware page. If MIPI is connected to the Low Power Domain (LPD) in the PL design, refer the MIPI stability Answer Record for solutions to the MIPI stability issue.

Load and Validate Artifacts: Once VEK385 board setup is complete (Xylon FMC Card and Sensor Setup), based on the boot mode, copy the artifacts to their respective location. See the Steps to Run ISP Pipeline MIMO/LIMO/LILO section in the Linux build page to load the artifacts onto the target.

3. ISP IP/Driver Features

The below table describes the list of ISP features, input/output formats, max resolutions, and output paths supported as part of the 2025.1 release.

I/O Type 

Driver (2025.1)

I/P Formats

 

O/P Formats
(Gstreamer Terminology)

No of I/P Streams

Input Type

Max Resolution

Memory/Live Input

Output Path

Format

Media-ctl Name

 

MIMO

 NON-MCM

gbrg,
grbg,
rggb,
bg10,
gb10,
ba10,
rg10,
bg12,
gb12,
ba12,
rg12

BGRG8,
GRBG8, RGGB8, SBGGR10, SGBRG10, SBGGR10, SRGGB10, SBGGR12, SGBRG12, SBGGR12, SRGGB12

NV16,
NV12,
YUY2,
GRAY8,
RGB

1

File input

1920x1080

Memory

Primary Path

LIMO

NON-MCM

rg12

SRGGB12

NV16, NV12, YUY2, RGB.

1 stream per ISP (4 total ISP Instances)

3MP/8MP

1920x1080 / 3840x2160

Live

Primary Path and Secondary Path

MCM

rg12

SRGGB12

NV16, NV12, YUY2, RGB.

4 streams on single ISP instance

3MP

1920x1080

Live

Primary Path and Secondary Path

LILO

NON-MCM

rg12

SRGGB12

RGB

1 stream per ISP (2 total ISP Instances)

3MP

1920x1080

Live

Primary Path

Only RPU6 firmware is supported. ISP Firmware runs only on RPU6 which manages the MCM 4 stream use case and Non-MCM 4 ISP instance pipeline use case.

4. Features Not Supported in ISP stack for 2025.1

  • RGB-IR sensor (5 MP)

  • Multiple ISP instance combinations for MCM mode:

    • 4 streams on ISP0 + 1 stream on ISP1 instance

    • 3 streams on ISP0 + 2 streams on ISP1 instance

    • 2 streams on ISP0 + 2 streams on ISP1 instance

  • Multiple instances of ISP firmware running in AMP mode

5. Known Issues

  • MCM is validated using a maximum of four sensors operating at 15 fps. Running a pipeline at 30 fps can in the long run lead to frame-drops or pipeline hangs.

  • Closing a single pipeline in the MCM use case causes hangs on other running pipelines of the same ISP instance

  • If a design includes AXI-Switch and AXI-Broadcaster IPs as part of ISP pipeline, errors are observed while converting pl.dtsi to dtbo because of end point linking issues. To overcome this issue, users have to edit the remote end point in pl.dtsi.

  • Memory In Memory Out (MIMO) pipeline with fakesink is not working.

  • Live In Live Out (LILO) driver is not validated with designs that have MIPI connected to the FPD in the PL design.

  • Closing the media_server application using Ctrl+C causes a core dump. Instead use the pkill -9 isp_media_serve command to close the media_server application.

  • Unloading of the ISP overlay using dfx-manager leads to Kernal Crashes. A reboot is needed for rerun.

  • Use only the shared Tuning files, custom Tuning Parameters cannot be supported.

6. Limitations

  • Only a single instance of firmware can be loaded and run on RPU cluster-3, core-0 (R52_6).

  • In Non-MCM mode, only two ISP tiles can be validated with four sensors.

  • In MCM-mode, the firmware can handle up to 4 MCM streams only on a single ISP.

  • 3 ISP Tiles can support max 15 sensors. Due to a XylonFMC + VEK385 board pin limitation, only 4 streams are supported.

  • On a VEK385 board, Xylon FMC supports only 3 MIPI interfaces, only MIPI-2 and MIPI-6 is validated. MIPI-5 is not validated due to design limitations.

  • An End to End pipeline with 8MP (3840x2160) sensor can achieve up to 15 fps.

  • In certain Vivado example designs, ISP SLCR Clocks are not being programmed. See the ISP SLCR Clocks Answer Record for guidance on addressing ISP SLCR Clocks.

  • MIPI instability issues are observed with some VEK385 boards. See the MIPI stability Answer Record for guidance on addressing the MIPI stability issue.

If the MIPI Answer Record is used on FPD designs, an additional module visp_dummy_drv.ko needs to loaded along with visp modules.

7. Boards Supported

  • VEK385-A1 (P5A)

  • Xylon logiFMC-GMSL2-9296A-12C - 12-Ch GMSL2 FMC Board (Xylon Part Number:  11-00216-20-00001)

8. Reference Links

 

 

© Copyright 2019 - 2022 Xilinx Inc. Privacy Policy