1 1. RPU Firmware Build Steps
2 2. Overview of FMC Card
3 3. Sensor Drivers
- 3.1 3.1 About Sensor Lib Architecture
- 3.2 3.2 Steps to Integrate New Sensor Driver
4 4. Independent Sensor Interface Functions
- 4.1 4.1 General API Functions

1. RPU Firmware Build Steps

1.1 The Firmware Package Structure

The shared package has the following components.
1. Sensor library source code (sensor_drivers)
2. ISP Firmware library binaries (prebuilt_lib)
3. Main application (main_app)

1.1.1 Sensor Library Source Code (`sensor_drivers`)

The following is the folder structure:

── sensor_drivers
    └── src
        ├── hal
        │   └── include
        │       ├── common
        │       ├── ebase
        │       ├── hal
        │       ├── mbox
        │       └── oslayer
        └── usr_sensor
            ├── include
            │   ├── fmc
            │   ├── isi
            │   └── sensor_drv
            └── source
                ├── fmc
                ├── Multi_sensor_framework
                └── sensor_drv

This source code is designed to integrate a custom sensor driver, specifically tailored to meet user requirements.
The sensor library supports the following.

FMC Card	Image Sensor Driver	Serializer	De-serializer

FMC Card	Image Sensor Driver	Serializer	De-serializer
logiFMC-GMSL2-9296A-12C	Ox03f10	max9295A	max9296A
logiFMC-GMSL2-9296A-12C	Ox08b40	max9295A	max9296A

1.1.2 ISP Firmware Library Binaries (`prebuilt_lib`)

The following is the folder structure:

── prebuilt_lib
│   ├── libfreertos.a
│   ├── libisp_fw_lib.a
│   ├── libsensor_lib.a
│   ├── libxil.a
│   ├── libxilffs.a
│   ├── libxilstandalone.a
│   ├── libxiltimer.a
│   ├── LICENSE.txt
├── README.md

These ISP firmware libraries encompass the essential functionalities required to manage both the ISP and sensor drivers effectively.

1.1.3 Main Application (`main_app`)

The following is the folder structure:

├── main_app
│   └── src
│       ├── CMakeLists.txt
│       ├── isp_fw_main.h
│       ├── lscript.ld
│       ├── main.c
│       └── UserConfig.cmake

This application is designed to activate the functionality of ISP firmware. Additionally, it facilitates the integration of all sensor libraries and ISP firmware libraries, compiling them into a single executable.

1.2 Creating the Platform

Follow the steps to create a platform for a VEK385 device.

Clone the following GitHub repository from ISP Firmware.
To build ISP firmware, use custom lopper, empyro binaries and Vitis launch steps as described in the Answer Record.
- Refer to (Answer Record 000037970) to launch Vitis.
Create a New Platform
1. File → New Component → Platform
Click Browse to select the location where the component should be stored.
Browse the XSA. (Refer to PG432 to generate the ISP XSA).
Select the OS and Processor
1. Choose FreeRTOS as the Operating System
2. Choose cortexr52_6 as the Processor
Click Finish to generate the Platform

1.3 Creating the ISP Firmware Application

Follow the steps to create and link the ISP firmware application.

Create an Empty Application in the same project created in Creating the Platform.
1. File → New Component → Application
Set the Component name to main_app and Browse the Path where the application will be located.
Select the Platform created in Creating the Platform.
Click Next
Click Add folder, and select the /src path from <GIT_REPO>/main_app/src.
Click Finish
Verify all the source files are present in src directory as shown in the below image:
Set the LIB_ROOT path in the CMakeLists.txt to point the prebuilt_lib.
- Set (LIB_ROOT <GIT_REPO>/prebuilt_lib) in the file main_app->Settings->CMakeLists.txt
Once all the changes are completed, select main_app and click build to compile application.
After build completes successfully, check the <**project_name>.elf binary in the output section.

1.4 Creating a Sensor Library (Optional)

If a user chooses to go with other sensors (which is not supported by AMD in this release), they can hook their sensor drivers as per the Independent Sensor Interface Functions section and build it using the following process.

Create a Static Library
File → New Component → Static Library
Browse the Path where the sensor library will be located
Select the Platform created in Creating the Platform.
Verify cortexr52_6 processor is selected and the OS is FreeRTOS, then click Next.
Click on add directory button to add the sensor driver source, from the <sensor_sources> directory to the Vitis project.
Click Finish.
Verify the project and build the sensor_lib component.
Find the library in the output sections as libsensor_lib.a
1. Users can use this library to add to the main applications LIB_ROOT path and build the latest elf as per the Creating the ISP Firmware Application steps.

2. Overview of FMC Card

The Xylon FMC (logiFMC-GMSL2-9296A-12C) is used to interface multiple image sensors to the VEK385 board.

The Xylon logiFMC-GMSL2 (logiFMC-GMSL2-9296A-12C) 12-Channel GMSL2 FMC daughter card supports Maxim Integrated Gigabit Multimedia Serial Link (GMSL2).

The card supports multiple image sensors, enabling sophisticated applications such as surround vision cameras. It is also designed to enable quick prototyping and evaluation of automotive multi-camera Advanced Driver Assistance (ADAS) and Autonomous Driving (AD) applications, while providing the flexibility to adapt to various sensor types.

_{Xylon FMC Card}

_{FMC Internal Architecture}

2.1 Xylon FMC Card and Sensor Setup

The FMC card supports 12 Sensors using 6 De-Serializers (D1-D6). Each De-serializer supports two sensors that are connected through FAKRA connectors with Serializers in between, a total of 12 Serializers (S1-S12) and Sensors (Sensor1-Sensor12) can be connected. Data is transmitted between Serializers and De-Serializers through GMSL/GMSL2 link.

_{FMC and Sensor Connections}

When you examine the FMC card, you will see three connectors labeled as CN2, CN3, and CN4 on one side. A user can connect a new sensor to any of these channels.

The connector numbers are shown in the following diagram, and these are passed it as a parameter during the execution of the binaries for capturing pipeline enablement.

Screenshot 2025-06-05 125729-20250605-074640.jpg

_{FMC Pin Configuration}

A new user should follow the complete ISP firmware execution guide to determine which ISP is being enabled. and correspondingly, which MIPI interface is active (refer to the section FMC Card and Sensor Setup). The sensor must be connected only to the port associated with the enabled MIPI interface.

2.2 Connecting a Sensor to FMC

The sensors are connected through FAKRA Connectors to the de-serializer pins of the FMC. This setup is then linked to the VEK385 board through the Mobile Industrial Processor Interface (MIPI).

The following image gives a visual representation of the connection between the Sensors, FMC and VEK385 board.

_{Sensor, FMC and VEK385 Board Connections}

2.3 Overview of Sensor Library

The sensor library provides a reference codebase for the Sensor and FMC drivers. Vendors are expected to modify this codebase to accommodate their specific sensor and FMC configurations.

Refer to the GitHub page GitHub - Xilinx/ISP_Firmware for the 2025.1 ISP firmware release.

The file tree structure for the sensor library files, which includes all the necessary include and source files, as outlined below.

To check on the sensor driver files, expand usr_sensor folder located in sensor_drv.

_{Sensor Driver File Structure}

Supported Sensor List

As illustrated in the following table, the sensor driver includes files corresponding to the reference sensor drivers.

Resolution	Sensor Make	Sensor Model	Part Number	Frames per second (FPS)
3MP	Omni Vision	Ox03f10	LI-VENUS-OX03F-075H	30 fps
8MP	Omni Vision	Ox08b40	LI-VENUS-OX08B40-9295-030H	15 fps

2.4 Compiling of ISP Firmware with Sensor Lib

Refer to the section Creating a Sensor Library (Optional) and the execution guide shared with the GitHub path.

2.5 Execution Setup

Refer to the ISP Linux Driver [2025.1] guide for details on executing image sensor capture pipeline.

3. Sensor Drivers

3.1 About Sensor Lib Architecture

The ISP firmware invokes a specific sensor driver through the ISI Layer integrated with the RPU firmware control layer Application Programming Interface (APIs). It gets the driver handle of the specific sensor and register with the ISP firmware for that streaming operation.

The yellow box in the following image is the user_sensor static library (refer to section FMC Card and Sensor Setup) integrated with the RPU firmware through the Independent Sensor Interface (ISI) layer.

The following image explains the basic architecture about the integration of the Sensor Library with RPU ISP firmware.

_{Sensor Library Integration with RPU Firmware}

Refer to the Independent Sensor Interface Functions section for further information about sensor ISI APIs.

3.2 Steps to Integrate New Sensor Driver

Follow the steps below to integrate a new sensor driver into the RPU ISP firmware.

Update the sensor driver configuration entry in the sensor_drv.c file.
1. Refer to the following usr_sensor folder for sensor driver files and update the sensor_drv.c file.
  _{Sensor Library File Tree}
2. Modify the sensor_drv.c file for new sensor configuration as given below.
  _{Addition to New Sensor Instance}
Copy the new sensor driver include and source files into the sensor lib code base: new_sensor.c, new_sensor.h similar to the existing Ox03f10.c and Ox03f10.h files.
1. Follow the existing code directory to add the new files.
  _{Include and Source files for New Sensor}
Each sensor driver needs to declare this structure for mapping sensor driver calls to ISI.
1. Refer to the existing sensor_drv.c file.
2. Update the IsiCamDrvConfig_t structure for the new camera instance as follows:
  _{Addition of New Camera Instance}
3. Update the following ISI function pointer calls to the new sensor driver specific call.
  _{ISI Function Pointer APIs}
Implement the sensor driver ISI calls as per the new sensor driver requirements.
Once Completed, compile the updated sensor library codebase as per Creating a Sensor Library (Optional).

4. Independent Sensor Interface Functions

4.1 General API Functions

IsiCreateIss

This function creates sensor context for the given config.

Syntax:
      RESULT IsiCreateIss (
              IsiSensorInstanceConfig_t    *pconfig,
              IsiSensorHandle_t            *phandle
      );
Parameters:
      pConfig     Gives Sensor Config
      handle      Sensor instance handle.
      
Returns
RESULT Return Code: RET_SUCCESS, RET_NULL_POINTER

IsiOpenIss

This function opens a sensor.

Syntax:
      RESULT IsiOpenIss (
              IsiSensorHandle_t    handle,
              uint32_t             mode
      );
Parameters:
      handle      Sensor instance handle.
      mode        Current sensor mode
Returns
RESULT Return Code: RET_SUCCESS, RET_NULL_POINTER, RET_OUTOFMEM, RET_NOTSUPP

IsiCloseIss

This function closes a sensor.

Syntax:
      RESULT IsiCloseIss (
            IsiSensorHandle_t    handle
      );
Parameters:
        handle        Sensor instance handle.
Returns
RESULT Return Code: RET_SUCCESS, RET_NULL_POINTER, RET_OUTOFMEM, RET_NOTSUPP

IsiReadRegIss

This function reads the value from the specified register from the image sensor device.

Syntax:
      RESULT IsiReadRegIss (
              IsiSensorHandle_t     handle,
              const uint16_t        addr,
              uint16_t              *pValue
      );
Parameters:
        handle      Sensor instance handle.
        addr        Register address.
        *pValue     Register value read from the register.
Returns
RESULT Return Code: RET_SUCCESS, RET_FAILURE, RET_WRONG_HANDLE, RET_NULL_POINTER, RET_NOTSUPP

IsiWriteRegIss

This function writes a given number of bytes to the image sensor device by calling the corresponding sensor function.

Syntax:
      RESULT IsiWriteRegIss (
              IsiSensorHandle_t    handle,
              const uint16_t       addr,
              const uint16_t       value
      );
Parameters:
        handle      Sensor instance handle.
        addr        Register address.
        Value       Register value to write.
Returns
RESULT Return Code: RET_SUCCESS, RET_FAILURE, RET_WRONG_HANDLE, RET_NULL_POINTER, RET_NOTSUPP

IsiGetModeIss

This function gets sensor mode information corresponding to the sensor mode index.

Syntax:
      RESULT IsiGetModeIss (
              IsiSensorHandle_t     handle,
              IsiSensorMode_t       *pMode
      );
Parameters:
        handle      Sensor instance handle.
        *pMode      Pointer to the IsiSensorMode_t data structure.
Returns
RESULT Return Code: RET_SUCCESS, RET_WRONG_HANDLE, RET_NULL_POINTER, RET_NOTSUPP

IsiEnumModeIss

This function enumerates sensor mode information corresponding to the sensor mode index.

Syntax:
      RESULT IsiEnumModeIss (
              IsiSensorHandle_t        handle,
              IsiSensorEnumMode_t      *pMode
      );
Parameters:
        handle      Sensor instance handle.
        *pMode      Pointer to the IsiSensorEnumMode_t data structure.
Returns
RESULT Return Code: RET_SUCCESS, RET_WRONG_HANDLE, RET_NULL_POINTER, RET_NOTSUPP

IsiGetCapsIss

This function returns the sensor description structure.

Syntax:
      RESULT IsiGetCapsIss (
              IsiSensorHandle_t     handle,
              IsiCaps_t             *pCaps
      );
Parameters:
        handle      Sensor instance handle.
        *pCaps      Pointer to the IsiCaps_t data structure.
Returns
RESULT Return Code: RET_SUCCESS, RET_WRONG_HANDLE, RET_NULL_POINTER

IsiCheckConnectionIss

This function checks the connection to the camera sensor, if available.

Syntax:
      RESULT IsiCheckConnectionIss (
              IsiSensorHandle_t     handle
      );
Parameters:
        handle      Sensor instance handle.
Returns
RESULT Return Code: RET_SUCCESS, RET_FAILURE, RET_NULL_POINTER, RET_NOTSUPP

IsiGetRevisionIss

This function reads the sensor revision register and returns the register value.

Syntax:
      RESULT IsiGetRevisionIss (
              IsiSensorHandle_t     handle,
              uint32_t              *pRevision
      );
Parameters:
        handle          Sensor instance handle.
        *pRevision      Pointer to the returned sensor revision register value.
Returns
RESULT Return Code: RET_SUCCESS, RET_FAILURE, RET_WRONG_HANDLE, RET_NULL_POINTER, RET_NOTSUPP

IsiSetStreamingIss

This function enables/disables streaming of sensor data, if possible.

Syntax:
      RESULT IsiSetStreamingIss (
              IsiSensorHandle_t     handle,
              bool_t                on
      );
Parameters:
        handle      Sensor instance handle.
        on          New Streaming State.
                      BOOL_TRUE=on
                      BOOL_FALSE=off
Returns
RESULT Return Code: RET_SUCCESS, RET_FAILURE, RET_WRONG_HANDLE, RET_NULL_POINTER, RET_NOTSUPP, RET_WRONG_STATE

IsiGetAeBaseInfoIss

This function returns the Ae related information of the specified sensor instance.

Syntax:
      RESULT IsiGetAeBaseInfoIss (
              IsiSensorHandle_t     handle,
              IsiAeBaseInfo_t       *pAeBaseInfo
      );
Parameters:
        handle            Sensor instance handle.
        *pAeBaseInfo      Pointer to the sensor AE base information (IsiAeBaseInfo_t).
Returns
RESULT Return Code: RET_SUCCESS, RET_WRONG_HANDLE, RET_NULL_POINTER, RET_NOTSUPP

IsiGetAGainIss

This function returns the sensor analog gain.

Syntax:
      RESULT IsiGetAGainIss (
             IsiSensorHandle_t      handle,
             IsiSensorGain_t        *pSensorAGain
      );
Parameters:
        handle          Sensor instance handle (e.g., OV14825).
        *pSensorAGain   Pointer to the sensor analog gain (IsiSensorGain_t).
Returns
RESULT Return Code: RET_SUCCESS, RET_WRONG_HANDLE, RET_NULL_POINTER, RET_NOTSUPP

IsiSetAGainIss

This function sets the sensor analog gain.

Syntax:
      RESULT IsiSetAGainIss (
             IsiSensorHandle_t     handle,
             IsiSensorGain_t       *pSensorAGain
      );
Parameters:
        handle          Sensor instance handle.
        *pSensorAGain   Pointer to the sensor analog gain (IsiSensorGain_t).
Returns
RESULT Return Code: RET_SUCCESS, RET_WRONG_HANDLE, RET_NULL_POINTER, RET_NOTSUPP

IsiGetDGainIss

This function returns the sensor digital gain.

Syntax:
      RESULT IsiGetDGainIss (
             IsiSensorHandle_t      handle,
             IsiSensorGain_t        *pSensorDGain
      );
Parameters:
        handle          Sensor instance handle.
        *pSensorDGain   Pointer to the sensor digital gain (IsiSensorGain_t).
Returns
RESULT Return Code: RET_SUCCESS, RET_NULL_POINTER, RET_NOTSUPP, RET_WRONG_HANDLE

IsiSetDGainIss

This function sets the sensor digital gain.

Syntax:
      RESULT IsiSetDGainIss (
             IsiSensorHandle_t      handle,
             IsiSensorGain_t        *pSensorDGain
      );
Parameters:
        handle          Sensor instance handle.
        *pSensorDGain   Pointer to the sensor digital gain (IsiSensorGain_t).
Returns
RESULT Return Code: RET_SUCCESS, RET_WRONG_HANDLE, RET_NULL_POINTER

IsiGetIntTimeIss

This function gets the sensor integration time.

Syntax:
      RESULT IsiGetIntTimeIss (
             IsiSensorHandle_t      handle,
             IsiSensorIntTime_t     *pSensorIntTime
      );
Parameters:
        handle              Sensor instance handle.
        *pSensorIntTime     Pointer to the sensor integration time (IsiSensorIntTime_t).
Returns
RESULT Return Code: RET_SUCCESS, RET_WRONG_HANDLE, RET_NULL_POINTER

IsiSetIntTimeIss

This function sets the sensor integration time.

Syntax:
      RESULT IsiSetIntTimeIss (
             IsiSensorHandle_t           handle,
             const IsiSensorIntTime_t    *pSensorIntTime
      );
Parameters:
        handle             Sensor instance handle.
        *pSensorIntTime    Pointer to the sensor integration time (IsiSensorIntTime_t).
Returns
RESULT Return Code: RET_SUCCESS, RET_WRONG_HANDLE, RET_NULL_POINTER

IsiGetFpsIss

This function returns the current frame rate of the sensor.

Syntax:
      RESULT IsiGetFpsIss (
             IsiSensorHandle_t        handle,
             uint32_t                 *pFps
      );
Parameters:
        handle           Sensor instance handle.
        *pFps            Pointer to the sensor’s frame rate.
Returns
RESULT Return Code: RET_SUCCESS, RET_WRONG_HANDLE, RET_NULL_POINTER, RET_NOTSUPP

IsiSetFpsIss

This function sets the sensor frame rate.

Syntax:
      RESULT IsiSetFpsIss (
             IsiSensorHandle_t    handle,
             uint32_t             fps
      );
Parameters:
        handle           Sensor instance handle.
        fps              Frame rate to set.
Returns
RESULT Return Code: RET_SUCCESS, RET_WRONG_HANDLE, RET_NULL_POINTER, RET_NOTSUPP

IsiGetIspStatusIss

This function returns the sensor ISP status.

Syntax:
      RESULT IsiGetIspStatusIss (
             IsiSensorHandle_t        handle,
             IsiIspStatus_t           *pIspStatus
      );
Parameters:
        handle           Sensor instance handle.
        *pIspStatus      Pointer to the IsiIspStatus_t structure.
Returns
RESULT Return Code: RET_SUCCESS, RET_WRONG_HANDLE, RET_NULL_POINTER

IsiSetWBIss

This function sets the sensor white balance.

Syntax:
      RESULT IsiSetWBIss (
             IsiSensorHandle_t     handle,
             const IsiSensorWb_t   *pWb
      );
Parameters:
        handle           Sensor instance handle.
        *pWb             Pointer to the IsiSensorWb_t structure.

Returns
RESULT Return Code: RET_SUCCESS, RET_WRONG_HANDLE, RET_NULL_POINTER, RET_NOTSUPP

IsiGetWBIss

This function gets the sensor white balance.

Syntax:
      RESULT IsiGetWBIss (
             IsiSensorHandle_t     handle,
             const IsiSensorWb_t   *pWb
      );
Parameters:
        handle           Sensor instance handle.
        *pWb             Pointer to the IsiSensorWb_t structure.

Returns
RESULT Return Code: RET_SUCCESS, RET_WRONG_HANDLE, RET_NULL_POINTER, RET_NOTSUPP

IsiSetBlcIss

This function sets the sensor black level.

Syntax:
      RESULT IsiSetBlcIss (
             IsiSensorHandle_t     handle,
             const IsiSensorBlc_t  *pBlc
      );
Parameters:
        handle           Sensor instance handle.
        *pBlc            Pointer to the IsiSensorBlc_t structure.
Returns
RESULT Return Code: RET_SUCCESS, RET_WRONG_HANDLE, RET_NULL_POINTER, RET_NOTSUPP

IsiGetBlcIss

This function gets the sensor black level.

Syntax:
      RESULT IsiGetBlcIss (
             IsiSensorHandle_t    handle,
             IsiSensorBlc_t       *pBlc
      );
Parameters:
        handle           Sensor instance handle.
        *pBlc             Pointer to the IsiSensorBlc_t structure.
Returns
RESULT Return Code: RET_SUCCESS, RET_WRONG_HANDLE, RET_NULL_POINTER, RET_NOTSUPP

IsiSetTpgIss

This function sets the sensors test pattern (default pattern: color bar).

Syntax:
      RESULT IsiSetTpgIss (
             IsiSensorHandle_t     handle,
             IsiSensorTpg_t        tpg
      );
Parameters:
        handle           Sensor instance handle.
        tpg              Sensor test pattern information(IsiSensorTpg_t).
Returns
RESULT Return Code: : RET_SUCCESS, RET_WRONG_HANDLE, RET_WRONGSTATE, RET_NULL_POINTER, RET_NOTSUPP

IsiGetTpgIss

This function gets the sensors test pattern (default pattern: color bar).

Syntax:
      RESULT IsiGetTpgIss (
             IsiSensorHandle_t handle,
             IsiSensorTpg_t *pTpg
      );
Parameters:
        handle           Sensor instance handle.
        *pTpg            Sensor test pattern information pointer(IsiSensorTpg_t).
Returns
RESULT Return Code: : RET_SUCCESS, RET_WRONG_HANDLE, RET_WRONGSTATE, RET_NULL_POINTER, RET_NOTSUPP

IsiGetExpandCurveIss

This function returns the sensor expand curve.

Syntax:
      RESULT IsiGetExpandCurveIss (
             IsiSensorHandle_t        handle,
             IsiSensorCompandCurve_t  *pCurve
      );
Parameters:
        handle           Sensor instance handle.
        *pCurve          Pointer to the IsiSensorCompandCurve_t structure.
Returns
RESULT Return Code: : RET_SUCCESS, RET_FAILURE, RET_WRONG_HANDLE, RET_NULL_POINTER, RET_NOTSUPP

ISP Firmware [2025.1]

1. RPU Firmware Build Steps

1.1 The Firmware Package Structure

1.1.1 Sensor Library Source Code (sensor_drivers)

1.1.2 ISP Firmware Library Binaries (prebuilt_lib)

1.1.3 Main Application (main_app)

1.2 Creating the Platform

1.3 Creating the ISP Firmware Application

1.4 Creating a Sensor Library (Optional)

2. Overview of FMC Card

2.1 Xylon FMC Card and Sensor Setup

2.2 Connecting a Sensor to FMC

2.3 Overview of Sensor Library

Supported Sensor List

2.4 Compiling of ISP Firmware with Sensor Lib

2.5 Execution Setup

3. Sensor Drivers

3.1 About Sensor Lib Architecture

3.2 Steps to Integrate New Sensor Driver

4. Independent Sensor Interface Functions

4.1 General API Functions

IsiCreateIss

IsiOpenIss

IsiCloseIss

IsiReadRegIss

IsiWriteRegIss

IsiGetModeIss

IsiEnumModeIss

IsiGetCapsIss

IsiCheckConnectionIss

IsiGetRevisionIss

IsiSetStreamingIss

IsiGetAeBaseInfoIss

IsiGetAGainIss

IsiSetAGainIss

IsiGetDGainIss

IsiSetDGainIss

IsiGetIntTimeIss

IsiSetIntTimeIss

IsiGetFpsIss

IsiSetFpsIss

IsiGetIspStatusIss

IsiSetWBIss

IsiGetWBIss

IsiSetBlcIss

IsiGetBlcIss

IsiSetTpgIss

IsiGetTpgIss

IsiGetExpandCurveIss

1.1.1 Sensor Library Source Code (`sensor_drivers`)

1.1.2 ISP Firmware Library Binaries (`prebuilt_lib`)

1.1.3 Main Application (`main_app`)