Solution ZynqMP PL Programming

Introduction‍


The Zynq UltraScale+ MPSoC Programmable Logic (PL) can be programmed either using First Stage Boot-loader(FSBL), U-Boot or through Linux.
This page provides the details about programming the PL from Linux world using Linux FPGA Manager framework.

Flow:

HW IP Features

  • Full Bitstream and Partial Bitstream loading
  • Encrypted and Authenticated Full/Partial Bitstream loading
  • Readback of Configuration Registers
  • Readback of Bitstream(Configuration Data)
  • Compressed Bitstream

Features supported in the driver

  • Full Bitstream Support 
    • Non Secure Bitstream
    • Encrypted  Bitstream
    • Authenticated Bitstream 
    • Authenticated and Encrypted Bitstream
    • Compressed Bitstream
    • Loading Bitstream using Devicetree overlay 
  • Partial Bitstream Support that doesn't require PL drivers 
    • Non Secure Partial Bitstream
    • Encrypted Partial Bitstream 
    • Authenticated Partial Bitstream 
    • Authenticated and Encrypted Partial Bitstream 
  • Readback 
    • Configuration Registers Readback
    • Configuration Data Readback 

Missing Features, Known Issues and Limitations in the driver

  • No tool flow support for Partial Bitstream loading using Devicetree Overlay (DTG, Petalinux and Yocto)

        Important AR Links

             https://www.xilinx.com/support/answers/70504.html 

    • Valid for all the releases 

Note 1: The descriptions in subsequent sections refer to use of Device Tree Overlay (DTO) fragments with FPGA manager framework. It has to be noted that the generation of DTO fragments are not supported in 2018.2 and earlier official Xilinx Petalinux releases . If you are using older Petalinux release (earlier to 2018.3 release) please go through the  OSL Flow section for building software images.

Note 2: Usage of FPGA Manager with partial bitstreams is not supported by Xilinx Worldwide Technical Support and is considered an advanced use case.

Note 3: Axi Interrupt controller driver doesn't work as kernel modules/overlay like other device drivers due to the below limitation

                   Axi Interrupt controller driver uses IRQCHIP_DECLARE macro to register with irq sub-system. So it will follow the below sequence to invoke the xilinx_intc_of_init() (callback function)

                           start_kernel() –> init_IRQ() --> irqchip_init() --> of_irq_init() --> call-back function(xilinx_intc_of_init)

                   If The Driver was registered using IRQCHIP_DECLARE  it must be compiled into the kernel and the callback function will be invoked at kernel boot time.

Kernel Configuration

The following config options has to be enabled in order to use FPGA Manager, Please note that these options are enabled by default  through xilinx_zynqmp_defconfig except for the fpga debugfs option. If user wants to test readback feature they have enable it.

Zynq UltraScale+ MPSoC FPGA Manager Configuration:

Select: Device Drivers → FPGA Configuration Framework →   Xilinx Zynq UltraScale+ MPSoC FPGA 

In-order to test readback feature user needs to enable FPGA debug fs 

Select: Device Drivers →  FPGA Configuration Framework -->   FPGA debug fs 



DT overlay ConfigFS interface Configuration:
In-order to load Bitstream with DTBO user needs to enable below options

Select: Device Drivers --> Device Tree and Open Firmware support


Contiguous Memory Allocator Configuration:

CONFIG_CMA

Select: Kernel Features --> Contiguous Memory Allocator

CONFIG_DMA_CMA

Select: Device Drivers --> Generic Driver Options → DMA Contiguous Memory Allocator 


Devicetree


For more details about  devicetree bindings Reference Below link: Devicetree

Note: Above devicetree node is by default present in the zynqmp.dtsi file

Bitstream Format

From 2018.3 release onwards FPGA Manager supports loading of vivado and bootgen generated Bitstream and Bin files vivadobootgen[1]

Note: For releases earlier to 2018.3 FPGA Manager was capable of loading only bootgen generated bin files. bootgen

 .Bit Format 

  • Generated by Vivado 

 .Bin Format

  • Generated by Vivado 
  • Generated by Bootgen by converting vivado generated bit file
    Bootgen Command to generate bin file:
    # bootgen -image Bitstream.bif -arch zynqmp -o ./Bitstream.bin -w (2018.1 or later releases)
    # bootgen -image Bitstream.bif -arch zynqmp -process_Bitstream bin (2017.4 and earlier releases)
    Bitstream.bif file should contains the below lines:

Building Software Images

Xilinx tools (Petalinux, Yocto) provide an option with the name FPGA Manager which if enabled builds the device tree overlay fragments automatically and copies the Bitstream and DTBO files into root file system. 

Petalinux Flow

 Petalinux setup and build:  Petalinux

  1. Enable FPGA Manager in the top level Petalinux menuconfig 

# petalinux-config 

(OR)

Edit file build/conf/plnxtool.conf 

Add below line 

IMAGE_FEATURES += " fpga-manager"

      2. Select FPGA Manager user 

      3. Select the Specify hw directory path, in the prompt provide HDF path

Note: 

  •   By default it will pack base hdf Bitstream and dtbo to the /lib/firmware/base in the rootfs
  •   Specified hw directory path can contain multiple HDF's 

      4. Use petalinux-build command to build the required images

# petalinux-build 

Once build is complete, binaries are available at images/linux directory            

      5. Boot the hardware with newly built images

    • Directory /lib/firmware in the rootfs contains the Bitstreams and dtbo files  

Yocto Flow

Yocto setup and build: yocto

Once Yocto environment is set 

  1. Edit conf/local.conf add below lines of code

IMAGE_FEATURES += " fpga-manager"

EXTRA_HDF = "Path to the HDF"

Note: 

  •   By default it will pack base hdf Bitstream and dtbo to the /lib/firmware/base in the rootfs
  •   Extra hdf path can contain multiple HDF's 

      2. Build the required images (run below command)

# bitbake core-image-minimal 

Once build is complete, binaries are available at images/linux directory            

      3. Boot the hardware with newly built images

    • Directory /lib/firmware in the rootfs contains the Bitstreams and dtbo files  

OSL Flow

Setup and Build: OSL 

Images required for testing the FPGA Manager 

  • Bin/Bit file
  • DTBO file

DTBO File Generation:

  1. Steps to Create the DT-Overlay Fragment
    1. Clone the device-tree-xlnx repo:
      git clone device-tree-xlnx.git
    2. Helper script: dt_overlay.tcl 
      1. Command:  # xsct dt_overaly.tcl system.hdf psu_cortexa53_0 device-tree-xlnx output_dir

                            Ex: xsct dt_overaly.tcl system.hdf psu_cortexa53_0 ${DTS_REPO}/device-tree-xlnx overlay

      2. Create Device Tree Overlay Blob (.dtbo) file from the pl.dtsi file

                     # dtc -O dtb -o pl.dtbo -b 0 -@ pl.dtsi

                     Ex: ./scripts/dtc/dtc -O dtb -o pl.dtbo -b 0 -@ pl.dtsi

      3. Copy the generated bit and dtbo files to target 

Note: PL nodes should not be the part of Base Device-tree (system.dtb).

  • If the PL nodes are part of static dtb (system.dtb), these nodes cannot be removed at run-time.

FPGA programming 

  •   Linux FPGA Manager framework provides sysfs (Bitstream loading), debugfs (readback), configfs (Bitstream loading along with DTBO for PL drivers) attributes.
  •   Alternatively users can opt for Xilinx developed fpgautil. This utility provides an easy to use interface for programmers for all FPGA Manager use cases.  
  •   See Note 2 above with respect to partial bitstreams.

Exercising FPGA programming using fpgautil 

Source code for the fpgautil is available here fpgautil.c

Loading an overlay along with it's Bitstream

Usage: fpgautil -b <bin file path> -o <overlay file path> 


Removing an overlay

Usage: fpgautil -R 

Loading Bitstream only

Usage: fpgautil -b <bit/bin file absolute path> 

Loading Partial Bitstream that doesn't require PL drivers (See Note 2 above)

Usage: fpgautil -b <bit/bin file full path> -f Partial 

Loading Secure Authenticated Bitstream

Usage: fpgautil -b <bit/bin file full path> -f <flags> -s AuthDDR


Read back of FPGA Configuration Registers

Usage: fpgautil -r 


Read back of FPGA Configuration Data (Frames)

Usage: fpgautil -r <filename> -t 1

Verification of Read-back Data
  • To verify the readback data stream, compare it to the RBD golden readback file and masking readback bits with the MSD file. This approach is simple because there is a 1:1 correspondence between the start of the readback data stream and the start of the RBD and MSD files, making the task of aligning readback, mask, and expected data easier.

  • The RBD and MSD files contain an ASCII representation of the readback and mask data along with a file header that lists the file name, etc. This header information should be ignored or deleted. The ASCII 1s and 0s in the RBD and MSD files correspond to the binary readback data from the device.

  • For generating RBD and MSD refer UG908 (Vivado Design Suite User Guide)

    By compiling the below utility(verify_readback.c file) user can verify the read back contents.
    Source code: verify_readback

Note:

utility compares the readback with the RBD golden file, by masking readback bits with the MSD file. It is expected from user to remove/strip the header in the msd and rdb files and also remove the print statement in the readback contents file(readback.bin).

Example

Exercising FPGA programming using fpga framework attributes  

FPGA programming using sysfs attributes 

 Linux FPGA Manager driver creates /sys/class/fpga_manager/<fpga> folder, which contains file attributes that provides control and state. 

  • firmware:  /sys/class/fpga_manager/<fpga>/firmware
    firmware attribute requests an fpga image using the firmware class, then write output to the FPGA
  • flags:  /sys/class/fpga_manager/<fpga>/flags
    flags attribute determines the type of Bitstream 
    0 - Full Bitstream 1 - Partial Bitstream (default: 0, See Note 2 above)
  • state:  /sys/class/fpga_manager/<fpga>/state
    state attribute is a superset of FPGA states and FPGA Manager driver states
  • name:  /sys/class/fpga_manager/<fpga>/name
    name attribute is name of the low level FPGA Manager driver 
  • key: /sys/class/fpga_manager/<fpga>/key 
    key attribute stores the key value useful for Encrypted Bitstream loading to read the userkey
     

Steps for programming the Full Bitstream 

Set flags for Full Bitstream

  • echo 0 > /sys/class/fpga_manager/fpga0/flags

Load the Bitstream 

  • mkdir -p /lib/firmware

  • cp /media/design_1_wrapper.bit.bin /lib/firmware/

  • echo design_1_wrapper.bit.bin > /sys/class/fpga_manager/fpga0/firmware

Steps for programming the Partial Bitstream (See Note 2 above)

Set flags for Partial Bitstream

  • echo 1 > /sys/class/fpga_manager/fpga0/flags

Load the Bitstream Partial Bitstream

  • mkdir -p /lib/firmware

  • cp /media/partail_wrapper.bit.bin /lib/firmware/

  • echo partail_wrapper.bit.bin  > /sys/class/fpga_manager/fpga0/firmware

FPGA Readback using debugfs attributes

Linux FPGA Manager driver creates /sys/kernel/debug/fpga/ folder, which contains image attributes that provides readback contents.

  • readback_type:  /sys/module/zynqmp_fpga/parameters/readback_type
    readback_type is a module parameter which tells the readback type (default : 0) 
     0 – Configuration Register read 1 - Configuration Data read 
  • image:  /sys/kernel/debug/fpga/<fpga>/image
    image is a debugfs attribute which provides the readback information based on the readback_type module param

Steps for Readback of Configuration Registers

Set flags for readback type

echo 0 > /sys/module/zynqmp_fpga/parameters/readback_type

Perform Readback operation by reading the image debug attribute 

cat /sys/kernel/debug/fpga/fpga0/image

Example:


Steps for Readback of Configuration DataFrames 

Set flags for readback type

echo 1 > /sys/module/zynqmp_fpga/parameters/readback_type

Perform Readback operation by reading the image debug attribute 

cat /sys/kernel/debug/fpga/fpga0/image

Example:


FPGA programming using Device Tree Overlay (DTO)

The Device Tree Overlay (DTO) is used to reprogram an FPGA while Linux is running. The DTO overlay will add the child node and the fragments from the .dtbo file to the base device tree,, The newly added device node/drivers will be probed after Bitstream programming

DTO contains:

  • Target FPGA Region
    - "target-path" or "target" - The insertion point where the the contents of the overlay will go into the live tree.
    target-path is a full path, while target is a phandle.
  • FPGA Image firmware file name
    - "firmware-name" - Specifies the name of the FPGA image file on the firmware search path.
    The search path is described in the firmware class documentation.
  • Image specific information
    - external-fpga-config : boolean, set if the FPGA has already been configured prior to Linux boot up.
  • Child devices
    - child nodes corresponding to hardware that will be loaded in this region of the FPGA.

Steps for programming the Full Bitstream using overlay

Copy the Full Bitstream (.bin) and pl.dtbo files into firmware folder

  • mkdir /configfs

  • mount -t configfs configfs /configfs

  • cd /configfs/device-tree/overlays/

  • mkdir full

  • echo -n "pl.dtbo" > full/path

Steps to remove the drivers got added as part of DTO

  • rmdir /configfs/device-tree/overlays/full

Devicetree overlay file contents example:


Partial Bitstream Solution that needs PL driver support (See Note 2 above)

There is no out-of-box tool flow support for loading Partial Bitstreams that need PL drivers. The creation of DTO files needed for this approach is not supported in the tool flow (DTG, Petalinux, Yocto).

Users can still achieve their goal of programming partial bitstreams that need PL driver support by using hand-written DTSi files.

Steps for programming the Partial Bitstream using handwritten dtsi file

Copy the Partial Bitstream (.bin) and rm.dtbo files into lib/firmware folder

  • mkdir /configfs

  • mount -t configfs configfs /configfs

  • cd /configfs/device-tree/overlays/

  • mkdir partial

  • echo 1 > /sys/class/fpga_manager/fpga0/flags
  • echo -n "pl.dtbo" > partial/path

Steps to remove the drivers got added as part of Partial DTO

  • rmdir /configfs/device-tree/overlays/partial

Mainline Status

The current driver availble in the xilinx linux git is in sync with the open source 4.19 kernel driver except for the following

  • Encrypted and Authenticated Full/Partial Bitstream loading
  • Readback of Configuration Registers
  • Readback of Bitstream(Configuration Data)

Release history

2020.2

Summary:

  • fpga: zynqmp: Use the scatterlist interface
  • fpga: zynqmp: Initialized variables before using it
  • fpga: zynqmp: Fix incorrect variables type

Commits:

  • 4823227 fpga: zynqmp: Use the scatterlist interface
  • aac8be7 fpga: zynqmp: Initialized variables before using it
  • 2899bc8 fpga: zynqmp: Fix incorrect variables type

2020.1

  • None

2019.2

Summary:

  • fpga: zynqmp-fpga: Adds status interface

Commits:

  • 8e85861 fpga: zynqmp-fpga: Adds status interface

2019.1

Summary:

  • fpga: Fix bitstream typo error
  • Merge tag 'v4.19' into master
  • fpga: ZynqMP: Adds support for Authentication of bitstreams usning User-key
  • drivers: xilinx: Reorganize firmware driver for zynqmp
  • drivers: Defer probe if firmware is not ready
  • fpga: zynqmp: Revert Authentication of bitstreams using User-key changes
  • fpga: zynqmp: Use SPDX license header

Commits:

  • e732840 fpga: Fix bitstream typo error
  • 26a8a4d Merge tag 'v4.19' into master
  • c609186 fpga: ZynqMP: Adds support for Authentication of bitstreams usning User-key
  • 3f9e46d drivers: xilinx: Reorganize firmware driver for zynqmp
  • 6a63448 drivers: Defer probe if firmware is not ready
  • 8d43780 fpga: zynqmp: Revert Authentication of bitstreams using User-key changes
  • 3384284 fpga: zynqmp: Use SPDX license header

2018.3

Summary:

  • Added support for vivado generated bit and bin file loading
  • Added support for PL configuration readback 
  • Added support for clock framework 

Commits:

  • daca3d fpga: zynqmp: Adds support to load vivado generated .bit and .bin files
  • bd1f10  fpga: zynqmp-fpga: Add support for pl configuration readback
  • 097ea7 fpga: zynqmp-fpga: Add support for clock framework 


2018.2
Summary:

  • Added Support for Partial Reconfiguration

Commits:

  • No Changes in the driver (Changes are done in the xilfpga library).


2018.1
Summary:

  • soc: zynqmp: Define EEMI ops structure
  • soc: zynqmp: Use new firmware APIs
  • fpga: zynqmp: sync driver with xilfpga library enhancements
  • fpga: zynqmp: Remove useless blank line in zynqmp_fpga_remove

Commits:

  • 31c2a5e soc: zynqmp: Define EEMI ops structure
  • 0b7483c soc: zynqmp: Use new firmware APIs
  • 45ec97f fpga: zynqmp: sync driver with xilfpga library enhancements
  • fbf0102 fpga: zynqmp: Remove useless blank line in zynqmp_fpga_remove


2017.4

  • None

2017.3
Summary:

  • zynqmp: Use new firmware.h instead of pm.h
  • Revert "fpga manager: Adopted Authenticated Bitstream loading support for Xilinx"

Commits:

  • 5e81ba5 zynqmp: Use new firmware.h instead of pm.h
  • a7fbcf3 Revert "fpga manager: Adopted Authenticated Bitstream loading support for Xilinx"

2017.2

  • None


2017.1
Summary:

  • fpga manager: Adopted Encrypted Bitstream loading support for Xilinx ZynqMp.
  • fpga: zynqmp: Fix ZynqMP name in print.
  • fpga manager: Adopted Authenticated Bitstream loading support for Xilinx.
  • fpga: zynqmp: Add fpga image information struct.

Commits:

  • a17a6e1 fpga manager: Adopted Encrypted Bitstream loading support for Xilinx ZynqMp.
  • 7130ff7 fpga: zynqmp: Fix ZynqMP name in print.
  • ed5a141 fpga manager: Adopted Authenticated Bitstream loading support for Xilinx.
  • b8bc48c fpga: zynqmp: Add fpga image information struct.


2016.4

  • None


2016.3
Summary:

  • fpga manager Adding FPGA Manager support for Xilinx zynqmp Soc
  • fpga: Removed warning from zynqmp-fpga.c compilation.

Commits:

  • 280ca3f fpga manager: Adding FPGA Manager support for Xilinx zynqmp Soc
  • ad53092 fpga: Removed -warning from zynqmp-fpga.c compilation.

See also 

Zynq FPGA Manager

References

https://github.com/Xilinx/linux-xlnx/blob/master/drivers/fpga/zynqmp-fpga.c

https://github.com/Xilinx/linux-xlnx/blob/master/Documentation/fpga/fpga-mgr.txt

https://github.com/Xilinx/linux-xlnx/blob/master/Documentation/devicetree/bindings/fpga/fpga-region.txt

https://github.com/Xilinx/embeddedsw/tree/master/lib/sw_services/xilfpga/doc