Introduction‍


The Zynq UltraScale+ MPSoC Programmable Logic (PL) can be programmed either using First Stage Boot-loader(FSBL), U-Boot or through Linux.
This page provides the details about programming the PL from Linux world using Linux FPGA Manager framework.

Flow:


HW IP Features


Features supported in the driver

Known Issues for PL Programing

The Xilinx device tree generator (https://github.com/Xilinx/device-tree-xlnx) currently lacks automated support for device tree overlay generation for Partial Reconfiguration / DFX designs.  For systems requiring runtime device tree overlay (eg, Linux device driver support) support, hand-crafted device trees can be deployed and loaded. See the "Working with Device Tree Overlay (DTBO)" section below for details on how to load these at runtime.

Occasionally, bare-metal applications relying on the XilFPGA library may produce an error getting the DONE status.  See Answer Record 70504 for more details - https://www.xilinx.com/support/answers/70504.html 

Note1: If the user tries to load the un-aligned bit/bin file the PL configuration takes a longer time when compared with aligned(word-aligned) bit/bin files.

Note 2: Using overlay’s we can add a new node or add/update the existing node properties. But it will not allow replacing the existing nodes which is already part of the live tree. 

Note 3: If the PL design is having axi-intc IP the relevant DT node should be the top-node in the overlay.dtsi or pl.dtsi file

Example DT node.

/dts-v1/;
/plugin/;
/ {
	fragment@0 {
		target = <&fpga_full>;
		overlay0: __overlay__ {
			#address-cells = <2>;
			#size-cells = <2>;
			firmware-name = "system.bit.bin";
			resets = <&zynqmp_reset 116>;
		};
	};
	fragment@1 {
		target = <&amba>;
		overlay1: __overlay__ {
			afi0: afi0 {
				compatible = "xlnx,afi-fpga";
				config-afi = < 0 0>, <1 0>, <2 0>, <3 0>, <4 0>, <5 0>, <6 0>, <7 0>, <8 0>, <9 0>, <10 0>, <11 0>, <12 0>, <13 0>, <14 0xa00>, <15 0x000>;
			};
		};
	};
	fragment@2 {
		target = <&amba>;
		overlay2: __overlay__ {
			#address-cells = <2>;
			#size-cells = <2>;
			axi_intc_1: interrupt-controller@a00d0000 { /* Top node */
				#interrupt-cells = <2>;
				clock-names = "s_axi_aclk";
				clocks = <&clk_wiz_0 0>;
				compatible = "xlnx,axi-intc-4.1", "xlnx,xps-intc-1.00.a";
				interrupt-controller ;
				interrupt-names = "irq";
				interrupt-parent = <&axi_intc_0>;
				interrupts = <1 2>;
				reg = <0x0 0xa00d0000 0x0 0x10000>;
				xlnx,kind-of-intr = <0x360>;
				xlnx,num-intr-inputs = <0xe>;
			};
			axi_iic_0: i2c@a0000000 {
				#address-cells = <1>;
				#size-cells = <0>;
				clock-names = "s_axi_aclk";
				clocks = <&clk_wiz_0 0>;
				compatible = "xlnx,axi-iic-2.1", "xlnx,xps-iic-2.00.a";
				interrupt-names = "iic2intc_irpt";
				interrupt-parent = <&gic>;
				interrupts = <0 91 4>;
				reg = <0x0 0xa0000000 0x0 0x1000>;
			};
			axi_iic_1: i2c@a0008000 {
				#address-cells = <1>;
				#size-cells = <0>;
				clock-names = "s_axi_aclk";
				clocks = <&clk_wiz_0 0>;
				compatible = "xlnx,axi-iic-2.1", "xlnx,xps-iic-2.00.a";
				interrupt-names = "iic2intc_irpt";
				interrupt-parent = <&axi_intc_1>;
				interrupts = <1 2>;
				reg = <0x0 0xa0008000 0x0 0x1000>;
			};
		};
	};
};





Summary Table for Partial Reconfiguration / Dynamic Function Exchange Feature Support

Partial Reconfiguration / Dynamic Function Exchange (DFX) is an advanced feature of Xilinx Zynq UltraScale+ MPSoC devices.  Refer to the table below for details.  For PetaLinux releases prior to 2018.3 or for building Linux system images manually, please refer to the Linux OSL Flow page.

Release \ FeatureVivado RTL Design FlowVivado IPI Design FlowLinux Device Tree GeneratorLinux Programming Framework
<= 2018.3SupportedNot SupportedAutomation Not Supported - Hand CraftedFPGA Manager
2019.1 - 2020.2SupportedDFX LoungeDFX LoungeFPGA Manager / fpgautil






Kernel Configuration

The following config options has to be enabled in order to use FPGA Manager, Please note that these options are enabled by default  through xilinx_zynqmp_defconfig except for the fpga debugfs option. If user wants to test readback feature they have enable it.

Zynq UltraScale+ MPSoC FPGA Manager Configuration:

Select: Device Drivers → FPGA Configuration Framework →   Xilinx Zynq UltraScale+ MPSoC FPGA 

In-order to test readback feature user needs to enable FPGA debug fs 

Select: Device Drivers →  FPGA Configuration Framework -->   FPGA debug fs 



DT overlay ConfigFS interface Configuration:
In-order to load Bitstream with DTBO user needs to enable below options

Select: Device Drivers --> Device Tree and Open Firmware support


Contiguous Memory Allocator Configuration:

CONFIG_CMA

Select: Kernel Features --> Contiguous Memory Allocator

CONFIG_DMA_CMA

Select: Device Drivers --> Generic Driver Options → DMA Contiguous Memory Allocator 


Devicetree

pcap {
       compatible = "xlnx,zynqmp-pcap-fpga";
       clocks = <&clk 41>;
};

fpga_full: fpga-full {
       compatible = "fpga-region";
       fpga-mgr = <&pcap>;
       #address-cells = <2>;
       #size-cells = <2>;
	   power-domains = <&zynqmp_firmware PL_PD>;
};


For more details about  devicetree bindings Reference Below link: Devicetree

Note:

Bitstream Format

From 2018.3 release onwards FPGA Manager supports loading of vivado and bootgen generated Bitstream and Bin files vivadobootgen[1]

Note: For releases earlier to 2018.3 FPGA Manager was capable of loading only bootgen generated bin files. bootgen

 .Bit Format 

 .Bin Format

all:
{
        [destination_device = pl] Full.bit (or)  Partial.bit /* Bitstream file name */
}

Building Software Images

Xilinx tools (Petalinux, Yocto) provide an option with the name FPGA Manager which if enabled builds the device tree overlay fragments automatically and copies the Bitstream and DTBO files into root file system. 

PetaLinux Flow

 PetaLinux setup and build:  PetaLinux

  1. Enable FPGA Manager in the top level PetaLinux menuconfig options.
$ petalinux-config ---> FPGA Manager

(OR)

Edit <plnx-proj-root>/project-spec/meta-user/conf/petalinuxbsp.conf file and add FPGA Manager package to EXTRA_IMAGE_FEATURES variable.

EXTRA_IMAGE_FEATURES += " fpga-manager"


      2. Select FPGA Manager user 

Preview

      3. Select the Specify hw directory path, in the prompt provide HDF path

Note: 

      4. Use petalinux-build command to build the required images

$ petalinux-build

Once build is complete, binaries are available at images/linux directory            

      5. Boot the hardware with newly built images

Yocto Flow

Yocto setup and build: yocto

Once Yocto environment is set 

  1. Edit <yocto-proj-root>/build/conf/local.conf add below lines of code
EXTRA_IMAGE_FEATURES += " fpga-manager"
EXTRA_HDF = "<PATH_TO_HDF_or_XSA>"

Note: 

      2. Build the required images (run below command)

$ bitbake petalinux-image-minimal

Once build is complete, binaries are available at ${DEPLOY_DIR_IMAGE} (${TMPDIR}/deploy/images/${MACHINE}/) directory            

      3. Boot the hardware with newly built images

OSL Flow

Setup and Build: OSL 

Images required for testing the FPGA Manager 

DTBO File Generation:

  1. Steps to Create the DT-Overlay Fragment
    1. Clone the device-tree-xlnx repo:
      git clone device-tree-xlnx.git
    2. Helper script: dt_overlay.tcl 
      1. Command:  # xsct dt_overaly.tcl system.hdf psu_cortexa53_0 device-tree-xlnx output_dir

                            Ex: xsct dt_overaly.tcl system.hdf psu_cortexa53_0 ${DTS_REPO}/device-tree-xlnx overlay

      2. Create Device Tree Overlay Blob (.dtbo) file from the pl.dtsi file

                     # dtc -O dtb -o pl.dtbo -b 0 -@ pl.dtsi

                     Ex: ./scripts/dtc/dtc -O dtb -o pl.dtbo -b 0 -@ pl.dtsi

      3. Copy the generated bit and dtbo files to target 

Note: PL nodes should not be the part of Base Device-tree (system.dtb).

FPGA programming 

Exercising FPGA programming using fpgautil 

Source code for the fpgautil is available here fpgautil.c

root@xilinx-zcu102-2018_3:~# fpgautil -h

fpgautil: FPGA Utility for Loading/reading PL Configuration in zynqMP

Usage:  fpgautil -b <bin file path> -o <dtbo file path>

Options: -b <binfile>           (Bin file path)
         -o <dtbofile>          (DTBO file path)
         -f <flags>             Optional: <Bitstream type flags>
                                   f := <Full | Partial > 
                                Default: <Full>
          -s <secure flags>     Optional: <Secure flags>
                                   s := <AuthDDR | AuthOCM | EnUsrKey | EnDevKey | AuthEnUsrKeyDDR | AuthEnUsrK>
          -k <AesKey>           Optional: <AES User Key>
          -r <Readback>         Optional: <file name>
                                Default: By default Read back contents will be stored in readback.bin file
          -t                    Optional: <Readback Type>
                                   0 - Configuration Register readback
                                   1 - Configuration Data Frames readback
                                Default: 0 (Configuration register readback)
          -R                    Optional: Remove overlay from a live tree
 
Examples:
(Load Full Bitstream using Overlay)
fpgautil -b top.bit.bin -o can.dtbo
(Load Partial Bitstream through the sysfs interface)
fpgautil -b top.bit.bin -f Partial 
(Load Authenticated Bitstream through the sysfs interface)
fpgautil -b top.bit.bin -f Full -s AuthDDR 
(Load Parital Encrypted Userkey Bitstream using Overlay)
fpgautil -b top.bit.bin -o pl.dtbo -f Partial -s EnUsrKey -k <32byte key value>
(Read PL Configuration Registers)
fpgautil -b top.bit.bin -r

Loading an overlay along with it's Bitstream


Usage: fpgautil -b <bin file path> -o <overlay file path> 


root@xilinx-zcu102-2018_3:~#
root@xilinx-zcu102-2018_3:~# fpgautil -b /lib/firmware/design_1_wrapper/design_1_wrapper.bit.bin -o /lib/firmwaree/design_1_wrapper/design_1_wrapper.dtbo 
[   43.642776] fpga_manager fpga0: writing design_1_wrapper.bit.bin to Xilinx ZynqMP FPGA Manager
[   43.871149] XGpio: /amba/gpio@a0000000: registered, base is 298
[   43.877420] XGpio: /amba/gpio@a0000000: dual channel registered, base is 290
Time taken to load DTBO is 244.000000 Milli Seconds
DTBO loaded through zynqMP FPGA manager successfully
root@xilinx-zcu102-2018_3:~# 
root@xilinx-zcu102-2018_3:/sys/class/gpio# dmesg |grep gpio
[    5.399518] input: gpio-keys as /devices/platform/gpio-keys/input/input0
[   43.871149] XGpio: /amba/gpio@a0000000: registered, base is 298
[   43.877420] XGpio: /amba/gpio@a0000000: dual channel registered, base is 290
root@xilinx-zcu102-2018_3:/sys/class/gpio# dmesg |grep gpio
[    5.399518] input: gpio-keys as /devices/platform/gpio-keys/input/input0
[   43.871149] XGpio: /amba/gpio@a0000000: registered, base is 298
[   43.877420] XGpio: /amba/gpio@a0000000: dual channel registered, base is 290
root@xilinx-zcu102-2018_3:/sys/class/gpio# echo 290 > export 
root@xilinx-zcu102-2018_3:/sys/class/gpio# cd gpio290
root@xilinx-zcu102-2018_3:/sys/class/gpio/gpio290# ls
active_low  device      direction   edge        power       subsystem   uevent      value
root@xilinx-zcu102-2018_3:/sys/class/gpio/gpio290# echo 1 out > direction 
root@xilinx-zcu102-2018_3:/sys/class/gpio/gpio290# echo 0 > value 
root@xilinx-zcu102-2018_3:/sys/class/gpio/gpio290# echo 1 > value 
root@xilinx-zcu102-2018_3:/sys/class/gpio/gpio290# 

Removing an overlay

Usage: fpgautil -R 

root@xilinx-zcu102-2018_3:~# fpgautil -R 
root@xilinx-zcu102-2018_3:~# 

Loading Bitstream only

Usage: fpgautil -b <bit/bin file absolute path> 

root@xilinx-zcu102-2018_3:~# fpgautil  -b /mnt/design_1_wrapper.bit
Time taken to load BIN is 213.000000 Milli Seconds
BIN FILE loaded through zynqMP FPGA manager successfully

Loading Partial Bitstream that doesn't require PL drivers

Usage: fpgautil -b <bit/bin file full path> -f Partial 

root@xilinx-zcu102-2018_3:~# fpgautil -b pr1rm1.bit.bin -f Partial
Flags : 0x1
Time taken to load BIN is 56.000000 Milli Seconds
BIN FILE loaded through zynqMP FPGA manager successfully

Loading Secure Authenticated Bitstream

Usage: fpgautil -b <bit/bin file full path> -f <flags> -s AuthDDR

root@xilinx-zcu102-2018_3:~# fpgautil -b Authentication_system_wrapper.bin -f Full -s AuthDDR
Flags : 0x0
Flags : 0x2
[  467.885628] fpga_manager fpga0: writing Authentication_system_wrapper.bin to Xilinx ZynqMP FPGA Manager
Time taken to load BIN FILE is 263.000000 Milli Seconds
BIN FILE loaded through zynqMP FPGA manager successfully


Read back of FPGA Configuration Registers

Usage: fpgautil -r 

root@xilinx-zcu102-2018_3:~#fpgautil -r
Readback contents are stored in the file readback.bin
root@xilinx-zcu102-2018_3:~# vi readback.bin 
zynqMP FPGA Configuration register contents are
CRC --> 	 0 	
FAR --> 	 7fc0000 	
FDRI --> 	 0 	
FDRO --> 	 effffffe 	
CMD --> 	 d 	
CTRL0 --> 	 101 	
MASK --> 	 0 	
STAT --> 	 16907ffc 	
LOUT --> 	 0 	
COR0 --> 	 38003fe5 	
MFWR --> 	 0 	
CBC --> 	 0 	
IDCODE --> 	 1484a093 	
AXSS --> 	 0 	
COR1 --> 	 400000 	
WBSTR --> 	 0 	
TIMER --> 	 0 	
BOOTSTS --> 	 1 	
CTRL1 --> 	 0 


Read back of FPGA Configuration Data (Frames)

Usage: fpgautil -r <filename> -t 1

root@xilinx-zcu102-2018_3:~# fpgautil -r readbackdata -t 1
root@xilinx-zcu102-2018_3:~# hexdump readbackdata.bin > data.txt
Verification of Read-back Data

Note:

utility compares the readback with the RBD golden file, by masking readback bits with the MSD file. It is expected from user to remove/strip the header in the msd and rdb files and also remove the print statement in the readback contents file(readback.bin).

Example

$./verify_readback -msd design_1_wrapper.msd -rbd design_1_wrapper.rbd -data reabackdata_2.bin 
=============================================
Starting verification of readback data...
 
>>>> Readback data is correct! <<<<
=============================================

Exercising FPGA programming using fpga framework attributes  

FPGA programming using sysfs attributes 

 Linux FPGA Manager driver creates /sys/class/fpga_manager/<fpga> folder, which contains file attributes that provides control and state. 

Steps for programming the Full Bitstream 

Set flags for Full Bitstream

Load the Bitstream 

Steps for programming the Partial Bitstream

Set flags for Partial Bitstream

Load the Bitstream Partial Bitstream

FPGA Readback using debugfs attributes

Linux FPGA Manager driver creates /sys/kernel/debug/fpga/ folder, which contains image attributes that provides readback contents.

Steps for Readback of Configuration Registers

Set flags for readback type

echo 0 > /sys/module/zynqmp_fpga/parameters/readback_type

Perform Readback operation by reading the image debug attribute 

cat /sys/kernel/debug/fpga/fpga0/image

Example:

root@xilinx-zcu102-2018_3:~# echo 0 > /sys/module/zynqmp_fpga/parameters/readback_type
root@xilinx-zcu102-2018_3:~# cat /sys/kernel/debug/fpga/fpga0/image
zynqMP FPGA Configuration register contents are
CRC -->   0 
FAR -->   7fc0000   
FDRI -->      0 
FDRO -->      effffffe  
CMD -->   d 
CTRL0 -->     101   
MASK -->      0 
STAT -->      16907ffc  
LOUT -->      0 
COR0 -->      38003fe5  
MFWR -->      0 
CBC -->   0 
IDCODE -->    1484a093  
AXSS -->      0 
COR1 -->      400000    
WBSTR -->     0 
TIMER -->     0 
BOOTSTS -->   1 
CTRL1 -->     0 


Steps for Readback of Configuration DataFrames 

Set flags for readback type

echo 1 > /sys/module/zynqmp_fpga/parameters/readback_type

Perform Readback operation by reading the image debug attribute 

cat /sys/kernel/debug/fpga/fpga0/image

Example:

root@xilinx-zcu102-2018_3:~# echo 1 > /sys/module/zynqmp_fpga/parameters/readback_type
root@xilinx-zcu102-2018_3:~# cat /sys/kernel/debug/fpga/fpga0/image > reabackdata.bin
root@xilinx-zcu102-2018_3:~# hexdump readbackdata.bin > data.txt


FPGA programming using Device Tree Overlay (DTO)

The Device Tree Overlay (DTO) is used to reprogram an FPGA while Linux is running. The DTO overlay will add the child node and the fragments from the .dtbo file to the base device tree,, The newly added device node/drivers will be probed after Bitstream programming

DTO contains:

Notes:

  • If the user tries to load the un-aligned bit/bin file the PL configuration takes a longer time when compared with aligned(word-aligned) bit/bin files.
  • Using overlay’s we can add a new node or add/update the existing node properties. But it will not allow replacing the existing nodes which are already part of the live tree. 

Steps for programming the Full Bitstream using overlay

Copy the Full Bitstream (.bin) and pl.dtbo files into firmware folder

Steps to remove the drivers got added as part of DTO


Devicetree overlay file contents example:


//Device Tree Example: Full Reconfiguration along with PS PL Configuration (Clock, resets and AFI)
// HSI Generated overlay/pl.dtsi file.
// Enable the axi-gpio interface
/dts-v1/;
/plugin/;
/ {
	fragment@0 { /* Bitstream  Fragment */ 
		target = <&fpga_full>;
		overlay0: __overlay__ {
			#address-cells = <2>;
			#size-cells = <2>;
			firmware-name = "design_1_wrapper.bit.bin";
			resets = <&rst 116>;
		};
	};
	fragment@1 { /* PS-PL configuration Fragment */
		target = <&amba>;
		overlay1: __overlay__ {
			afi0: afi0 {
				compatible = "xlnx,afi-fpga";
				config-afi = < 0 0>, <1 0>, <2 0>, <3 0>, <4 0>, <5 0>, <6 0>, <7 0>, <8 0>, <9 0>, <10 0>, <11 0>, <12 0>, <13 0>, <14 0>, <14 0>, <15 0x000>;
			};
			clocking0: clocking0 {
				#clock-cells = <0>;
				assigned-clock-rates = <99990005>;
				assigned-clocks = <&clk 71>;
				clock-output-names = "fabric_clk";
				clocks = <&clk 71>;
				compatible = "xlnx,fclk";
			};
		};
	};
	fragment@2 { /* PL Drivers Fragment */
		target = <&amba>;
		overlay2: __overlay__ {
			axi_gpio_0: gpio@a0000000 {
				#gpio-cells = <3>;
				#interrupt-cells = <2>;
				clock-names = "s_axi_aclk";
				clocks = <&clk 71>;
				compatible = "xlnx,axi-gpio-2.0", "xlnx,xps-gpio-1.00.a";
				gpio-controller ;
				interrupt-controller ;
				interrupt-names = "ip2intc_irpt";
				interrupt-parent = <&gic>;
				interrupts = <0 89 4>;
				reg = <0x0 0xa0000000 0x0 0x1000>;
				xlnx,all-inputs = <0x1>;
				xlnx,all-inputs-2 = <0x0>;
				xlnx,all-outputs = <0x0>;
				xlnx,all-outputs-2 = <0x1>;
				xlnx,dout-default = <0x00000000>;
				xlnx,dout-default-2 = <0xAAAAAAAA>;
				xlnx,gpio-width = <0x8>;
				xlnx,gpio2-width = <0x8>;
				xlnx,interrupt-present = <0x1>;
				xlnx,is-dual = <0x1>;
				xlnx,tri-default = <0xFFFFFFFF>;
				xlnx,tri-default-2 = <0xFFFFFFFF>;
			};
			psu_ctrl_ipi: PERIPHERAL@ff380000 {
				/* This is a place holder node for a custom IP, user may need to update the entries */
				compatible = "xlnx,PERIPHERAL-1.0";
				reg = <0x0 0xff380000 0x0 0x80000>;
			};
			psu_message_buffers: PERIPHERAL@ff990000 {
				/* This is a place holder node for a custom IP, user may need to update the entries */
				compatible = "xlnx,PERIPHERAL-1.0";
				reg = <0x0 0xff990000 0x0 0x10000>;
			};
		};
	};
};

Working with Device Tree Overlay (DTBO)

Copy the Partial Bitstream (.bin) and rm.dtbo files into lib/firmware folder

To remove a device tree overlay:

Exercising FPGA programming using Libdfx

Refer this link: libdfx - Linux User Space Solution for FPGA Programming

Mainline Status

The current driver availble in the xilinx linux git is in sync with the open source 4.19 kernel driver except for the following

Release history

2021.2

2021.1

2020.2

Summary:

Commits:

2020.1

2019.2

Summary:

Commits:

2019.1

Summary:

Commits:

2018.3

Summary:

Commits:


2018.2
Summary:

Commits:


2018.1
Summary:

Commits:


2017.4

2017.3
Summary:

Commits:

2017.2


2017.1
Summary:

Commits:


2016.4


2016.3
Summary:

Commits:

See also 

Zynq FPGA Manager

References

https://github.com/Xilinx/linux-xlnx/blob/master/drivers/fpga/zynqmp-fpga.c

https://github.com/Xilinx/linux-xlnx/blob/master/Documentation/fpga/fpga-mgr.txt

https://github.com/Xilinx/linux-xlnx/blob/master/Documentation/devicetree/bindings/fpga/fpga-region.txt

https://github.com/Xilinx/embeddedsw/tree/master/lib/sw_services/xilfpga/doc