XAPP1289 PCIe Root DMA
This page provides supplemental information for XAP1289
Table of Contents
Introduction
The Zynq UltraScale+ Controller for PCI Express has a built-in DMA engine that can be used in Endpoint as well as Root Port mode.
This application note provides an example that demonstrates how to configure and use the DMA in the Controller for PCI Express when configured as a Root Port.
This wiki provides instructions on how to build the design - hardware and software components and test them.
This requires Vivado 2016.1 and PetaLinux 2016.1 and ZCU102 board and a KCU105 card to be used as an Endpoint.
Design is available here
Directory Structure
Note: No hardware design is provided as it requires ZCU102 board support files available on the MPSoC lounge. Steps to build hardware after using the board preset template are provided below.
This requires ZCU102 board support files which can be obtained from Zynq UltraScale+ MPSoC lounge.
Note: No build steps or source code is provided for the KCU105 Endpoint design. It uses Xilinx IP- AXI Bridge for PCIe Gen3 with AXI interface connect to DDR4 (via MIG-IP). AXI Performance monitor is added on this AXI interface.
Address translations are set as
$XAPP1289 is used to refer to the path where you copied the design zip file
$PETALINUX_PROJECT_PATH is used to refer to the path of your PetaLinux project.
Important Note
The PS-DDR APM on ZCU102 needs to be brought out of reset for the Root DMA drivers to poll APM for performance. This can be achieved via two options:
//- Bring PS-DDR APM out of Reset
XFsbl_Out32(0xFD1A0108, 0x3);
6. Device-tree update: Add the following to $PETALINUX_PROJECT_PATH/Xilinx-ZCU102-2016.1/subsystem/linux/configs/device-tree/zynqmp.dtsi
8. Create BIN file for SD boot using the BIF file provided under $XAPP1289/software/util folder. Ensure you have XSDK environment setup for bootgen to work. Copy this bif file under $PETALINUX_PROJECT_PATH/images/linux
Say $KERNEL_BUILD_PATH (based on PetaLinux build above is)- $PETALINUX_PROJECT_PATH/build/linux/kernel/linux-xlnx
Ensure that you have the Petalinux build environment settings done and define CROSS_COMPILE as aarch64-linux-gnu-
(This is also added in the Makefile provided)
Password : root
6. Mount the SD card to /mnt and navigate to the SD mount area and insert the drivers.
7. Run the test to see performance
The MPS is 128B with the AXI-PCIe Bridge Burst size on MPSoC configured as 128B.
DMA Channel-0 and channel-2 for System to Card traffic (Root DMA pushing data into Endpoint)
This application note provides an example that demonstrates how to configure and use the DMA in the Controller for PCI Express when configured as a Root Port.
This wiki provides instructions on how to build the design - hardware and software components and test them.
This requires Vivado 2016.1 and PetaLinux 2016.1 and ZCU102 board and a KCU105 card to be used as an Endpoint.
Design Build Instructions
This section captures hardware and software component build steps.Design is available here
Directory Structure
- ready_to_test folder contains prebuilt SD image binaries for the design
- software/patch contains the patch required to be applied to the root port driver in kernel. This patch sets up the DMA register aperture (DREG) and enables the DMA interrupts in AXI domain from AXI-PCIe bridge
- software/hw_export contains the exported HDF
- software/root_dma contains the source code for the Root DMA driver and application
Note: No hardware design is provided as it requires ZCU102 board support files available on the MPSoC lounge. Steps to build hardware after using the board preset template are provided below.
Hardware Design
This design configures the controller for PCI Express in Root mode with x4 Gen2 Link. The following provides step by step instructions with Vivado PCW snapshot.This requires ZCU102 board support files which can be obtained from Zynq UltraScale+ MPSoC lounge.
- Create Vivado project with ZCU102 board support template. Create a block design, add Zynq UltraScale+ MPSoC IP.
- Click on the "Run Block Automation" ribbon and let it apply board preset settings. Once the ZCU102 preset settings are done, make changes as below for x4 Gen2 PCIe root port.
- Double click on Zynq UltraScale+ MPSoC to customize the configuration.
- On the IO Configuration page, disable USB 3.0 and SATA and enable PCIe for 4 lanes.
- On the Clock configuration page, select 100MHz reference clock for all 4 GTR lanes.
- On the PS-PL Configuration page, disable HPM-LPD master interface as it is not being used here.
- On Advanced configuration page, set the reference clocks to REFCLK0 as shown
- On PCIe Configuration page, change the Link width and speed to reflect x4 Gen2. Modify device ID to 0xD024 and rest of the settings can be retained as it.
- This completes PCW customization for Root mode.
- Save the block design and validate it.
- Create HDL wrapper and Export Hardware to get HDF file which can be consumed by PetaLinux for building the various components.
Note: No build steps or source code is provided for the KCU105 Endpoint design. It uses Xilinx IP- AXI Bridge for PCIe Gen3 with AXI interface connect to DDR4 (via MIG-IP). AXI Performance monitor is added on this AXI interface.
Address translations are set as
- BAR2 to MIG-DDR4 (used for Root DMA transfers)
- BAR4 to APM registers (used for performance measurement)
Software Build
Follow the instructions below to create the software binaries.$XAPP1289 is used to refer to the path where you copied the design zip file
$PETALINUX_PROJECT_PATH is used to refer to the path of your PetaLinux project.
Important Note
The PS-DDR APM on ZCU102 needs to be brought out of reset for the Root DMA drivers to poll APM for performance. This can be achieved via two options:
- Add the code as FSBL hook in xfsbl_hooks.c. The following can be added under XFsbl_HookBeforeHandofffunction.
//- Bring PS-DDR APM out of Reset
XFsbl_Out32(0xFD1A0108, 0x3);
- Alternatively, the same can be done via Linux before inserting Root DMA drivers using devmem
Building Kernel
- Create project using ZCU102 BSP from PetaLinux. Copy the patch to the cloned kernel folder.
- 2. Clone kernel and apply patch provided as shown above.
- 3. Configure PetaLinux to use the cloned kernel
- 4. Import HDF exported from Vivado
- Enable pciutils as described here- http://www.wiki.xilinx.com/ZynqMP+Linux+PCIe+Root+Port#Rootfs%20Configuration
- Enable libgcc-s1: Filesystem Packages --> base --> libgcc-xilinx -->libgcc-s1
6. Device-tree update: Add the following to $PETALINUX_PROJECT_PATH/Xilinx-ZCU102-2016.1/subsystem/linux/configs/device-tree/zynqmp.dtsi
8. Create BIN file for SD boot using the BIF file provided under $XAPP1289/software/util folder. Ensure you have XSDK environment setup for bootgen to work. Copy this bif file under $PETALINUX_PROJECT_PATH/images/linux
Building Root DMA Drivers
This design uses loadable kernel modules and this section describes how to build the Root DMA driver modules and associated application.Say $KERNEL_BUILD_PATH (based on PetaLinux build above is)- $PETALINUX_PROJECT_PATH/build/linux/kernel/linux-xlnx
Ensure that you have the Petalinux build environment settings done and define CROSS_COMPILE as aarch64-linux-gnu-
(This is also added in the Makefile provided)
- Modify the KDIR path in the Makefile.variable file provided ($XAPP1289/software/root_dma/) to reflect $KERNEL_BUILD_PATH
- Navigate to $XAPP1289/software/root_dma and build the kernel modules and application by using 'make'
- This should build the following
- xdma/root_dma_pcie.ko
- Appdriver/XRaw_Data0.ko
- Appdriver/XRaw_Data1.ko
- Navigate to $XAPP1289/software/root_dma/root_dma_test_app/ and build the application by using 'make'. This builds Root_dma_test_App
Setup Details
The XAPP uses ZCU102 as Root port and KCU105 as Endpoint. Bitstream for Endpoint design is provided (source code for this design is not provided).- Connect the setup as shown below.
Test Instructions
Things to know before running the design:- The SD-MMC card has to be formatted as FAT32 using a SD-MMC card reader.
- To use the prebuilt binaries provided, copy the entire folder content from $XAPP1289/ready_to_test/zcu102_sd onto the primary partition of the SD-MMC.
- Alternatively, you can use the images built based on instructions above and copy the following to SD card-
- boot.bin created above
- image.ub (from $PETALINUX_PROJECT_PATH/images/linux/
- Kernel modules and application executable from $XAPP1289/software/root_dma
- Petalinux console login details
Password : root
- Connect the setup as shown above.
- Set the ZCU102 to boot from SD (SW6 set to 0101 configuration), connect USB-UART cable. Setup teraterm on laptop for 115200 baud rate to see UART prints.
- Power on the KCU105 Endpoint card and program it with the bitfile provided $XAPP1289/ready_to_test/kcu105_axi_pcie.bit
- Then power on the ZCU102 and let Linux boot (prints seen on UART terminal)
- Once Linux boots, log in and do 'lspci' to list the PCI devices. It would show the PCIe root port on MPSoC and KCU105 Endpoint. Look at Link Status to check if it linked at x4 Gen2 configuration.
6. Mount the SD card to /mnt and navigate to the SD mount area and insert the drivers.
7. Run the test to see performance
The MPS is 128B with the AXI-PCIe Bridge Burst size on MPSoC configured as 128B.
DMA Channel-0 and channel-2 for System to Card traffic (Root DMA pushing data into Endpoint)
Troubleshooting
- 'lspci' does not show up the Endpoint Card
- Ensure the Endpoint card has fit in properly on the PCIe slot on ZCU102
- Check if the Endpoint KCU105 board has been programmed with the provided bitfile
- On ZCU102 Linux prompt, read the PCIe status by doing 'devmem 0xFD480228' if bit[0] is set then link is up.
- If you find link is up via PCIe Status register but do not see the device in 'lspci'
- Check if you programmed Endpoint card before ZCU102 booted
- Try doing a rescan of PCIe bus by the following command on ZCU102 Linux prompt