Xilinx Linux PL PCIe Root Port

Owned by Confluence Wiki Admin (Unlicensed)
Last updated: Apr 25, 2023 by William Cassells
21 min read


This page gives an overview of Root Port driver for Xilinx XDMA (Bridge mode) IP, when connected to PCIe block in Zynq UltraScale+ MPSoC PL and PL PCIe4 in Versal Adaptive SoC.  The driver is available as part of Xilinx Linux distribution as drivers/pci/controller/pcie-xdma-pl.c

Table of Contents

XDMA IP

The documentation for XDMA IP is available in PG194. It is used in bridge mode for Root port.


Root Port Driver Configuration

The PCI/PCIe subsystem support in ZynqMP kernel configuration. For selecting XDMA PL PCIe root port driver enable CONFIG_PCIE_XDMA_PL option. (The driver file is same for both ZU+ MPSoC PL and Versal PL PCIe4)

ZynqMP XDMA PL PCIe Root Port:

Hardware setup

The hardware setup uses Xilinx ZCU106 hardware platform along with Root port FMC on HPC FMC slot. The design uses XDMA-bridge mode IP with PL-PCIe and targets GTs routed to HPC FMC.

The following features are supported:

  • Reception of legacy & MSI interrupts
  • 64-bit addressing i.e. BARs can be 32-bit or 64-bit on Endpoints

Tested End Points

1. Broadcom PCIe NIC card
2. Intel NVMe SSD
3. Intel NIC card

Standard PetaLinux build flow for Linux image/devicetree generation using HDF from hardware design should be followed.


End Point Driver Configuration


This page demonstrates the Root Port driver using Broadcom NIC Endpoint, for which NIC driver should be enabled in kernel as shown,



For other End point cards, ensure the respective driver is enabled in kernel.

Device Tree binding


The device tree node for AXI PCIe core will be automatically generated, if the core is configured in the HW design, using the Device Tree BSP.

Steps to generate device-tree is documented here,
http://www.wiki.xilinx.com/Build+Device+Tree+Blob

And a sample binding is shown below and the description of DT property is documented here

xdma_0: axi-pcie@a0000000 {
		#address-cells = <3>;
		#interrupt-cells = <1>;
		#size-cells = <2>;
		compatible = "xlnx,xdma-host-3.00";
		device_type = "pci";
		interrupt-map = <0 0 0 1 &pcie_intc_0 1>, <0 0 0 2 &pcie_intc_0 2>, <0 0 0 3 &pcie_intc_0 3>, <0 0 0 4 &pcie_intc_0 4>;
		interrupt-map-mask = <0 0 0 7>;
		interrupt-parent = <&gic>;
		interrupt-names = "misc", "msi0", "msi1";
		interrupts = <0 89 4>, <0 90 4>, <0 91 4>;
		ranges = <0x02000000 0x00000000 0xB0000000 0x0 0xB0000000 0x00000000 0x01000000>,
			<0x43000000 0x00000005 0x00000000 0x00000005 0x00000000 0x00000000 0x01000000>;
		reg = <0x0 0xA0000000 0x0 0x10000000>;
		pcie_intc_0: interrupt-controller {
			#address-cells = <0>;
			#interrupt-cells = <1>;
			interrupt-controller ;
		};
	};

Test Procedure

  • Boot linux onto ZCU106.
  • After successful booting of the linux, the Broadcom NIC endpoint driver is seen to be probed.
  • Run the command ‘lspci’ from the user prompt, which shows the the device id and vendor id of the Broadcom NIC card. This ensures the enumeration of the device is fine.
  • The Broadcom NIC is shown up as an eth0 interface in the linux. Run the command ‘ifconfig eth0 up’ which brings up the Ethernet interface. This step ensures all the memory transactions on the PCIe bus are working. (the ethX assigned may be dependent on other Ethernet ports enabled on your setup)
  • Assign a IP address either static/dhcp.
  • Ping to a device of known IP address.


Kernel Console Output


Driver Initialization for ZynqMP


[    1.568000] xilinx-pcie a0000000.axi-pcie: PCIe Link is UP
[    1.568019] OF: PCI: host bridge /amba_pl@0/axi-pcie@a0000000 ranges:
[    1.568034] OF: PCI:   No bus range found for /amba_pl@0/axi-pcie@a0000000, using [bus 00-ff]
[    1.568060] OF: PCI:   MEM 0xb0000000..0xb0ffffff -> 0xb0000000
[    1.568075] OF: PCI:   MEM 0x500000000..0x500ffffff -> 0x500000000
[    1.568176] xilinx-pcie a0000000.axi-pcie: PCI host bridge to bus 0000:00
[    1.568193] pci_bus 0000:00: root bus resource [bus 00-ff]
[    1.568207] pci_bus 0000:00: root bus resource [mem 0xb0000000-0xb0ffffff]
[    1.568222] pci_bus 0000:00: root bus resource [mem 0x500000000-0x500ffffff pref]
[    1.568257] pci 0000:00:00.0: [10ee:9114] type 01 class 0x060400
[    1.568279] pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0x3fffffff]
[    1.568414] pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    1.568512] pci 0000:01:00.0: [14e4:1677] type 00 class 0x020000
[    1.568552] pci 0000:01:00.0: reg 0x10: [mem 0x986220140a200000-0x986220140a20ffff 64bit]
[    1.568615] pci 0000:01:00.0: reg 0x30: [mem 0x06000000-0x0600ffff pref]
[    1.568716] pci 0000:01:00.0: PME# supported from D3hot
[    1.568835] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01
[    1.568867] pci 0000:00:00.0: BAR 0: no space for [mem size 0x40000000]
[    1.568882] pci 0000:00:00.0: BAR 0: failed to assign [mem size 0x40000000]
[    1.568898] pci 0000:00:00.0: BAR 8: assigned [mem 0xb0000000-0xb00fffff]
[    1.568915] pci 0000:01:00.0: BAR 0: assigned [mem 0xb0000000-0xb000ffff 64bit]
[    1.568949] pci 0000:01:00.0: BAR 6: assigned [mem 0xb0010000-0xb001ffff pref]
[    1.568969] pci 0000:00:00.0: PCI bridge to [bus 01]
[    1.568984] pci 0000:00:00.0:   bridge window [mem 0xb0000000-0xb00fffff]
[    1.569773] xilinx-dpdma fd4c0000.dma: Xilinx DPDMA engine is probed
[    1.570058] Write failed gate address:1000f02
[    1.570150] xilinx-zynqmp-dma fd500000.dma: ZynqMP DMA driver Probe success
[    1.570276] xilinx-zynqmp-dma fd510000.dma: ZynqMP DMA driver Probe success
[    1.570401] xilinx-zynqmp-dma fd520000.dma: ZynqMP DMA driver Probe success
[    1.570525] xilinx-zynqmp-dma fd530000.dma: ZynqMP DMA driver Probe success
[    1.570651] xilinx-zynqmp-dma fd540000.dma: ZynqMP DMA driver Probe success
[    1.570777] xilinx-zynqmp-dma fd550000.dma: ZynqMP DMA driver Probe success
[    1.570903] xilinx-zynqmp-dma fd560000.dma: ZynqMP DMA driver Probe success
[    1.571028] xilinx-zynqmp-dma fd570000.dma: ZynqMP DMA driver Probe success
[    1.571227] xilinx-zynqmp-dma ffa80000.dma: ZynqMP DMA driver Probe success
[    1.571354] xilinx-zynqmp-dma ffa90000.dma: ZynqMP DMA driver Probe success
[    1.571482] xilinx-zynqmp-dma ffaa0000.dma: ZynqMP DMA driver Probe success
[    1.571625] xilinx-zynqmp-dma ffab0000.dma: ZynqMP DMA driver Probe success
[    1.571754] xilinx-zynqmp-dma ffac0000.dma: ZynqMP DMA driver Probe success
[    1.571881] xilinx-zynqmp-dma ffad0000.dma: ZynqMP DMA driver Probe success
[    1.572013] xilinx-zynqmp-dma ffae0000.dma: ZynqMP DMA driver Probe success
[    1.572139] xilinx-zynqmp-dma ffaf0000.dma: ZynqMP DMA driver Probe success
[    1.572304] zynqmp_pm firmware: Power management API v0.3
[    1.572413] xenfs: not registering filesystem on non-xen platform
[    1.598776] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
[    1.600017] ff000000.serial: ttyPS0 at MMIO 0xff000000 (irq = 41, base_baud = 6249375) is a xuartps
[    2.794394] console [ttyPS0] enabled
[    2.798278] ff010000.serial: ttyPS1 at MMIO 0xff010000 (irq = 42, base_baud = 6249375) is a xuartps
[    2.807433] [drm] Initialized
[    2.810641] [drm] load() is defered & will be called again
[    2.816506] xilinx-drm-dp-sub fd4aa000.dp_sub: Xilinx DisplayPort Subsystem is probed
[    2.824414] Unable to detect cache hierarchy from DT for CPU 0
[    2.834635] brd: module loaded
[    2.840268] loop: module loaded
[    2.854238] ahci-ceva fd0c0000.ahci: AHCI 0001.0301 32 slots 2 ports 6 Gbps 0x3 impl platform mode
[    2.863137] ahci-ceva fd0c0000.ahci: flags: 64bit ncq sntf pm clo only pmp fbs pio slum part ccc sds apst
[    2.873538] scsi host0: ahci-ceva
[    2.876965] scsi host1: ahci-ceva
[    2.880325] ata1: SATA max UDMA/133 mmio [mem 0xfd0c0000-0xfd0c1fff] port 0x100 irq 39
[    2.888173] ata2: SATA max UDMA/133 mmio [mem 0xfd0c0000-0xfd0c1fff] port 0x180 irq 39
[    2.896194] mtdoops: mtd device (mtddev=name/number) must be supplied
[    2.903439] m25p80 spi0.0: n25q512a (131072 Kbytes)
[    2.908248] 3 ofpart partitions found on MTD device spi0.0
[    2.913703] Creating 3 MTD partitions on "spi0.0":
[    2.918479] 0x000000000000-0x000000100000 : "boot"
[    2.923754] 0x000000100000-0x000000140000 : "bootenv"
[    2.929163] 0x000000140000-0x000001740000 : "kernel"
[    2.935211] libphy: Fixed MDIO Bus: probed
[    2.940201] tun: Universal TUN/TAP device driver, 1.6
[    2.945184] tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
[    2.951458] CAN device driver interface
[    2.956159] macb ff0e0000.ethernet: Not enabling partial store and forward
[    2.963345] libphy: MACB_mii_bus: probed
[    2.971086] macb ff0e0000.ethernet eth0: Could not attach to PHY
[    3.011805] tg3.c:v3.137 (May 11, 2014)
[    3.015577] pci 0000:00:00.0: enabling device (0000 -> 0002)

Broadcom NIC card probing

[    3.021218] tg3 0000:01:00.0: enabling device (0000 -> 0002)
[    3.058190] tg3 0000:01:00.0 eth0: Tigon3 [partno(BCM95751A519) rev 4101] (PCI Express) MAC address 00:07:13:04:3d:18
[    3.068733] tg3 0000:01:00.0 eth0: attached PHY is 5750 (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[0])
[    3.078450] tg3 0000:01:00.0 eth0: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] TSOcap[1]
[    3.086261] tg3 0000:01:00.0 eth0: dma_rwctrl[76180000] dma_mask[64-bit]


Ethernet Interface

root@:~# ifconfig -a
eth0      Link encap:Ethernet  HWaddr 00:07:13:04:3D:18
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
          Interrupt:48


Testing of Ethernet interface

root@:~# ifconfig eth0 192.168.1.4
tg3 0000:01:00.0 eth0: Link is up at 1000 Mbps, full duplex
tg3 0000:01:00.0 eth0: Flow control is on for TX and on for RX
IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
 
root@:~# ping 192.168.1.2
PING 192.168.1.2 (192.168.1.2): 56 data bytes
64 bytes from 192.168.1.2: seq=0 ttl=128 time=1.502 ms
64 bytes from 192.168.1.2: seq=1 ttl=128 time=1.131 ms
64 bytes from 192.168.1.2: seq=2 ttl=128 time=1.305 ms
64 bytes from 192.168.1.2: seq=3 ttl=128 time=0.699 ms
64 bytes from 192.168.1.2: seq=4 ttl=128 time=0.762 ms
64 bytes from 192.168.1.2: seq=5 ttl=128 time=0.736 ms
64 bytes from 192.168.1.2: seq=6 ttl=128 time=1.096 ms
64 bytes from 192.168.1.2: seq=7 ttl=128 time=0.680 ms
64 bytes from 192.168.1.2: seq=8 ttl=128 time=0.783 ms
64 bytes from 192.168.1.2: seq=9 ttl=128 time=0.885 ms
--- 192.168.1.2 ping statistics ---
10 packets transmitted, 10 packets received, 0% packet loss

Interrupts

root@:~# cat /proc/interrupts
           CPU0       CPU1       CPU2       CPU3
  2:          0          0          0          0     GICv2  29 Level     arch_timer
  3:      17031       4305       1793       2036     GICv2  30 Level     arch_timer
 10:          0          0          0          0     GICv2  67 Level     zynqmp_pm
 13:          0          0          0          0     GICv2 156 Level     zynqmp-dma
 14:          0          0          0          0     GICv2 157 Level     zynqmp-dma
 15:          0          0          0          0     GICv2 158 Level     zynqmp-dma
 16:          0          0          0          0     GICv2 159 Level     zynqmp-dma
 17:          0          0          0          0     GICv2 160 Level     zynqmp-dma
 18:          0          0          0          0     GICv2 161 Level     zynqmp-dma
 19:          0          0          0          0     GICv2 162 Level     zynqmp-dma
 20:          0          0          0          0     GICv2 163 Level     zynqmp-dma
 22:          0          0          0          0     GICv2 109 Level     zynqmp-dma
 23:          0          0          0          0     GICv2 110 Level     zynqmp-dma
 24:          0          0          0          0     GICv2 111 Level     zynqmp-dma
 25:          0          0          0          0     GICv2 112 Level     zynqmp-dma
 26:          0          0          0          0     GICv2 113 Level     zynqmp-dma
 27:          0          0          0          0     GICv2 114 Level     zynqmp-dma
 28:          0          0          0          0     GICv2 115 Level     zynqmp-dma
 29:          0          0          0          0     GICv2 116 Level     zynqmp-dma
 33:          0          0          0          0     GICv2  49 Level     cdns-i2c
 34:          0          0          0          0     GICv2  50 Level     cdns-i2c
 35:          0          0          0          0     GICv2  42 Level     ff960000.memory-controller
 36:         14          0          0          0     GICv2  47 Level     ff0f0000.spi
 37:          0          0          0          0     GICv2  58 Level     ffa60000.rtc
 38:          0          0          0          0     GICv2  59 Level     ffa60000.rtc
 39:          0          0          0          0     GICv2 165 Level     ahci-ceva[fd0c0000.ahci]
 40:        144          0          0          0     GICv2  81 Level     mmc0
 41:        891          0          0          0     GICv2  53 Level     xuartps
 43:          0          0          0          0     GICv2 145 Edge      fd4d0000.watchdog
 44:          0          0          0          0     GICv2  88 Level     ams-irq
 45:          0          0          0          0     GICv2 151 Level     fd4a0000.dp
 46:          0          0          0          0     GICv2 154 Level     fd4c0000.dma
 47:        647          0          0          0     GICv2 121 Level     xilinx-pcie
 48:        647          0          0          0     dummy   1 Level     eth0
223:          0          0          0          0     GICv2  97 Level     xhci-hcd:usb1
IPI0:      1954        759       1664       2660       Rescheduling interrupts
IPI1:        63         57         71         43       Function call interrupts
IPI2:         0          0          0          0       CPU stop interrupts
IPI3:       349       2284       2398       2370       Timer broadcast interrupts
IPI4:         3          0          0          0       IRQ work interrupts
IPI5:         0          0          0          0       CPU wake-up interrupts
Err:          0




QDMA IP

The documentation for QDMA IP is available in PG344. It is used in bridge mode for Root port.


Root Port Driver Configuration

The PCI/PCIe subsystem support in Versal kernel configuration. For selecting QDMA PL PCIe root port driver enable CONFIG_PCIE_XDMA_PL option.

Versal QDMA PL PCIe4 Root Port:

Please refer AR76647 to add QDMA related driver patch and sample device tree.

Hardware setup

The design uses QDMA-bridge mode IP with Versal PL-PCIe4.

The following features are supported:

  • Reception of legacy & MSI interrupts
  • 32-bit BAR support

Tested End Points

1. Broadcom PCIe NIC card
2. Samsung 970 Evo SSD
3. Samsung 980 EVO SSD
4. Intel SSD

Standard Petalinux build flow for Linux image/devicetree generation using XSA from hardware design should be followed.


End Point Driver Configuration


The following driver is needed to be enabled for using NVMe SSD as EndPoint.



For other End point cards, ensure the respective driver is enabled in kernel.

Device Tree binding


And a sample binding is shown below and the description of DT property is documented here

     pcie_dma_versal_0: axi-pcie@80000000 {
                        #address-cells = <3>;
                        #interrupt-cells = <1>;
                        #size-cells = <2>;
                        compatible = "xlnx,qdma-host-3.00";
                        device_type = "pci";
                        interrupt-map = <0 0 0 1 &psv_pcie_intc_0 1>, <0 0 0 2 &psv_pcie_intc_0 2>, <0 0 0 3 &psv_pcie_intc_0 3>, <0 0 0 4 &psv_pcie_intc_0 4>;
                        interrupt-map-mask = <0 0 0 7>;
                        interrupt-names = "misc", "msi0", "msi1";
                        interrupt-parent = <&gic>;
                        interrupts = <0 84 4 0 85 4 0 86 4>;
                        ranges = <0x02000000 0x00000000 0xA8000000 0x0 0xA8000000 0x00000000 0x1000000>;
                        reg = <0x0 0x90000000 0x0 0x5000>,
                                <0x0 0x80000000 0x0 0x1000000>;
                        reg-names = "breg", "cfg";

                        psv_pcie_intc_0: interrupt-controller {
                                #address-cells = <0>;
                                #interrupt-cells = <1>;
                                interrupt-controller ;
                        };
                };

Test Procedure

  • Boot linux onto Versal.
  • Check lspci to confirm NVMe card is detected and do the following steps.
    • mount /dev/nvmeXX  /mnt
    • cd /mnt
    • dd if=/dev/zero of=tmp.txt bs=4096 count=200000

Kernel Console Output


Driver Initialization for Versal


[ 3.122832] io scheduler mq-deadline registered
[ 3.127401] io scheduler kyber registered
[ 3.131713] xilinx-xdma-pcie 90000000.axi-pcie: host bridge /amba_pl@0/axi-pcie@80000000 ranges:
[ 3.140586] xilinx-xdma-pcie 90000000.axi-pcie: No bus range found for /amba_pl@0/axi-pcie@80000000, using [bus 00-ff]
[ 3.151569] xilinx-xdma-pcie 90000000.axi-pcie: MEM 0x00a8000000..0x00a80fffff -> 0x00a8000000
[ 3.160775] xilinx-xdma-pcie 90000000.axi-pcie: PCIe Link is UP
[ 3.166841] xilinx-xdma-pcie 90000000.axi-pcie: PCI host bridge to bus 0000:00
[ 3.174134] pci_bus 0000:00: root bus resource [bus 00-ff]
[ 3.179668] pci_bus 0000:00: root bus resource [mem 0xa8000000-0xa80fffff]
[ 3.186620] pci 0000:00:00.0: [10ee:b048] type 01 class 0x060400
[ 3.192688] pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0xffffffff 64bit pref]
[ 3.200006] pci 0000:00:00.0: PME# supported from D3cold
[ 3.205838] pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 3.213973] pci 0000:01:00.0: [144d:a808] type 00 class 0x010802
[ 3.220054] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x00003fff 64bit]
[ 3.227467] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01
[ 3.234150] pci 0000:00:00.0: BAR 0: no space for [mem size 0x100000000 64bit pref]
[ 3.241877] pci 0000:00:00.0: BAR 0: failed to assign [mem size 0x100000000 64bit pref]
[ 3.249949] pci 0000:00:00.0: BAR 8: assigned [mem 0xa8000000-0xa80fffff]
[ 3.256794] pci 0000:01:00.0: BAR 0: assigned [mem 0xa8000000-0xa8003fff 64bit]
[ 3.264179] pci 0000:00:00.0: PCI bridge to [bus 01]
[ 3.269189] pci 0000:00:00.0: bridge window [mem 0xa8000000-0xa80fffff]
[ 3.276445] ps_pcie_dma init()
[ 3.295314] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
[ 3.302325] Serial: AMBA driver
[ 3.306196] cacheinfo: Unable to detect cache hierarchy for CPU 0

NVMe card probing

[ 4.709047] nvme nvme0: pci function 0000:01:00.0
[ 4.710192] pci 0000:00:00.0: enabling device (0000 -> 0002)
[ 4.715796] nvme 0000:01:00.0: enabling device (0000 -> 0002)


Testing of NVMe Access

root@:~# mount /dev/nvme0n1  /mnt
cd /mnt
dd if=/dev/zero of=tmp.txt bs=4096 count=200000

Interrupts

root@:/mnt# cat /proc/interrupts 
           CPU0       CPU1       
 11:       8828      22802     GICv3  30 Level     arch_timer
 14:          0          0     GICv3  92 Level     zynqmp-dma
 15:          0          0     GICv3  93 Level     zynqmp-dma
 16:          0          0     GICv3  94 Level     zynqmp-dma
 17:          0          0     GICv3  95 Level     zynqmp-dma
 18:          0          0     GICv3  96 Level     zynqmp-dma
 19:          0          0     GICv3  97 Level     zynqmp-dma
 20:          0          0     GICv3  98 Level     zynqmp-dma
 21:          0          0     GICv3  99 Level     zynqmp-dma
 23:          0          0     GICv3 174 Level     f12a0000.rtc
 24:          0          0     GICv3 175 Level     f12a0000.rtc
 25:       2428          0     GICv3  50 Level     uart-pl011
 27:          0          0     GICv3 176 Level     sysmon-irq
 28:          0          0     GICv3  62 Level     zynqmp_ipi
 29:          0          0     GICv3 116 Level     xilinx-pcie
 34:       4786          0  xilinx_pcie:msi 524288 Edge      nvme0q0, nvme0q1
IPI0:       211        247       Rescheduling interrupts
IPI1:      1080       2723       Function call interrupts
IPI2:         0          0       CPU stop interrupts
IPI3:         0          0       CPU stop (for crash dump) interrupts
IPI4:         0          0       Timer broadcast interrupts
IPI5:         0          0       IRQ work interrupts
IPI6:         0          0       CPU wake-up interrupts
Err:          0


Related Links


© Copyright 2019 - 2022 Xilinx Inc. Privacy Policy