Xilinx Linux PL PCIe Root Port
- Confluence Wiki Admin (Unlicensed)
- William Cassells
- Terry O'Neal
Owned by Confluence Wiki Admin (Unlicensed)
This page gives an overview of Root Port driver for Xilinx XDMA (Bridge mode) IP, when connected to PCIe block in Zynq UltraScale+ MPSoC PL and PL PCIe4 in Versal Adaptive SoC. The driver is available as part of Xilinx Linux distribution as drivers/pci/controller/pcie-xdma-pl.c
Table of Contents
XDMA IP
The documentation for XDMA IP is available in PG194. It is used in bridge mode for Root port.
Root Port Driver Configuration
The PCI/PCIe subsystem support in ZynqMP kernel configuration. For selecting XDMA PL PCIe root port driver enable CONFIG_PCIE_XDMA_PL option. (The driver file is same for both ZU+ MPSoC PL and Versal PL PCIe4)
ZynqMP XDMA PL PCIe Root Port:
Hardware setup
The hardware setup uses Xilinx ZCU106 hardware platform along with Root port FMC on HPC FMC slot. The design uses XDMA-bridge mode IP with PL-PCIe and targets GTs routed to HPC FMC.
The following features are supported:
- Reception of legacy & MSI interrupts
- 64-bit addressing i.e. BARs can be 32-bit or 64-bit on Endpoints
Tested End Points
1. Broadcom PCIe NIC card
2. Intel NVMe SSD
3. Intel NIC card
Standard PetaLinux build flow for Linux image/devicetree generation using HDF from hardware design should be followed.
End Point Driver Configuration
This page demonstrates the Root Port driver using Broadcom NIC Endpoint, for which NIC driver should be enabled in kernel as shown,
For other End point cards, ensure the respective driver is enabled in kernel.
Device Tree binding
The device tree node for AXI PCIe core will be automatically generated, if the core is configured in the HW design, using the Device Tree BSP.
Steps to generate device-tree is documented here,
http://www.wiki.xilinx.com/Build+Device+Tree+Blob
And a sample binding is shown below and the description of DT property is documented here
xdma_0: axi-pcie@a0000000 {
#address-cells = <3>;
#interrupt-cells = <1>;
#size-cells = <2>;
compatible = "xlnx,xdma-host-3.00";
device_type = "pci";
interrupt-map = <0 0 0 1 &pcie_intc_0 1>, <0 0 0 2 &pcie_intc_0 2>, <0 0 0 3 &pcie_intc_0 3>, <0 0 0 4 &pcie_intc_0 4>;
interrupt-map-mask = <0 0 0 7>;
interrupt-parent = <&gic>;
interrupt-names = "misc", "msi0", "msi1";
interrupts = <0 89 4>, <0 90 4>, <0 91 4>;
ranges = <0x02000000 0x00000000 0xB0000000 0x0 0xB0000000 0x00000000 0x01000000>,
<0x43000000 0x00000005 0x00000000 0x00000005 0x00000000 0x00000000 0x01000000>;
reg = <0x0 0xA0000000 0x0 0x10000000>;
pcie_intc_0: interrupt-controller {
#address-cells = <0>;
#interrupt-cells = <1>;
interrupt-controller ;
};
};
Test Procedure
- Boot linux onto ZCU106.
- After successful booting of the linux, the Broadcom NIC endpoint driver is seen to be probed.
- Run the command ‘lspci’ from the user prompt, which shows the the device id and vendor id of the Broadcom NIC card. This ensures the enumeration of the device is fine.
- The Broadcom NIC is shown up as an eth0 interface in the linux. Run the command ‘ifconfig eth0 up’ which brings up the Ethernet interface. This step ensures all the memory transactions on the PCIe bus are working. (the ethX assigned may be dependent on other Ethernet ports enabled on your setup)
- Assign a IP address either static/dhcp.
- Ping to a device of known IP address.
Kernel Console Output
Driver Initialization for ZynqMP
[ 1.568000] xilinx-pcie a0000000.axi-pcie: PCIe Link is UP [ 1.568019] OF: PCI: host bridge /amba_pl@0/axi-pcie@a0000000 ranges: [ 1.568034] OF: PCI: No bus range found for /amba_pl@0/axi-pcie@a0000000, using [bus 00-ff] [ 1.568060] OF: PCI: MEM 0xb0000000..0xb0ffffff -> 0xb0000000 [ 1.568075] OF: PCI: MEM 0x500000000..0x500ffffff -> 0x500000000 [ 1.568176] xilinx-pcie a0000000.axi-pcie: PCI host bridge to bus 0000:00 [ 1.568193] pci_bus 0000:00: root bus resource [bus 00-ff] [ 1.568207] pci_bus 0000:00: root bus resource [mem 0xb0000000-0xb0ffffff] [ 1.568222] pci_bus 0000:00: root bus resource [mem 0x500000000-0x500ffffff pref] [ 1.568257] pci 0000:00:00.0: [10ee:9114] type 01 class 0x060400 [ 1.568279] pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0x3fffffff] [ 1.568414] pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring [ 1.568512] pci 0000:01:00.0: [14e4:1677] type 00 class 0x020000 [ 1.568552] pci 0000:01:00.0: reg 0x10: [mem 0x986220140a200000-0x986220140a20ffff 64bit] [ 1.568615] pci 0000:01:00.0: reg 0x30: [mem 0x06000000-0x0600ffff pref] [ 1.568716] pci 0000:01:00.0: PME# supported from D3hot [ 1.568835] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01 [ 1.568867] pci 0000:00:00.0: BAR 0: no space for [mem size 0x40000000] [ 1.568882] pci 0000:00:00.0: BAR 0: failed to assign [mem size 0x40000000] [ 1.568898] pci 0000:00:00.0: BAR 8: assigned [mem 0xb0000000-0xb00fffff] [ 1.568915] pci 0000:01:00.0: BAR 0: assigned [mem 0xb0000000-0xb000ffff 64bit] [ 1.568949] pci 0000:01:00.0: BAR 6: assigned [mem 0xb0010000-0xb001ffff pref] [ 1.568969] pci 0000:00:00.0: PCI bridge to [bus 01] [ 1.568984] pci 0000:00:00.0: bridge window [mem 0xb0000000-0xb00fffff] [ 1.569773] xilinx-dpdma fd4c0000.dma: Xilinx DPDMA engine is probed [ 1.570058] Write failed gate address:1000f02 [ 1.570150] xilinx-zynqmp-dma fd500000.dma: ZynqMP DMA driver Probe success [ 1.570276] xilinx-zynqmp-dma fd510000.dma: ZynqMP DMA driver Probe success [ 1.570401] xilinx-zynqmp-dma fd520000.dma: ZynqMP DMA driver Probe success [ 1.570525] xilinx-zynqmp-dma fd530000.dma: ZynqMP DMA driver Probe success [ 1.570651] xilinx-zynqmp-dma fd540000.dma: ZynqMP DMA driver Probe success [ 1.570777] xilinx-zynqmp-dma fd550000.dma: ZynqMP DMA driver Probe success [ 1.570903] xilinx-zynqmp-dma fd560000.dma: ZynqMP DMA driver Probe success [ 1.571028] xilinx-zynqmp-dma fd570000.dma: ZynqMP DMA driver Probe success [ 1.571227] xilinx-zynqmp-dma ffa80000.dma: ZynqMP DMA driver Probe success [ 1.571354] xilinx-zynqmp-dma ffa90000.dma: ZynqMP DMA driver Probe success [ 1.571482] xilinx-zynqmp-dma ffaa0000.dma: ZynqMP DMA driver Probe success [ 1.571625] xilinx-zynqmp-dma ffab0000.dma: ZynqMP DMA driver Probe success [ 1.571754] xilinx-zynqmp-dma ffac0000.dma: ZynqMP DMA driver Probe success [ 1.571881] xilinx-zynqmp-dma ffad0000.dma: ZynqMP DMA driver Probe success [ 1.572013] xilinx-zynqmp-dma ffae0000.dma: ZynqMP DMA driver Probe success [ 1.572139] xilinx-zynqmp-dma ffaf0000.dma: ZynqMP DMA driver Probe success [ 1.572304] zynqmp_pm firmware: Power management API v0.3 [ 1.572413] xenfs: not registering filesystem on non-xen platform [ 1.598776] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled [ 1.600017] ff000000.serial: ttyPS0 at MMIO 0xff000000 (irq = 41, base_baud = 6249375) is a xuartps [ 2.794394] console [ttyPS0] enabled [ 2.798278] ff010000.serial: ttyPS1 at MMIO 0xff010000 (irq = 42, base_baud = 6249375) is a xuartps [ 2.807433] [drm] Initialized [ 2.810641] [drm] load() is defered & will be called again [ 2.816506] xilinx-drm-dp-sub fd4aa000.dp_sub: Xilinx DisplayPort Subsystem is probed [ 2.824414] Unable to detect cache hierarchy from DT for CPU 0 [ 2.834635] brd: module loaded [ 2.840268] loop: module loaded [ 2.854238] ahci-ceva fd0c0000.ahci: AHCI 0001.0301 32 slots 2 ports 6 Gbps 0x3 impl platform mode [ 2.863137] ahci-ceva fd0c0000.ahci: flags: 64bit ncq sntf pm clo only pmp fbs pio slum part ccc sds apst [ 2.873538] scsi host0: ahci-ceva [ 2.876965] scsi host1: ahci-ceva [ 2.880325] ata1: SATA max UDMA/133 mmio [mem 0xfd0c0000-0xfd0c1fff] port 0x100 irq 39 [ 2.888173] ata2: SATA max UDMA/133 mmio [mem 0xfd0c0000-0xfd0c1fff] port 0x180 irq 39 [ 2.896194] mtdoops: mtd device (mtddev=name/number) must be supplied [ 2.903439] m25p80 spi0.0: n25q512a (131072 Kbytes) [ 2.908248] 3 ofpart partitions found on MTD device spi0.0 [ 2.913703] Creating 3 MTD partitions on "spi0.0": [ 2.918479] 0x000000000000-0x000000100000 : "boot" [ 2.923754] 0x000000100000-0x000000140000 : "bootenv" [ 2.929163] 0x000000140000-0x000001740000 : "kernel" [ 2.935211] libphy: Fixed MDIO Bus: probed [ 2.940201] tun: Universal TUN/TAP device driver, 1.6 [ 2.945184] tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com> [ 2.951458] CAN device driver interface [ 2.956159] macb ff0e0000.ethernet: Not enabling partial store and forward [ 2.963345] libphy: MACB_mii_bus: probed [ 2.971086] macb ff0e0000.ethernet eth0: Could not attach to PHY [ 3.011805] tg3.c:v3.137 (May 11, 2014) [ 3.015577] pci 0000:00:00.0: enabling device (0000 -> 0002)
Broadcom NIC card probing
[ 3.021218] tg3 0000:01:00.0: enabling device (0000 -> 0002) [ 3.058190] tg3 0000:01:00.0 eth0: Tigon3 [partno(BCM95751A519) rev 4101] (PCI Express) MAC address 00:07:13:04:3d:18 [ 3.068733] tg3 0000:01:00.0 eth0: attached PHY is 5750 (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[0]) [ 3.078450] tg3 0000:01:00.0 eth0: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] TSOcap[1] [ 3.086261] tg3 0000:01:00.0 eth0: dma_rwctrl[76180000] dma_mask[64-bit]
Ethernet Interface
root@:~# ifconfig -a
eth0 Link encap:Ethernet HWaddr 00:07:13:04:3D:18
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
Interrupt:48
Testing of Ethernet interface
root@:~# ifconfig eth0 192.168.1.4 tg3 0000:01:00.0 eth0: Link is up at 1000 Mbps, full duplex tg3 0000:01:00.0 eth0: Flow control is on for TX and on for RX IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready root@:~# ping 192.168.1.2 PING 192.168.1.2 (192.168.1.2): 56 data bytes 64 bytes from 192.168.1.2: seq=0 ttl=128 time=1.502 ms 64 bytes from 192.168.1.2: seq=1 ttl=128 time=1.131 ms 64 bytes from 192.168.1.2: seq=2 ttl=128 time=1.305 ms 64 bytes from 192.168.1.2: seq=3 ttl=128 time=0.699 ms 64 bytes from 192.168.1.2: seq=4 ttl=128 time=0.762 ms 64 bytes from 192.168.1.2: seq=5 ttl=128 time=0.736 ms 64 bytes from 192.168.1.2: seq=6 ttl=128 time=1.096 ms 64 bytes from 192.168.1.2: seq=7 ttl=128 time=0.680 ms 64 bytes from 192.168.1.2: seq=8 ttl=128 time=0.783 ms 64 bytes from 192.168.1.2: seq=9 ttl=128 time=0.885 ms --- 192.168.1.2 ping statistics --- 10 packets transmitted, 10 packets received, 0% packet loss
Interrupts
root@:~# cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3
2: 0 0 0 0 GICv2 29 Level arch_timer
3: 17031 4305 1793 2036 GICv2 30 Level arch_timer
10: 0 0 0 0 GICv2 67 Level zynqmp_pm
13: 0 0 0 0 GICv2 156 Level zynqmp-dma
14: 0 0 0 0 GICv2 157 Level zynqmp-dma
15: 0 0 0 0 GICv2 158 Level zynqmp-dma
16: 0 0 0 0 GICv2 159 Level zynqmp-dma
17: 0 0 0 0 GICv2 160 Level zynqmp-dma
18: 0 0 0 0 GICv2 161 Level zynqmp-dma
19: 0 0 0 0 GICv2 162 Level zynqmp-dma
20: 0 0 0 0 GICv2 163 Level zynqmp-dma
22: 0 0 0 0 GICv2 109 Level zynqmp-dma
23: 0 0 0 0 GICv2 110 Level zynqmp-dma
24: 0 0 0 0 GICv2 111 Level zynqmp-dma
25: 0 0 0 0 GICv2 112 Level zynqmp-dma
26: 0 0 0 0 GICv2 113 Level zynqmp-dma
27: 0 0 0 0 GICv2 114 Level zynqmp-dma
28: 0 0 0 0 GICv2 115 Level zynqmp-dma
29: 0 0 0 0 GICv2 116 Level zynqmp-dma
33: 0 0 0 0 GICv2 49 Level cdns-i2c
34: 0 0 0 0 GICv2 50 Level cdns-i2c
35: 0 0 0 0 GICv2 42 Level ff960000.memory-controller
36: 14 0 0 0 GICv2 47 Level ff0f0000.spi
37: 0 0 0 0 GICv2 58 Level ffa60000.rtc
38: 0 0 0 0 GICv2 59 Level ffa60000.rtc
39: 0 0 0 0 GICv2 165 Level ahci-ceva[fd0c0000.ahci]
40: 144 0 0 0 GICv2 81 Level mmc0
41: 891 0 0 0 GICv2 53 Level xuartps
43: 0 0 0 0 GICv2 145 Edge fd4d0000.watchdog
44: 0 0 0 0 GICv2 88 Level ams-irq
45: 0 0 0 0 GICv2 151 Level fd4a0000.dp
46: 0 0 0 0 GICv2 154 Level fd4c0000.dma
47: 647 0 0 0 GICv2 121 Level xilinx-pcie
48: 647 0 0 0 dummy 1 Level eth0
223: 0 0 0 0 GICv2 97 Level xhci-hcd:usb1
IPI0: 1954 759 1664 2660 Rescheduling interrupts
IPI1: 63 57 71 43 Function call interrupts
IPI2: 0 0 0 0 CPU stop interrupts
IPI3: 349 2284 2398 2370 Timer broadcast interrupts
IPI4: 3 0 0 0 IRQ work interrupts
IPI5: 0 0 0 0 CPU wake-up interrupts
Err: 0
QDMA IP
The documentation for QDMA IP is available in PG344. It is used in bridge mode for Root port.
Root Port Driver Configuration
The PCI/PCIe subsystem support in Versal kernel configuration. For selecting QDMA PL PCIe root port driver enable CONFIG_PCIE_XDMA_PL option.
Versal QDMA PL PCIe4 Root Port:
Please refer AR76647 to add QDMA related driver patch and sample device tree.
Hardware setup
The design uses QDMA-bridge mode IP with Versal PL-PCIe4.
The following features are supported:
- Reception of legacy & MSI interrupts
- 32-bit BAR support
Tested End Points
1. Broadcom PCIe NIC card
2. Samsung 970 Evo SSD
3. Samsung 980 EVO SSD
4. Intel SSD
Standard Petalinux build flow for Linux image/devicetree generation using XSA from hardware design should be followed.
End Point Driver Configuration
The following driver is needed to be enabled for using NVMe SSD as EndPoint.
For other End point cards, ensure the respective driver is enabled in kernel.
Device Tree binding
And a sample binding is shown below and the description of DT property is documented here
pcie_dma_versal_0: axi-pcie@80000000 {
#address-cells = <3>;
#interrupt-cells = <1>;
#size-cells = <2>;
compatible = "xlnx,qdma-host-3.00";
device_type = "pci";
interrupt-map = <0 0 0 1 &psv_pcie_intc_0 1>, <0 0 0 2 &psv_pcie_intc_0 2>, <0 0 0 3 &psv_pcie_intc_0 3>, <0 0 0 4 &psv_pcie_intc_0 4>;
interrupt-map-mask = <0 0 0 7>;
interrupt-names = "misc", "msi0", "msi1";
interrupt-parent = <&gic>;
interrupts = <0 84 4 0 85 4 0 86 4>;
ranges = <0x02000000 0x00000000 0xA8000000 0x0 0xA8000000 0x00000000 0x1000000>;
reg = <0x0 0x90000000 0x0 0x5000>,
<0x0 0x80000000 0x0 0x1000000>;
reg-names = "breg", "cfg";
psv_pcie_intc_0: interrupt-controller {
#address-cells = <0>;
#interrupt-cells = <1>;
interrupt-controller ;
};
};
Test Procedure
- Boot linux onto Versal.
- Check lspci to confirm NVMe card is detected and do the following steps.
- mount /dev/nvmeXX /mnt
- cd /mnt
- dd if=/dev/zero of=tmp.txt bs=4096 count=200000
Kernel Console Output
Driver Initialization for Versal
[ 3.122832] io scheduler mq-deadline registered [ 3.127401] io scheduler kyber registered [ 3.131713] xilinx-xdma-pcie 90000000.axi-pcie: host bridge /amba_pl@0/axi-pcie@80000000 ranges: [ 3.140586] xilinx-xdma-pcie 90000000.axi-pcie: No bus range found for /amba_pl@0/axi-pcie@80000000, using [bus 00-ff] [ 3.151569] xilinx-xdma-pcie 90000000.axi-pcie: MEM 0x00a8000000..0x00a80fffff -> 0x00a8000000 [ 3.160775] xilinx-xdma-pcie 90000000.axi-pcie: PCIe Link is UP [ 3.166841] xilinx-xdma-pcie 90000000.axi-pcie: PCI host bridge to bus 0000:00 [ 3.174134] pci_bus 0000:00: root bus resource [bus 00-ff] [ 3.179668] pci_bus 0000:00: root bus resource [mem 0xa8000000-0xa80fffff] [ 3.186620] pci 0000:00:00.0: [10ee:b048] type 01 class 0x060400 [ 3.192688] pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0xffffffff 64bit pref] [ 3.200006] pci 0000:00:00.0: PME# supported from D3cold [ 3.205838] pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring [ 3.213973] pci 0000:01:00.0: [144d:a808] type 00 class 0x010802 [ 3.220054] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x00003fff 64bit] [ 3.227467] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01 [ 3.234150] pci 0000:00:00.0: BAR 0: no space for [mem size 0x100000000 64bit pref] [ 3.241877] pci 0000:00:00.0: BAR 0: failed to assign [mem size 0x100000000 64bit pref] [ 3.249949] pci 0000:00:00.0: BAR 8: assigned [mem 0xa8000000-0xa80fffff] [ 3.256794] pci 0000:01:00.0: BAR 0: assigned [mem 0xa8000000-0xa8003fff 64bit] [ 3.264179] pci 0000:00:00.0: PCI bridge to [bus 01] [ 3.269189] pci 0000:00:00.0: bridge window [mem 0xa8000000-0xa80fffff] [ 3.276445] ps_pcie_dma init() [ 3.295314] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled [ 3.302325] Serial: AMBA driver [ 3.306196] cacheinfo: Unable to detect cache hierarchy for CPU 0
NVMe card probing
[ 4.709047] nvme nvme0: pci function 0000:01:00.0 [ 4.710192] pci 0000:00:00.0: enabling device (0000 -> 0002) [ 4.715796] nvme 0000:01:00.0: enabling device (0000 -> 0002)
Testing of NVMe Access
root@:~# mount /dev/nvme0n1 /mnt cd /mnt dd if=/dev/zero of=tmp.txt bs=4096 count=200000
Interrupts
root@:/mnt# cat /proc/interrupts
CPU0 CPU1
11: 8828 22802 GICv3 30 Level arch_timer
14: 0 0 GICv3 92 Level zynqmp-dma
15: 0 0 GICv3 93 Level zynqmp-dma
16: 0 0 GICv3 94 Level zynqmp-dma
17: 0 0 GICv3 95 Level zynqmp-dma
18: 0 0 GICv3 96 Level zynqmp-dma
19: 0 0 GICv3 97 Level zynqmp-dma
20: 0 0 GICv3 98 Level zynqmp-dma
21: 0 0 GICv3 99 Level zynqmp-dma
23: 0 0 GICv3 174 Level f12a0000.rtc
24: 0 0 GICv3 175 Level f12a0000.rtc
25: 2428 0 GICv3 50 Level uart-pl011
27: 0 0 GICv3 176 Level sysmon-irq
28: 0 0 GICv3 62 Level zynqmp_ipi
29: 0 0 GICv3 116 Level xilinx-pcie
34: 4786 0 xilinx_pcie:msi 524288 Edge nvme0q0, nvme0q1
IPI0: 211 247 Rescheduling interrupts
IPI1: 1080 2723 Function call interrupts
IPI2: 0 0 CPU stop interrupts
IPI3: 0 0 CPU stop (for crash dump) interrupts
IPI4: 0 0 Timer broadcast interrupts
IPI5: 0 0 IRQ work interrupts
IPI6: 0 0 CPU wake-up interrupts
Err: 0
Related Links
© Copyright 2019 - 2022 Xilinx Inc. Privacy Policy