Versal Adaptive SoC CCIX-PCIe Module (CPM) Root port Linux driver
This page provides details for using the Root Port Linux Driver for the Versal Integrated Block for PCIe (CPM4/CPM5).
Table of Contents
Introduction
The Versal Adaptive SoC devices include CCIX-PCIe Module (CPM). The integrated block for CPM4/CPM5 PCIe along with the integrated bridge can function as PCIe Root Port with up to x16 Gen4 link configuration for CPM4 and Gen5 x8 for CPM5.
There are two integrated PCIe controllers (each capable of x8 maximum link width) and only one of them has access to the integrated bridge required for root mode. This page discusses the use of integrated PCIe controller with integrated bridge in CPM4/CPM5 in root mode.
The integrated bridge has connectivity to memory via NoC and to PS via AXI-MM interface.
The interrupts (legacy/MSI/MSI-X) received from endpoints are routed to PS. The GIC Interrupt translation services (ITS) feature is enabled to process MSI/MSI-X interrupts.
Please refer PG346 and PG347 for more details.
Features Supported in HW
Supports ECAM for enumeration
Supports Non Prefetchable memory (32-bit)
Supports Prefetchable memory (64-bit)
Supports legacy interrupts
Supports MSI/MSI-X via GIC ITS feature
Known Issues:
For CPM4:
Please refer AR 72972.
Please refer AR 76652 (This AR applies from 2021.1 and above releases)
For CPM5:
Please refer AR000033976 (For 2022.1 both driver and device tree patches needed, For 2022.2 only device tree needed, driver is upstreamed.)
Features supported in driver
Enumeration of PCIe hierarchy
Assign Non Prefetchable memory
Assign Prefetchable memory
Handle legacy interrupt
Kernel Configuration
CPM4:
Enable CONFIG_PCIE_XDMA_PL option till 2021.1 release.
From 2021.2 use the CONFIG_PCIE_XILINX_CPM driver. (This is up streamed driver)
CPM5:
Enable CONFIG_PCIE_XILINX_CPM for CPM5 driver.
Device tree
cpm_pciea:@fca10000 {
#address-cells = <3>;
#interrupt-cells = <1>;
#size-cells = <2>;
compatible = "xlnx,versal-cpm-host-1.00";
interrupt-map = <0 0 0 1 &pcie_intc_0 1>,
<0 0 0 2 &pcie_intc_0 2>,
<0 0 0 3 &pcie_intc_0 3>,
<0 0 0 4 &pcie_intc_0 4>;
interrupt-map-mask = <0 0 0 7>;
interrupt-parent = <&gic>;
interrupt-names = "misc";
interrupts = <0 72 4>;
ranges = <0x02000000 0x00000000 0xE0000000 0x0 0xE0000000 0x00000000 0x10000000>,
<0x43000000 0x00000080 0x00000000 0x00000080 0x00000000 0x00000000 0x80000000>;
msi-map = <0x0 &its_gic 0x0 0x10000>;
reg = <0x6 0x00000000 0x0 0x1000000>,
<0x0 0xFCA10000 0x0 0x1000>;
reg-names = "cfg", "cpm_slcr";
pcie_intc_0: pci-interrupt-controller {
#address-cells = <0>;
#interrupt-cells = <1>;
interrupt-controller ;
};
};Tested End Points
Samsung 970 Evo Plus NVMe SSD
Intel NVMe 750 SSD
WD SN 750 NVMe SSD
Intel NIC card
Test Procedure
Note: This depends on end point functionality and use case.
The following test procedure shows NVMe device testing (Enable BLK_DEV_NVME in kernel configuration)
Boot linux onto Versal.
Check lspci to confirm NVMe card is detected and do the following steps.
mount /dev/nvmeXX /mnt
cd /mnt
dd if=/dev/zero of=tmp.txt bs=4096 count=200000
Kernel Console Output
Driver Initialization for Versal
[ 2.897050] NFS: Registering the id_resolver key type
[ 2.902165] Key type id_resolver registered
[ 2.906381] Key type id_legacy registered
[ 2.910435] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[ 2.917201] jffs2: version 2.2. (NAND) (SUMMARY) © 2001-2006 Red Hat, Inc.
[ 2.953047] NET: Registered protocol family 38
[ 2.957537] xor: measuring software checksum speed
[ 2.964107] 8regs : 5654 MB/sec
[ 2.970004] 32regs : 6520 MB/sec
[ 2.976080] arm64_neon : 5834 MB/sec
[ 2.980468] xor: using function: 32regs (6520 MB/sec)
[ 2.985557] Key type asymmetric registered
[ 2.989682] Asymmetric key parser 'x509' registered
[ 2.994612] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247)
[ 3.002069] io scheduler mq-deadline registered
[ 3.006633] io scheduler kyber registered
[ 3.011024] xilinx-cpm-pcie fca10000.pci: host bridge /axi/pci@fca10000 ranges:
[ 3.018401] xilinx-cpm-pcie fca10000.pci: No bus range found for /axi/pci@fca10000, using [bus 00-ff]
[ 3.027888] xilinx-cpm-pcie fca10000.pci: MEM 0x00e0010000..0x00f000ffff -> 0x00e0010000
[ 3.036491] xilinx-cpm-pcie fca10000.pci: MEM 0x8000000000..0x807fffffff -> 0x8000000000
[ 3.045128] xilinx-cpm-pcie fca10000.pci: ECAM area [mem 0x600000000-0x600ffffff] can only accommodate [bus 00-0f] (reduced from [bus 00-ff] desired)
[ 3.058640] xilinx-cpm-pcie fca10000.pci: ECAM at [mem 0x600000000-0x600ffffff] for [bus 00-0f]
[ 3.067411] xilinx-cpm-pcie fca10000.pci: PCIe Link is UP
[ 3.073350] xilinx-cpm-pcie fca10000.pci: PCI host bridge to bus 0000:00
[ 3.080105] pci_bus 0000:00: root bus resource [bus 00-ff]
[ 3.085633] pci_bus 0000:00: root bus resource [mem 0xe0010000-0xf000ffff]
[ 3.092562] pci_bus 0000:00: root bus resource [mem 0x8000000000-0x807fffffff]
[ 3.099862] pci 0000:00:00.0: [10ee:b03f] type 01 class 0x060400
[ 3.105920] pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0x7fffffff]
[ 3.112269] pci 0000:00:00.0: supports D1
[ 3.116309] pci 0000:00:00.0: PME# supported from D0 D1 D3hot D3cold
[ 3.123156] pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 3.131292] pci 0000:01:00.0: [144d:a80a] type 00 class 0x010802
[ 3.137360] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x00003fff 64bit]
[ 3.144706] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01
[ 3.151383] pci 0000:00:00.0: BAR 0: no space for [mem size 0x80000000]
[ 3.158051] pci 0000:00:00.0: BAR 0: failed to assign [mem size 0x80000000]
[ 3.165068] pci 0000:00:00.0: BAR 8: assigned [mem 0xe0100000-0xe01fffff]
[ 3.171912] pci 0000:01:00.0: BAR 0: assigned [mem 0xe0100000-0xe0103fff 64bit]
[ 3.179288] pci 0000:00:00.0: PCI bridge to [bus 01]
[ 3.184290] pci 0000:00:00.0: bridge window [mem 0xe0100000-0xe01fffff]
[ 3.191565] ps_pcie_dma init()
[ 3.210174] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
[ 3.217174] Serial: AMBA driver
[ 3.221035] cacheinfo: Unable to detect cache hierarchy for CPU 0
[ 3.230203] brd: module loaded
[ 3.236245] loop: module loaded
[ 3.240072] nvme nvme0: pci function 0000:01:00.0
[ 3.244846] pci 0000:00:00.0: enabling device (0000 -> 0002)
[ 3.245005] mtdoops: mtd device (mtddev=name/number) must be supplied
[ 3.250559] nvme 0000:01:00.0: enabling device (0000 -> 0002)
[ 3.257851] libphy: Fixed MDIO Bus: probed
[ 3.265491] nvme nvme0: Shutdown timeout set to 10 seconds
[ 3.267748] tun: Universal TUN/TAP device driver, 1.6
[ 3.277712] CAN device driver interface
[ 3.278125] nvme nvme0: 2/0/0 default/read/poll queues
[ 3.281846] usbcore: registered new interface driver asix
[ 3.292233] usbcore: registered new interface driver ax88179_178a
Testing of NVMe Access and Interrupts
root@xilinx:~# lspci
00:00.0 PCI bridge: Xilinx Corporation Device a03f
01:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981
root@xilinx:~# mount /dev/nvme0n1 /mnt/
root@xilinx:~# cd /mnt
root@xilinx:~# dd if=/dev/zero of=tmp.txt bs=8192 count=200000
200000+0 records in
200000+0 records out
root@xilinx:~# cat /proc/interrupts
CPU0 CPU1
3: 2741 8093 GICv3 30 Level arch_timer
9: 0 0 GICv3 92 Level zynqmp-dma
10: 0 0 GICv3 93 Level zynqmp-dma
11: 0 0 GICv3 94 Level zynqmp-dma
12: 0 0 GICv3 95 Level zynqmp-dma
13: 0 0 GICv3 96 Level zynqmp-dma
14: 0 0 GICv3 97 Level zynqmp-dma
15: 0 0 GICv3 98 Level zynqmp-dma
16: 0 0 GICv3 99 Level zynqmp-dma
21: 0 0 GICv3 47 Level cdns-i2c
22: 0 0 GICv3 174 Level f12a0000.rtc
23: 0 0 GICv3 175 Level f12a0000.rtc
24: 0 0 GICv3 160 Level mmc0
25: 1114 0 GICv3 50 Level uart-pl011
27: 3 0 GICv3 157 Level f1030000.spi
28: 0 0 GICv3 104 Level xilinx-pcie
29: 0 0 GICv3 62 Level zynqmp_ipi
31: 10 0 ITS-MSI 524288 Edge nvme0q0
32: 730 0 ITS-MSI 524289 Edge nvme0q1
33: 0 1741 ITS-MSI 524290 Edge nvme0q2
IPI0: 1881 1446 Rescheduling interrupts
IPI1: 1289 465 Function call interrupts
IPI2: 0 0 CPU stop interrupts
IPI3: 0 0 CPU stop (for crash dump) interrupts
IPI4: 0 0 Timer broadcast interrupts
IPI5: 0 0 IRQ work interrupts
IPI6: 0 0 CPU wake-up interrupts
Err: 0
Debug Information
If Link is Down
Check if EP is connected properly and all components in setup are proper.
Try changing EP device.
Test and confirm if EP is in working condition by connecting and checking on different host (Ex: x86).
If using a switch, check if switch settings for the slot are correct or not. Try changing switch slots.
Endpoint not detected/enumerated
Try using a different endpoint device.
If using a switch, check if switch settings for the slot are correct or not. Try changing switch slots.
If issue still persist raise support request along with boot log with “lspci -vvv” and design, EP details and release version used.
Change Log
2019.2
Add support for Versal CPM4 root port driver
2020.1
None
2020.2
None
2021.1
None
2021.2
Added support for Versal CPM4 root port mainline driver.
2022.1
Added support for Versal CPM5 root port patch.
2022.2
Added support for Versal CPM5 root port mainline driver.
© Copyright 2019 - 2022 Xilinx Inc. Privacy Policy