Versal Adaptive SoC CCIX-PCIe Module (CPM) Root port Linux driver

This page provides details for using the Root Port Linux Driver for the Versal Integrated Block for PCIe (CPM4/CPM5).

Table of Contents

Introduction

The Versal Adaptive SoC devices include CCIX-PCIe Module (CPM). The integrated block for CPM4/CPM5 PCIe along with the integrated bridge can function as PCIe Root Port with up to x16 Gen4 link configuration for CPM4 and Gen5 x8 for CPM5.

There are two integrated PCIe controllers (each capable of x8 maximum link width) and only one of them has access to the integrated bridge required for root mode. This page discusses the use of integrated PCIe controller with integrated bridge in CPM4/CPM5 in root mode.

The integrated bridge has connectivity to memory via NoC and to PS via AXI-MM interface.

The interrupts (legacy/MSI/MSI-X) received from endpoints are routed to PS. The GIC Interrupt translation services (ITS) feature is enabled to process MSI/MSI-X interrupts.

Please refer PG346 and PG347 for more details.

Features Supported in HW

  • Supports ECAM for enumeration

  • Supports Non Prefetchable memory (32-bit)

  • Supports Prefetchable memory (64-bit)

  • Supports legacy interrupts

  • Supports MSI/MSI-X via GIC ITS feature

Known Issues:

For CPM4:

Please refer AR 72972.
Please refer  AR 76652 (This AR applies from 2021.1 and above releases)

For CPM5:

Please refer AR000033976 (For 2022.1 both driver and device tree patches needed, For 2022.2 only device tree needed, driver is upstreamed.)

Features supported in driver

  • Enumeration of PCIe hierarchy

  • Assign Non Prefetchable memory

  • Assign Prefetchable memory

  • Handle legacy interrupt

Kernel Configuration

CPM4:

Enable CONFIG_PCIE_XDMA_PL option till 2021.1 release.
From 2021.2 use the CONFIG_PCIE_XILINX_CPM driver. (This is up streamed driver)

CPM5:

Enable CONFIG_PCIE_XILINX_CPM for CPM5 driver.

Device tree

 

cpm_pciea:@fca10000 { #address-cells = <3>; #interrupt-cells = <1>; #size-cells = <2>; compatible = "xlnx,versal-cpm-host-1.00"; interrupt-map = <0 0 0 1 &pcie_intc_0 1>, <0 0 0 2 &pcie_intc_0 2>, <0 0 0 3 &pcie_intc_0 3>, <0 0 0 4 &pcie_intc_0 4>; interrupt-map-mask = <0 0 0 7>; interrupt-parent = <&gic>; interrupt-names = "misc"; interrupts = <0 72 4>; ranges = <0x02000000 0x00000000 0xE0000000 0x0 0xE0000000 0x00000000 0x10000000>, <0x43000000 0x00000080 0x00000000 0x00000080 0x00000000 0x00000000 0x80000000>; msi-map = <0x0 &its_gic 0x0 0x10000>; reg = <0x6 0x00000000 0x0 0x1000000>, <0x0 0xFCA10000 0x0 0x1000>; reg-names = "cfg", "cpm_slcr"; pcie_intc_0: pci-interrupt-controller { #address-cells = <0>; #interrupt-cells = <1>; interrupt-controller ; }; };

Tested End Points

  1. Samsung 970 Evo Plus NVMe SSD

  2. Intel NVMe 750 SSD

  3. WD SN 750 NVMe SSD

  4. Intel NIC card


Test Procedure

Note: This depends on end point functionality and use case.

The following test procedure shows NVMe device testing (Enable BLK_DEV_NVME in kernel configuration)

  • Boot linux onto Versal.

  • Check lspci to confirm NVMe card is detected and do the following steps.

    • mount /dev/nvmeXX  /mnt

    • cd /mnt

    • dd if=/dev/zero of=tmp.txt bs=4096 count=200000

 

Kernel Console Output

Driver Initialization for Versal

[ 2.897050] NFS: Registering the id_resolver key type [ 2.902165] Key type id_resolver registered [ 2.906381] Key type id_legacy registered [ 2.910435] nfs4filelayout_init: NFSv4 File Layout Driver Registering... [ 2.917201] jffs2: version 2.2. (NAND) (SUMMARY) © 2001-2006 Red Hat, Inc. [ 2.953047] NET: Registered protocol family 38 [ 2.957537] xor: measuring software checksum speed [ 2.964107] 8regs : 5654 MB/sec [ 2.970004] 32regs : 6520 MB/sec [ 2.976080] arm64_neon : 5834 MB/sec [ 2.980468] xor: using function: 32regs (6520 MB/sec) [ 2.985557] Key type asymmetric registered [ 2.989682] Asymmetric key parser 'x509' registered [ 2.994612] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.002069] io scheduler mq-deadline registered [ 3.006633] io scheduler kyber registered [ 3.011024] xilinx-cpm-pcie fca10000.pci: host bridge /axi/pci@fca10000 ranges: [ 3.018401] xilinx-cpm-pcie fca10000.pci: No bus range found for /axi/pci@fca10000, using [bus 00-ff] [ 3.027888] xilinx-cpm-pcie fca10000.pci: MEM 0x00e0010000..0x00f000ffff -> 0x00e0010000 [ 3.036491] xilinx-cpm-pcie fca10000.pci: MEM 0x8000000000..0x807fffffff -> 0x8000000000 [ 3.045128] xilinx-cpm-pcie fca10000.pci: ECAM area [mem 0x600000000-0x600ffffff] can only accommodate [bus 00-0f] (reduced from [bus 00-ff] desired) [ 3.058640] xilinx-cpm-pcie fca10000.pci: ECAM at [mem 0x600000000-0x600ffffff] for [bus 00-0f] [ 3.067411] xilinx-cpm-pcie fca10000.pci: PCIe Link is UP [ 3.073350] xilinx-cpm-pcie fca10000.pci: PCI host bridge to bus 0000:00 [ 3.080105] pci_bus 0000:00: root bus resource [bus 00-ff] [ 3.085633] pci_bus 0000:00: root bus resource [mem 0xe0010000-0xf000ffff] [ 3.092562] pci_bus 0000:00: root bus resource [mem 0x8000000000-0x807fffffff] [ 3.099862] pci 0000:00:00.0: [10ee:b03f] type 01 class 0x060400 [ 3.105920] pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0x7fffffff] [ 3.112269] pci 0000:00:00.0: supports D1 [ 3.116309] pci 0000:00:00.0: PME# supported from D0 D1 D3hot D3cold [ 3.123156] pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring [ 3.131292] pci 0000:01:00.0: [144d:a80a] type 00 class 0x010802 [ 3.137360] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x00003fff 64bit] [ 3.144706] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01 [ 3.151383] pci 0000:00:00.0: BAR 0: no space for [mem size 0x80000000] [ 3.158051] pci 0000:00:00.0: BAR 0: failed to assign [mem size 0x80000000] [ 3.165068] pci 0000:00:00.0: BAR 8: assigned [mem 0xe0100000-0xe01fffff] [ 3.171912] pci 0000:01:00.0: BAR 0: assigned [mem 0xe0100000-0xe0103fff 64bit] [ 3.179288] pci 0000:00:00.0: PCI bridge to [bus 01] [ 3.184290] pci 0000:00:00.0: bridge window [mem 0xe0100000-0xe01fffff] [ 3.191565] ps_pcie_dma init() [ 3.210174] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled [ 3.217174] Serial: AMBA driver [ 3.221035] cacheinfo: Unable to detect cache hierarchy for CPU 0 [ 3.230203] brd: module loaded [ 3.236245] loop: module loaded [ 3.240072] nvme nvme0: pci function 0000:01:00.0 [ 3.244846] pci 0000:00:00.0: enabling device (0000 -> 0002) [ 3.245005] mtdoops: mtd device (mtddev=name/number) must be supplied [ 3.250559] nvme 0000:01:00.0: enabling device (0000 -> 0002) [ 3.257851] libphy: Fixed MDIO Bus: probed [ 3.265491] nvme nvme0: Shutdown timeout set to 10 seconds [ 3.267748] tun: Universal TUN/TAP device driver, 1.6 [ 3.277712] CAN device driver interface [ 3.278125] nvme nvme0: 2/0/0 default/read/poll queues [ 3.281846] usbcore: registered new interface driver asix [ 3.292233] usbcore: registered new interface driver ax88179_178a

 

Testing of NVMe Access and Interrupts

root@xilinx:~# lspci 00:00.0 PCI bridge: Xilinx Corporation Device a03f