PCIe Tips and Tricks
This wiki page documents a collection of PCIe tips and tricks.
Table of Contents
Introduction
The purpose of this page is to document some of the harder to find PCIe details. A lot of the information is general such that is applies across Xilinx architectures even though some specifics may be illustrated.
Linux Root Port Details
The Xilinx PCIe hardware typically supports both root port and endpoint. Linux refers to the drivers as Host drivers due to the legacy of PCI. Xilinx PCIe hardware is not a root complex as it only contains a root port. The following diagram illustrates the layers of device drivers in an MPSoC Linux system as there can be multiple PCIe drivers which may not be obvious.
The lspci Command
The lspci command is well documented in many other places yet there are a few details that are not obvious. The following output of the lspci command illustrates the host (root) driver details.
# lspci -v
00:00.0 PCI bridge: Xilinx Corporation Device d021 (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0, IRQ 95
Bus: primary=00, secondary=01, subordinate=0c, sec-latency=0
I/O behind bridge: 00000000-00000fff [size=4K]
Memory behind bridge: [disabled]
Prefetchable memory behind bridge: [disabled]
Capabilities: [40] Power Management version 3
Capabilities: [60] Express Root Port (Slot-), MSI 00
Capabilities: [100] Device Serial Number 00-00-00-00-00-00-00-00
Capabilities: [10c] Virtual Channel
Capabilities: [128] Vendor Specific Information: ID=1234 Rev=1 Len=018 <?>
Kernel driver in use: ps_pcie_dma
The “Kernel driver in use” does not indicate the host / root driver but a higher level driver running on the root. The previous example shows that the Xilinx PS PCIe DMA driver (a DMA driver shown as ps_pcie_dma) is running on the host for MPSoC. This field of the lspci command output may also be absent when there is no higher level driver (above the root driver) running. The pcie port bus driver (shown as pcieport) is common in systems requiring AER support.
Device Tree
Bridges are less typical in Xilinx systems and tend to be complex due to mapping memory and interrupts across the bridge. Each Xilinx PCIe root driver documents the device tree bindings unique to the driver, but only gives examples without the details of how the bridge bindings work with respect to translation of addresses and interrupts across the bridge. The PCIe device tree bindings in the kernel at Linux Kernel PCIe Device Tree Bindings refers the user to other documents and specifications, but does not explain the details directly. Other sites such as https://elinux.org/Device_Tree_Usage provide better examples for PCIe which also explain the address and interrupt mappings.
Interrupts
It’s important to remember that the PCIe root is a bridge such that there can be multiple PCIe interrupts that must come across the bridge and connect to a single interrupt on the root, such as to the GIC. The PCIe root acts as an interrupt controller to cascade the interrupts to the primary system interrupt controller as specified in the device tree. Interrupt controller functionality in Linux is a complex topic such that understanding all the details in /proc/interrupts is a non-trivial task for those that are not kernel experts.
The following interrupt information illustrates the MPSoC root driver using physical interrupt 150 and Linux (virtual) interrupt 37. It also illustrates another driver on the root that is using Linux (virtual) interrupt 54 from physical interrupt 0 of the interrupt controller of the PCIe root.
# cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3
37: 0 0 0 0 GICv2 150 Level nwl_pcie:misc
54: 0 0 0 0 nwl_pcie:legacy 0 Level PCIe PME
The lspci -vv command also illustrates interrupt details that can be useful. The “Interrupt: pin A routed to IRQ 54' illustrates when there is a higher level driver, such as the pcie port bus driver, running on the root port. Without a higher level driver running on the root a value of 255 is displayed for the interrupt.
# lspci -vv
00:00.0 PCI bridge: Xilinx Corporation Device d021 (prog-if 00 [Normal decode])
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
<...>
Interrupt: pin A routed to IRQ 54
Enabling An Endpoint
An endpoint without a device driver can be manipulated to verify basic functionality. The following lspci command shows an ethernet controller without a device driver such that the BAR at memory address 0xE0000000 is disabled. Any memory operations to the BAR using devmem fail.
The endpoint BAR is enabled as illustrated below.
The following lspci command now shows the endpoint BAR is enabled as it does not say it is disabled and the BAR can now be accessed with devmem.
Rescan The PCIe Bus
The following examples illustrate the ability to rescan the PCIe for endpoints in several scenarios.
The root board is booted with the endpoint in a slot but without any power. No endpoint is seen as illustrated below.
Power up the endpoint in the slot and rescan the PCIe bus seeing that the endpoint is found.
Power off the endpoint and see that the endpoint is not functional.
Power up the endpoint and rescan the PCIe bus to see the endpoint is found again and uses the same BAR address.
Linux End Point Details
AMD/Xilinx FPGAs and SoCs are commonly used in a PCIe endpoint. The following paragraphs describe methods for designing an endpoint and associated device driver running on a Linux host (root complex).
Introduction
AMD system designs with SoCs and FPGAs utilize AMD IP connected to CPUs with a dedicated AXI interface system with ARM and MicroBlaze CPUs. Linux on these CPUs utilizes the device tree to describe the hardware components of the system while X86 based Linux systems are generally not device tree based. The X86 based Linux systems tend to use PCIe rather than device tree due to the history around PCs. Regardless of the CPU, PCIe is a discoverable bus such that device trees are not typically used to describe the devices on the PCIe endpoint.
Users tend to want to use the AMD IP in the design of an endpoint and expect the existing Linux device drivers to function but device tree based drivers not function when the IP is implemented as a PCIe endpoint. The following paragraphs describe considerations for building PCIe endpoints using those same components (IP) such that they are accessed over PCIe. The PCIe root complex could be any CPU such as ARM or X86. An endpoint using a single IP is simple and straight forward while multiple IPs on the endpoint quickly become more complex.
Linux, PCIe, and Custom IP
In general, Linux expects PCIe devices to be pretty normal with specific functionality, like a network NIC, rather than with a bunch of unrelated devices, like I2C, SPI, etc… all together in a single device. Some AMD PCIe IP supports multiple functions which could help, but not all the devices support that functionality, such as the Artix device. A custom FPGA design can allow a designer to put all kinds of unrelated IP into one PCIe endpoint which can make the driver design a bit more challenging. It can also be an issue if the kernel driver is expected to be pushed upstream where they tend to dislike such designs. An example system is illustrated below.
AXI vs PCIe System Addresses
AMD systems with AXI IP use absolute addressing such that the addresses of each IP is fixed when the hardware is built in Vivado. The AXI infrastructure knows all the addresses to allow devices to be accessed at the specified addresses. IP on a PCIe endpoint uses relative addressing such that each IP is relative to the address where the BAR is assigned by the CPU of the root complex. The address assignment of a PCIe endpoint BAR by a CPU of the root complex can be dynamic across different system configurations. A PCIe to AXI Bridge may also provide address translation between the two sides of the bridge.
Interrupts
Interrupts require some thought and planning since they play a key role for the device driver. Interrupts can be challenging since they require a lot of testing to verify there are no issues, and when there are issues they can be challenging to debug (such as a missing interrupt). Linux drivers with a lot of debug can change the behavior such that the operation is completely broken or so slow as to alter the system a lot.
The interrupts from the AMD IP varies with some being level triggered and some being edge triggered. IP interrupts are translated into PCIe interrupts. The interrupt input for the PCIe IP varies across IP but is generally a request followed by a grant to indicate the IP accepted the request. Some translation from the IP interrupts to the PCIe interrupt may be needed. The AXI IIC IP core is a level triggered interrupt while the AXI QSPI IP core is an edge triggered interrupt. Level triggered interrupts should be reviewed carefully in the driver to verify interrupts are not dropped depending on how the translation from the IP to the PCIe domain happens.
It can be challenging to find the interrupt trigger type for the IP in documentation but it can be seen in Vivado as shown below. The interrupt pin is selected and then the configuration is viewed showing the sensitivity (edge or level) of the signal.
Linux Device Driver Architecture
Platform devices in Linux are devices that are typically on the board with the CPU at some fixed address rather than at an unknown discovered address such as PCIe. Device tree drivers use a kernel layer to read the device tree and then instantiate platform devices, but require the kernel to support device tree which is not typical with X86.
Existing Example
The MACB ethernet driver, from Cadence, as described at Macb Driver, is an example of driver with both device tree and PCIe support. The MACB PCI support is not tested or described by AMD/Xilinx. The PCI support can be seen in the driver directory in the macb_pci.c source file at https://github.com/Xilinx/linux-xlnx/tree/master/drivers/net/ethernet/cadence. This example is used to create a new driver to support the IIC IP and can be used for other IP.
Related Links
© Copyright 2019 - 2022 Xilinx Inc. Privacy Policy