This wiki page documents a collection of PCIe tips and tricks.

Table of Contents

Introduction

The purpose of this page is to document some of the harder to find PCIe details. A lot of the information is general such that is applies across Xilinx architectures even though some specifics may be illustrated.

Linux Root Port Details

The Xilinx PCIe hardware typically supports both root port and endpoint. Linux refers to the drivers as Host drivers due to the legacy of PCI. Xilinx PCIe hardware is not a root complex as it only contains a root port. The following diagram illustrates the layers of device drivers in an MPSoC Linux system as there can be multiple PCIe drivers which may not be obvious.

Open

The lspci Command

The lspci command is well documented in many other places yet there are a few details that are not obvious. The following output of the lspci command illustrates the host (root) driver details.

The “Kernel driver in use” does not indicate the host / root driver but a higher level driver running on the root. The previous example shows that the Xilinx PS PCIe DMA driver (a DMA driver shown as ps_pcie_dma) is running on the host for MPSoC. This field of the lspci command output may also be absent when there is no higher level driver (above the root driver) running. The pcie port bus driver (shown as pcieport) is common in systems requiring AER support.

Device Tree

Bridges are less typical in Xilinx systems and tend to be complex due to mapping memory and interrupts across the bridge. Each Xilinx PCIe root driver documents the device tree bindings unique to the driver, but only gives examples without the details of how the bridge bindings work with respect to translation of addresses and interrupts across the bridge. The PCIe device tree bindings in the kernel at Linux Kernel PCIe Device Tree Bindings refers the user to other documents and specifications, but does not explain the details directly. Other sites such as https://elinux.org/Device_Tree_Usage provide better examples for PCIe which also explain the address and interrupt mappings.

Interrupts

It’s important to remember that the PCIe root is a bridge such that there can be multiple PCIe interrupts that must come across the bridge and connect to a single interrupt on the root, such as to the GIC. The PCIe root acts as an interrupt controller to cascade the interrupts to the primary system interrupt controller as specified in the device tree. Interrupt controller functionality in Linux is a complex topic such that understanding all the details in /proc/interrupts is a non-trivial task for those that are not kernel experts.

The following interrupt information illustrates the MPSoC root driver using physical interrupt 150 and Linux (virtual) interrupt 37. It also illustrates another driver on the root that is using Linux (virtual) interrupt 54 from physical interrupt 0 of the interrupt controller of the PCIe root.

The lspci -vv command also illustrates interrupt details that can be useful. The “Interrupt: pin A routed to IRQ 54' illustrates when there is a higher level driver, such as the pcie port bus driver, running on the root port. Without a higher level driver running on the root a value of 255 is displayed for the interrupt.

Enabling An Endpoint

An endpoint without a device driver can be manipulated to verify basic functionality. The following lspci command shows an ethernet controller without a device driver such that the BAR at memory address 0xE0000000 is disabled. Any memory operations to the BAR using devmem fail.

The endpoint BAR is enabled as illustrated below.

The following lspci command now shows the endpoint BAR is enabled as it does not say it is disabled and the BAR can now be accessed with devmem.

Rescan The PCIe Bus

The following examples illustrate the ability to rescan the PCIe for endpoints in several scenarios.

The root board is booted with the endpoint in a slot but without any power. No endpoint is seen as illustrated below.

Power up the endpoint in the slot and rescan the PCIe bus seeing that the endpoint is found.

Power off the endpoint and see that the endpoint is not functional.

Power up the endpoint and rescan the PCIe bus to see the endpoint is found again and uses the same BAR address.

Related Links

• https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18841873

• Accessing PCI device resources through sysfs — The Linux Kernel documentation

• In-depth Linux Interrupt Details