This page gives an overview of AXI PCIe Root Complex driver for the Xilinx AXI PCIe Soft IP, which is available as part of the Zynq and Microblaze Linux distributions.

Table of Contents

AXI PCIe Soft IP

PCI Express (abbreviated as PCIe) is the newest bus standard designed to replace the old PCI/PCI-X and AGP standards. PCIe is used in servers, consumer, and industrial applicatios either as a motherboard-level interconnection to link peripherals or as an expansion card interface for add on boards. The latest PCIe IP released by XILINX (axi_pcie_v2_7) could be configured at hardware build time either as a root port or as an end point. The support for Root Port configuration has been integrated with the latest Zynq as well as Microblaze Linux Kernel. This PCIe core supports the Zynq and 7-series Device family. This page mainly discusses the Root Port driver and an example end point driver is demonstrated in TRD release with links pointed at the end of this page. For more information about AXI PCIe IP, please refer to documentation provided in the "Related Links" section.

Hardware Setup

The hardware setup is identically same for both Zynq and Microblaze platforms. The following figure shows how the Zynq board is configured to use the PCIe,

For Microblaze platform, the board should be replaced with a 7-series board.

FMC to PCIe Daughter card should be connected to one of the FMC connector slot either HPC/LPC based on the HW design. The FMC card information can be found at,
http://www.xilinx.com/products/boards_kits/fmc.htm

Tested End Point cards:
1. Broadcom PCIe NIC card
2. Realtek NIC card

Root Port Driver Configuration

Starting from 2015.4 there is unified Root Port driver for Zynq and Microblaze platforms. Earlier to 2015.4 there are two separate drivers for Zynq and Microblaze. The unified driver support PCIe MSI feature. The unified driver can be found here,

For Zynq/Microblaze:
https://github.com/Xilinx/linux-xlnx/blob/master/drivers/pci/host/pcie-xilinx.c

The PCI/PCIe subsystem support and Root Port driver is enabled by default in Zynq/Microblaze kernel configuration. The related code is always built with the kernel whether the hardware build includes PCIe IP or not. So, the user does not need to change anything in the configuration files to bring in PCIe support into Zynq/Microblaze kernel. There are other "optional" configuration items for configuring PCI/PCIe driver MSI support and those can be configured from "menuconfig" -> Bus options ->Message Signaled Interrupts (MSI and MSI-X).

End Point Driver Configuration

This page demonstrates the Root Port driver using Broadcom NIC Endpoint, for which NIC driver should be enabled in kernel as shown,

For other End point cards, ensure the respective driver is loaded into kernel.

Device Tree binding

The device tree node for AXI PCIe core will be automatically generated, if the core is configured in the HW design, using the Device Tree BSP.

Steps to generate device-tree is documented here,
http://www.wiki.xilinx.com/Build+Device+Tree+Blob

And a sample binding is shown below and the description of DT property is documented here

    pci_express: axi-pcie@50000000 {
        #address-cells = <3>;
        #size-cells = <2>;
        #interrupt-cells = <1>;
        compatible = "xlnx,axi-pcie-host-1.00.a";
        reg = < 0x50000000 0x10000000 >;
        device_type = "pci";
        interrupts = <0 52 4>;
        interrupt-map-mask = <0 0 0 7>;
        interrupt-map = <0 0 0 1 &pcie_intc 1>,
                        <0 0 0 2 &pcie_intc 2>,
                        <0 0 0 3 &pcie_intc 3>,
                        <0 0 0 4 &pcie_intc 4>;
        ranges = < 0x02000000 0 0x60000000 0x60000000 0 0x10000000 >;
        pcie_intc: interrupt-controller {
                 interrupt-controller;
                 #address-cells = <0>;
                 #interrupt-cells = <1>;
        };
    };

Test Procedure

Load the linux onto Zynq ZC706 board/Microblaze KC705 board.
After successful booting of the linux, will be able to see the Broadcom NIC endpoint driver is been probed. Run the command ‘lspci’ from the user prompt, which shows the the device id and manufacturer id of the Broadcom NIC card. This ensures the enumeration of the device is fine.
The Broadcom NIC will show up as an eth0 interface in the linux. Run the command ‘ifconfig eth0 up’ which brings up the Ethernet interface. This step ensures all the memory transactions on the PCIe bus are working.
Assign a IP address either static/dhcp. For dhcp, run ‘udhcpc –i eth0’ to lease an IP and then assign the address.
Ping to a device of known IP address.
Repeat the above steps with MSI supported linux image. This can be checked by observing the increase in MSI interrupts (cat /proc/interrupts).

Kernel Console Output

Driver Initialization for Zynq

xilinx-pcie 50000000.axi-pcie: PCIe Link is UP
PCI host bridge /amba/axi-pcie@50000000 ranges:
  No bus range found for /amba/axi-pcie@50000000, using [bus 00-ff]
  MEM 0x60000000..0x6fffffff -> 0x60000000
xilinx-pcie 50000000.axi-pcie: PCI host bridge to bus 0000:00
pci_bus 0000:00: root bus resource [bus 00-ff]
pci_bus 0000:00: root bus resource [mem 0x60000000-0x6fffffff]
PCI: bus0: Fast back to back transfers disabled
pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
PCI: bus1: Fast back to back transfers disabled
pci 0000:00:00.0: BAR 0: no space for [mem size 0x40000000]
pci 0000:00:00.0: BAR 0: failed to assign [mem size 0x40000000]
pci 0000:00:00.0: BAR 8: assigned [mem 0x60000000-0x600fffff]
pci 0000:01:00.0: BAR 0: assigned [mem 0x60000000-0x6000ffff 64bit]
pci 0000:01:00.0: BAR 6: assigned [mem 0x60010000-0x6001ffff pref]
pci 0000:00:00.0: PCI bridge to [bus 01]
pci 0000:00:00.0:   bridge window [mem 0x60000000-0x600fffff]
dma-pl330 f8003000.dmac: Loaded driver for PL330 DMAC-241330
dma-pl330 f8003000.dmac:     DBUFF-128x8bytes Num_Chans-8 Num_Peri-4 Num_Events-16
e0001000.serial: ttyPS0 at MMIO 0xe0001000 (irq = 26, base_baud = 3125000) is a xuartps
console [ttyPS0] enabled
[drm] Initialized drm 1.1.0 20060810
brd: module loaded
loop: module loaded
CAN device driver interface
libphy: MACB_mii_bus: probed
macb e000b000.ethernet eth0: Cadence GEM rev 0x00020118 at 0xe000b000 irq 28 (00:0a:35:00:01:22)
macb e000b000.ethernet eth0: attached PHY driver [Marvell 88E1116R] (mii_bus:phy_addr=e000b000.etherne:07, irq=-1)
tg3.c:v3.137 (May 11, 2014)
pci 0000:00:00.0: enabling device (0140 -> 0142)

Driver Initialization for Microblaze

xilinx-pcie 10000000.pciex: PCIe Link is UP
PCI host bridge /amba_pl/pciex@10000000 ranges:
  MEM 0x80000000..0x8fffffff -> 0x80000000
xilinx-pcie 10000000.pciex: PCI host bridge to bus 0000:00
pci_bus 0000:00: root bus resource [bus 00-ff]
pci_bus 0000:00: root bus resource [mem 0x80000000-0x8fffffff]
pci 0000:00:00.0: [10ee:0705] type 01 class 0x060400
pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0x7fffffff]
pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:01:00.0: [14e4:1677] type 00 class 0x020000
pci 0000:01:00.0: reg 0x10: [mem 0x80000000-0x8000ffff 64bit]
pci 0000:01:00.0: reg 0x30: [mem 0x90480000-0x9048ffff pref]
pci 0000:01:00.0: PME# supported from D3hot D3cold
pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01
pci 0000:00:00.0: BAR 0: no space for [mem size 0x80000000]
pci 0000:00:00.0: BAR 0: failed to assign [mem size 0x80000000]
pci 0000:00:00.0: BAR 8: assigned [mem 0x80000000-0x800fffff]
pci 0000:01:00.0: BAR 0: assigned [mem 0x80000000-0x8000ffff 64bit]
pci 0000:01:00.0: BAR 6: assigned [mem 0x80010000-0x8001ffff pref]
pci 0000:00:00.0: PCI bridge to [bus 01]
pci 0000:00:00.0:   bridge window [mem 0x80000000-0x800fffff]
ipmi message handler version 39.2
IPMI System Interface driver.
ipmi_si: Unable to find any System Interface(s)
Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
30600000.serial: ttyUL0 at MMIO 0x30600000 (irq = 3, base_baud = 0) is a uartlite
console [ttyUL0] enabled
brd: module loaded
loop: module loaded
libphy: Fixed MDIO Bus: probed
tg3.c:v3.137 (May 11, 2014)
pci 0000:00:00.0: enabling device (0000 -> 0002)

Broadcom NIC card probing

tg3 0000:01:00.0: enabling device (0140 -> 0142)
tg3 0000:01:00.0 eth1: Tigon3 [partno(BCM95751A519FLP) rev 4201] (PCI Express) MAC address 00:10:18:32:d2:a9
tg3 0000:01:00.0 eth1: attached PHY is 5750 (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[0])
tg3 0000:01:00.0 eth1: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] TSOcap[1]
tg3 0000:01:00.0 eth1: dma_rwctrl[76180000] dma_mask[64-bit]

lspci output

Linux> lspci
00:00.0 PCI bridge: Xilinx Corporation Device 0705
01:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5751 Gigabit Ethernet PCI Express (rev 21)

Ethernet Interface

ifconfig -a eth0
          Link encap:Ethernet  HWaddr 00:10:18:32:D2:A9
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
          Interrupt:48

Testing of Ethernet interface

Linux> ifconfig eth0 192.168.1.10
Linux> tg3 0000:01:00.0: eth0: Link is up at 1000 Mbps, full duplex
tg3 0000:01:00.0: eth0: Flow control is on for TX and on for RX
Linux>
Linux> ping -c 3 192.168.1.100
PING 192.168.1.100 (192.168.1.100): 56 data bytes
64 bytes from 192.168.1.100: seq=0 ttl=128 time=1.291 ms
64 bytes from 192.168.1.100: seq=1 ttl=128 time=0.537 ms
64 bytes from 192.168.1.100: seq=2 ttl=128 time=0.553 ms
--- 192.168.1.100 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.537/0.793/1.291 ms
Linux>

MSI Interrupts Zynq:

zynq>
zynq> cat /proc/interrupts
           CPU0       CPU1
 16:          0          0       GIC  27 Edge      gt
 17:          0          0       GIC  43 Level     ttc_clockevent
 18:       1499        868       GIC  29 Edge      twd
 21:         43          0       GIC  39 Level     f8007100.adc
 23:        453          0       GIC  57 Level     cdns-i2c
 26:        678          0       GIC  82 Level     xuartps
 28:          0          0       GIC  54 Level     eth0
 29:         71          0       GIC  56 Level     mmc0
 30:          0          0       GIC  45 Level     f8003000.dmac
 31:          0          0       GIC  46 Level     f8003000.dmac
 32:          0          0       GIC  47 Level     f8003000.dmac
 33:          0          0       GIC  48 Level     f8003000.dmac
 34:          0          0       GIC  49 Level     f8003000.dmac
 35:          0          0       GIC  72 Level     f8003000.dmac
 36:          0          0       GIC  73 Level     f8003000.dmac
 37:          0          0       GIC  74 Level     f8003000.dmac
 38:          0          0       GIC  75 Level     f8003000.dmac
 46:          0          0       GIC  41 Edge      f8005000.watchdog
 47:        370          0       GIC  61 Level     xilinx-pcie
 49:        370          0  Xilinx PCIe MSI   0 Edge      eth1
IPI0:          0          0  CPU wakeup interrupts
IPI1:          0          0  Timer broadcast interrupts
IPI2:        416        901  Rescheduling interrupts
IPI3:          0          0  Function call interrupts
IPI4:         34         19  Single function call interrupts
IPI5:          0          0  CPU stop interrupts
IPI6:          0          0  IRQ work interrupts
IPI7:          0          0  completion interrupts
Err:          0

MSI Interrupts Microblaze:

root@Xilinx-KC705-AXI-full-2016_1:~# cat /proc/interrupts [J
           CPU0
  1:      51474  Xilinx INTC   0-level     timer
  2:       1012  Xilinx INTC   2-level     xilinx-pcie
  3:       2889  Xilinx INTC   1-edge      uartlite
128:       1012  Xilinx PCIe MSI 128  eth0

Xilinx Wiki

Linux Soft PCIe Driver