Linux Soft PCIe Driver
This page gives an overview of AXI PCIe Root Complex driver for the Xilinx AXI PCIe Soft IP, which is available as part of the Zynq and Microblaze Linux distributions.
Table of Contents
AXI PCIe Soft IP
PCI Express (abbreviated as PCIe) is the newest bus standard designed to replace the old PCI/PCI-X and AGP standards. PCIe is used in servers, consumer, and industrial applicatios either as a motherboard-level interconnection to link peripherals or as an expansion card interface for add on boards. The latest PCIe IP released by XILINX (axi_pcie_v2_7) could be configured at hardware build time either as a root port or as an end point. The support for Root Port configuration has been integrated with the latest Zynq as well as Microblaze Linux Kernel. This PCIe core supports the Zynq and 7-series Device family. This page mainly discusses the Root Port driver and an example end point driver is demonstrated in TRD release with links pointed at the end of this page. For more information about AXI PCIe IP, please refer to documentation provided in the "Related Links" section.
Hardware Setup
The hardware setup is identically same for both Zynq and Microblaze platforms. The following figure shows how the Zynq board is configured to use the PCIe,
For Microblaze platform, the board should be replaced with a 7-series board.
FMC to PCIe Daughter card should be connected to one of the FMC connector slot either HPC/LPC based on the HW design. The FMC card information can be found at,
http://www.xilinx.com/products/boards_kits/fmc.htm
Tested End Point cards:
1. Broadcom PCIe NIC card
2. Realtek NIC card
Root Port Driver Configuration
Starting from 2015.4 there is unified Root Port driver for Zynq and Microblaze platforms. Earlier to 2015.4 there are two separate drivers for Zynq and Microblaze. The unified driver support PCIe MSI feature. The unified driver can be found here,
For Zynq/Microblaze:
https://github.com/Xilinx/linux-xlnx/blob/master/drivers/pci/host/pcie-xilinx.c
The PCI/PCIe subsystem support and Root Port driver is enabled by default in Zynq/Microblaze kernel configuration. The related code is always built with the kernel whether the hardware build includes PCIe IP or not. So, the user does not need to change anything in the configuration files to bring in PCIe support into Zynq/Microblaze kernel. There are other "optional" configuration items for configuring PCI/PCIe driver MSI support and those can be configured from "menuconfig" -> Bus options ->Message Signaled Interrupts (MSI and MSI-X).
End Point Driver Configuration
This page demonstrates the Root Port driver using Broadcom NIC Endpoint, for which NIC driver should be enabled in kernel as shown,
For other End point cards, ensure the respective driver is loaded into kernel.
Device Tree binding
The device tree node for AXI PCIe core will be automatically generated, if the core is configured in the HW design, using the Device Tree BSP.
Steps to generate device-tree is documented here,
http://www.wiki.xilinx.com/Build+Device+Tree+Blob
And a sample binding is shown below and the description of DT property is documented here
pci_express: axi-pcie@50000000 {
#address-cells = <3>;
#size-cells = <2>;
#interrupt-cells = <1>;
compatible = "xlnx,axi-pcie-host-1.00.a";
reg = < 0x50000000 0x10000000 >;
device_type = "pci";
interrupts = <0 52 4>;
interrupt-map-mask = <0 0 0 7>;
interrupt-map = <0 0 0 1 &pcie_intc 1>,
<0 0 0 2 &pcie_intc 2>,
<0 0 0 3 &pcie_intc 3>,
<0 0 0 4 &pcie_intc 4>;
ranges = < 0x02000000 0 0x60000000 0x60000000 0 0x10000000 >;
pcie_intc: interrupt-controller {
interrupt-controller;
#address-cells = <0>;
#interrupt-cells = <1>;
};
};
Test Procedure
- Load the linux onto Zynq ZC706 board/Microblaze KC705 board.
- After successful booting of the linux, will be able to see the Broadcom NIC endpoint driver is been probed. Run the command ‘lspci’ from the user prompt, which shows the the device id and manufacturer id of the Broadcom NIC card. This ensures the enumeration of the device is fine.
- The Broadcom NIC will show up as an eth0 interface in the linux. Run the command ‘ifconfig eth0 up’ which brings up the Ethernet interface. This step ensures all the memory transactions on the PCIe bus are working.
- Assign a IP address either static/dhcp. For dhcp, run ‘udhcpc –i eth0’ to lease an IP and then assign the address.
- Ping to a device of known IP address.
- Repeat the above steps with MSI supported linux image. This can be checked by observing the increase in MSI interrupts (cat /proc/interrupts).
Kernel Console Output
Driver Initialization for Zynq
xilinx-pcie 50000000.axi-pcie: PCIe Link is UP PCI host bridge /amba/axi-pcie@50000000 ranges: No bus range found for /amba/axi-pcie@50000000, using [bus 00-ff] MEM 0x60000000..0x6fffffff -> 0x60000000 xilinx-pcie 50000000.axi-pcie: PCI host bridge to bus 0000:00 pci_bus 0000:00: root bus resource [bus 00-ff] pci_bus 0000:00: root bus resource [mem 0x60000000-0x6fffffff] PCI: bus0: Fast back to back transfers disabled pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring PCI: bus1: Fast back to back transfers disabled pci 0000:00:00.0: BAR 0: no space for [mem size 0x40000000] pci 0000:00:00.0: BAR 0: failed to assign [mem size 0x40000000] pci 0000:00:00.0: BAR 8: assigned [mem 0x60000000-0x600fffff] pci 0000:01:00.0: BAR 0: assigned [mem 0x60000000-0x6000ffff 64bit] pci 0000:01:00.0: BAR 6: assigned [mem 0x60010000-0x6001ffff pref] pci 0000:00:00.0: PCI bridge to [bus 01] pci 0000:00:00.0: bridge window [mem 0x60000000-0x600fffff] dma-pl330 f8003000.dmac: Loaded driver for PL330 DMAC-241330 dma-pl330 f8003000.dmac: DBUFF-128x8bytes Num_Chans-8 Num_Peri-4 Num_Events-16 e0001000.serial: ttyPS0 at MMIO 0xe0001000 (irq = 26, base_baud = 3125000) is a xuartps console [ttyPS0] enabled [drm] Initialized drm 1.1.0 20060810 brd: module loaded loop: module loaded CAN device driver interface libphy: MACB_mii_bus: probed macb e000b000.ethernet eth0: Cadence GEM rev 0x00020118 at 0xe000b000 irq 28 (00:0a:35:00:01:22) macb e000b000.ethernet eth0: attached PHY driver [Marvell 88E1116R] (mii_bus:phy_addr=e000b000.etherne:07, irq=-1) tg3.c:v3.137 (May 11, 2014) pci 0000:00:00.0: enabling device (0140 -> 0142)
Driver Initialization for Microblaze
xilinx-pcie 10000000.pciex: PCIe Link is UP PCI host bridge /amba_pl/pciex@10000000 ranges: MEM 0x80000000..0x8fffffff -> 0x80000000 xilinx-pcie 10000000.pciex: PCI host bridge to bus 0000:00 pci_bus 0000:00: root bus resource [bus 00-ff] pci_bus 0000:00: root bus resource [mem 0x80000000-0x8fffffff] pci 0000:00:00.0: [10ee:0705] type 01 class 0x060400 pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0x7fffffff] pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring pci 0000:01:00.0: [14e4:1677] type 00 class 0x020000 pci 0000:01:00.0: reg 0x10: [mem 0x80000000-0x8000ffff 64bit] pci 0000:01:00.0: reg 0x30: [mem 0x90480000-0x9048ffff pref] pci 0000:01:00.0: PME# supported from D3hot D3cold pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01 pci 0000:00:00.0: BAR 0: no space for [mem size 0x80000000] pci 0000:00:00.0: BAR 0: failed to assign [mem size 0x80000000] pci 0000:00:00.0: BAR 8: assigned [mem 0x80000000-0x800fffff] pci 0000:01:00.0: BAR 0: assigned [mem 0x80000000-0x8000ffff 64bit] pci 0000:01:00.0: BAR 6: assigned [mem 0x80010000-0x8001ffff pref] pci 0000:00:00.0: PCI bridge to [bus 01] pci 0000:00:00.0: bridge window [mem 0x80000000-0x800fffff] ipmi message handler version 39.2 IPMI System Interface driver. ipmi_si: Unable to find any System Interface(s) Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled 30600000.serial: ttyUL0 at MMIO 0x30600000 (irq = 3, base_baud = 0) is a uartlite console [ttyUL0] enabled brd: module loaded loop: module loaded libphy: Fixed MDIO Bus: probed tg3.c:v3.137 (May 11, 2014) pci 0000:00:00.0: enabling device (0000 -> 0002)
Broadcom NIC card probing
tg3 0000:01:00.0: enabling device (0140 -> 0142) tg3 0000:01:00.0 eth1: Tigon3 [partno(BCM95751A519FLP) rev 4201] (PCI Express) MAC address 00:10:18:32:d2:a9 tg3 0000:01:00.0 eth1: attached PHY is 5750 (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[0]) tg3 0000:01:00.0 eth1: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] TSOcap[1] tg3 0000:01:00.0 eth1: dma_rwctrl[76180000] dma_mask[64-bit]
lspci output
Linux> lspci 00:00.0 PCI bridge: Xilinx Corporation Device 0705 01:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5751 Gigabit Ethernet PCI Express (rev 21)
Ethernet Interface
ifconfig -a eth0
Link encap:Ethernet HWaddr 00:10:18:32:D2:A9
BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
Interrupt:48
Testing of Ethernet interface
Linux> ifconfig eth0 192.168.1.10 Linux> tg3 0000:01:00.0: eth0: Link is up at 1000 Mbps, full duplex tg3 0000:01:00.0: eth0: Flow control is on for TX and on for RX Linux> Linux> ping -c 3 192.168.1.100 PING 192.168.1.100 (192.168.1.100): 56 data bytes 64 bytes from 192.168.1.100: seq=0 ttl=128 time=1.291 ms 64 bytes from 192.168.1.100: seq=1 ttl=128 time=0.537 ms 64 bytes from 192.168.1.100: seq=2 ttl=128 time=0.553 ms --- 192.168.1.100 ping statistics --- 3 packets transmitted, 3 packets received, 0% packet loss round-trip min/avg/max = 0.537/0.793/1.291 ms Linux>
MSI Interrupts Zynq:
zynq>
zynq> cat /proc/interrupts
CPU0 CPU1
16: 0 0 GIC 27 Edge gt
17: 0 0 GIC 43 Level ttc_clockevent
18: 1499 868 GIC 29 Edge twd
21: 43 0 GIC 39 Level f8007100.adc
23: 453 0 GIC 57 Level cdns-i2c
26: 678 0 GIC 82 Level xuartps
28: 0 0 GIC 54 Level eth0
29: 71 0 GIC 56 Level mmc0
30: 0 0 GIC 45 Level f8003000.dmac
31: 0 0 GIC 46 Level f8003000.dmac
32: 0 0 GIC 47 Level f8003000.dmac
33: 0 0 GIC 48 Level f8003000.dmac
34: 0 0 GIC 49 Level f8003000.dmac
35: 0 0 GIC 72 Level f8003000.dmac
36: 0 0 GIC 73 Level f8003000.dmac
37: 0 0 GIC 74 Level f8003000.dmac
38: 0 0 GIC 75 Level f8003000.dmac
46: 0 0 GIC 41 Edge f8005000.watchdog
47: 370 0 GIC 61 Level xilinx-pcie
49: 370 0 Xilinx PCIe MSI 0 Edge eth1
IPI0: 0 0 CPU wakeup interrupts
IPI1: 0 0 Timer broadcast interrupts
IPI2: 416 901 Rescheduling interrupts
IPI3: 0 0 Function call interrupts
IPI4: 34 19 Single function call interrupts
IPI5: 0 0 CPU stop interrupts
IPI6: 0 0 IRQ work interrupts
IPI7: 0 0 completion interrupts
Err: 0
MSI Interrupts Microblaze:
root@Xilinx-KC705-AXI-full-2016_1:~# cat /proc/interrupts [J
CPU0
1: 51474 Xilinx INTC 0-level timer
2: 1012 Xilinx INTC 2-level xilinx-pcie
3: 2889 Xilinx INTC 1-edge uartlite
128: 1012 Xilinx PCIe MSI 128 eth0
Related Links
© Copyright 2019 - 2022 Xilinx Inc. Privacy Policy