Axi Ethernet Linux driver for Microblaze, Zynq, Zynq Ultrascale+ MPSoC and Versal

HW IP features

AXI 1G/2.5G Ethernet Subsystem (PG138)

• Support for MII, GMII, RGMII, SGMII, and 1000BASE-X PHY interfaces
• Support for 1000BASE-X and SGMII over select Input/Output (I/O) low voltage differential signaling (LVDS)
• Support for pause frames for flow control
• Media Independent Interface Management 
• Ethernet Audio Video Bridging (AVB) support
• AXI4-Stream transmit/receive interface
• Support for 2.5G Ethernet
• IEEE Standard 1588 Support
• AXI4-Lite register interface

10 Gigabit Ethernet subsystem (PG157)

• Designed to 10 Gigabit Ethernet specification IEEE Standard 802.3-2012
• AXI4-Stream protocol support on client TX and RX interfaces
• Configured and monitored through an optional AXI4-Lite Management Data interface or using status and configuration vectors
• Supports 10GBASE-SR, -LR and -ER optical links in Zynq-7000, UltraScale™, Virtex-7, and Kintex-7 devices (LAN mode only)
• Supports 10GBASE-KR backplane links including Auto-Negotiation (AN), Training and Forward Error Correction (FEC)
• Supports Deficit Idle Count
• Comprehensive statistics gathering
• Supports 802.3 and 802.1Qbb flow control
• Supports VLAN and jumbo frames
• Custom Preamble mode
• Independent TX and RX Maximum Transmission Unit (MTU) frame length
• Supports high accuracy IEEE Standard 1588-2008 1-step and 2-step timestamping on a 10GBASE-R network interface

Features supported in the driver

• Support ethernet IPs- AXI 1G/2.5G Ethernet subsystem (PG138), 10G Ethernet subsystem(PG157), 10G Ethernet Subsystem(PG210), USXGMII(PG251), MRMAC, Switchable 1/10/25G.
• IEEE 1588 Support for 1G and legacy 10G MAC (PG157), 10G Ethernet subsystem and 25G Ethernet subsystem (PG210) and MRMAC
• Speed support for 10/100/1000 Mbps for 1G MAC
• 10G Base-R support for Legacy 10G MAC(PG157) and 10G MAC (PG210)
• 10G and 25G speed support for MRMAC
• Support for GMII/RGMII/SGMII/1000Base-X Phy Configurations
• Supports Independent 4K, 8K, 16K, or 32KB TX and RX frame buffer memory
• Support for common ethtool queries.
• NAPI support.
• Full/Partial Checksum offload support
• Support for Jumbo Frames
• Supports AXI DMA and AXI MCDMA dma configuration.
• Multi-queue support
• Switchable 1/10/25G IP supports  1G or 10G runtime switchable speeds in the driver.

Missing Features and Known Issues/Limitations in Driver

• The driver assumes that Axi Ethernet IP is connected to the DMA at the hardware level.
• The driver doesn't use dma engine framework and contains DMA programming sequence i.e doesn't use separate DMA driver. Hence compatibility string of axidma node (DTS) is set to a dummy device-tree property compatible = "xlnx,eth-dma";
• The driver doesn't support software time-stamping. It supports only hardware time-stamping. 
• PTP synchronization along with high speed traffic (iperf or netperf) is not supported as under heavy load,  timestamp in FIFO and DMA data in BD is expected to go out of sync and remain so until the interface is reset.
  • This is a limitation on Soft Ethernet designs which have HW timestamp FIFOs. This issue is not relevant to designs with inline HW timestamping.
• No support for fixed-link.
• For 1588 testing the Current driver assumes that AXI Stream FIFO is connected to the MAC TX Time stamp Stream interface at the design level.  For axiethernet 1G/10G subsystem only 2-step PTP is supported.
• 10G/25G and USXGMII configurations do not support dynamic link status/change in the background as there is no external PHY using PHY framework.
• Pause frame solution is not supported and hence there could be RX overruns errors in bidirectional throughput.
• The driver supports MCDMA using kernel config i.e CONFIG_AXIENET_HAS_MCDMA option. So in multi-instance scenario driver will only support a single DMA type i.e 1G + MCDMA and 10G + MCDMA.
• The driver doesn't support extended multicast and VLAN support. Limited validation of multicast and vlan support. 
• Runtime Switchable mode.
• MRMAC speeds 40G/50G/100G are not supported yet.
• MRMAC multi-lane support is not independent because if common GT reset logic exists in subsystem.
• AXI Ethernet is only validated on Zynq (1G), ZU+ (1G/2.5G), Versal (1G) and MB (US+ and 7 series, 1G) platforms.
• USXGMII driver is only validated on ZU+ based platforms.
• XXV driver is only validated on ZU+, RFSoC and Versal based platforms.
• MRMAC IP/driver is only available and validated on Versal based platforms
• On versal support is limited to AXI 1G Ethernet subsystem (without PTP, 2.5G support), XXV Ethernet subsystem (without PTP, validated at 25G) and MRMAC.
• Switchable 1/10/25G IP support is only validated at 1G and 10G on Zynq Ultrascale+ MPSoC. Dynamic speed switching is possible only if host switches speed and can’t be enforced from driver till 2024.1 release. Speed switching using ethtool is introduced after 2024.1 release.

• Multiple TX and RX channel in MCDMA have common configuration and reset registers and hence cannot be used independently by multiple MACs. For ex., if XXV Ethernet instance 1 uses channel 0-4 of MCDMA and then XXV Ethernet instance 2 uses channels 5-15, then resets during driver initialization and error management effect all channels and both instance need to use common registers. Due to this limitation, multiple MACs cannot be used with a single MCDMA.
• Ethtool or link/speed information support is available for AXI Ethernet and XXV only, not for MRMAC and 1G/10G/25G switching IP; this limitation is due to phylink framework and non-presence of an external PHY or PCS. However, there is a plan to add this support in an upcoming release.
• RSFEC configuration is not supported or tested in the driver.
• AXI Ethernet or any other high speed Ethernet does not have any module support with MCDMA. There are some dependencies.

Important AR links

• AXI Ethernet driver in specific MCDMA configuration throws swiotlb full error with jumbo frames. Please refer to 2020.x AR-75128.
• Default DTG generation for XXV Ethernet designs fails on 2020.2. Please refer to AR-76029 and AR-76457.

Kernel Configuration

Device-tree

axi_ethernet_eth_buf: ethernet@40c00000 {
axistream-connected = <&axi_dma_1>;
axistream-control-connected = <&axi_dma_1>;
clock-frequency = <100000000>;
clocks = <&clk_bus_0>;
compatible = "xlnx,axi-ethernet-1.00.a";
device_type = "network";
interrupt-parent = <&microblaze_1_axi_intc>;
interrupts = <4 2>;
reg = <0x40c00000 0x40000>;
xlnx,phy-type = <0x4>;
xlnx,phyaddr = <0x1>;
xlnx,rxcsum = <0x0>;
xlnx,rxmem = <0x8000>;
xlnx,txcsum = <0x0>;
phy-handle = <&phy0>;
mdio {
#address-cells = <1>;
#size-cells = <0>;
phy0: phy@7 {
device_type = "ethernet-phy";
reg = <7>;
};
};
};

Soft Ethernet MAC(1G, legacy 10G or 10G/25G MAC, MRMAC) Configured with MCDMA

 xxv_ethernet_0: ethernet@80020000 {
	axistream-connected = <&axi_dma_hier_axi_mcdma_0>;
	axistream-control-connected = <&axi_dma_hier_axi_mcdma_0>;
	clock-frequency = <100000000>;
	clock-names = "rx_core_clk_0", "dclk", "s_axi_aclk_0";
	clocks = <&misc_clk_0>, <&clk 72>, <&clk 71>;
	compatible = "xlnx,xxv-ethernet-2.5", "xlnx,xxv-ethernet-1.0";
	device_type = "network";
	local-mac-address = [00 0a 35 00 00 00];
	phy-mode = "base-r";
	reg = <0x0 0x80020000 0x0 0x10000>;
	xlnx = <0x0>;
	xlnx,add-gt-cntrl-sts-ports = <0x0>;
	xlnx,anlt-clk-in-mhz = <0x64>;
	xlnx,axis-tdata-width = <0x40>;
	xlnx,axis-tkeep-width = <0x7>;
	xlnx,base-r-kr = "BASE-R";
	xlnx,channel-ids = "1","2","3","4","5","6","7","8","9","a","b","c","d","e","f","10";
	xlnx,clocking = "Asynchronous";
	xlnx,core = "Ethernet MAC+PCS/PMA 64-bit";
	xlnx,data-path-interface = "AXI Stream";
	xlnx,enable-datapath-parity = <0x0>;
	xlnx,enable-pipeline-reg = <0x0>;
	xlnx,enable-preemption = <0x0>;
	xlnx,enable-preemption-fifo = <0x0>;
	xlnx,enable-rx-flow-control-logic = <0x0>;
	xlnx,enable-time-stamping = <0x1>;
	xlnx,enable-tx-flow-control-logic = <0x0>;
	xlnx,enable-vlane-adjust-mode = <0x0>;
	xlnx,family-chk = "zynquplus";
	xlnx,fast-sim-mode = <0x0>;
	xlnx,gt-diffctrl-width = <0x4>;
	xlnx,gt-drp-clk = "100.00";
	xlnx,gt-group-select = "Quad X0Y0";
	xlnx,gt-location = <0x1>;
	xlnx,gt-ref-clk-freq = "156.25";
	xlnx,gt-type = "GTH";
	xlnx,include-auto-neg-lt-logic = "None";
	xlnx,include-axi4-interface = <0x1>;
	xlnx,include-fec-logic = <0x0>;
	xlnx,include-rsfec-logic = <0x0>;
	xlnx,include-shared-logic = <0x1>;
	xlnx,include-user-fifo = <0x1>;
	xlnx,lane1-gt-loc = "X0Y4";
	xlnx,lane2-gt-loc = "NA";
	xlnx,lane3-gt-loc = "NA";
	xlnx,lane4-gt-loc = "NA";
	xlnx,line-rate = <0xa>;
	xlnx,mii-ctrl-width = <0x4>;
	xlnx,mii-data-width = <0x20>;
	xlnx,num-of-cores = <0x1>;
	xlnx,num-queues = /bits/ 16 <0x10>;
	xlnx,ptp-clocking-mode = <0x0>;
	xlnx,ptp-operation-mode = <0x2>;
	xlnx,runtime-switch = <0x0>;
	xlnx,rxmem = <0x40000>;
	xlnx,switch-1-10-25g = <0x0>;
	xlnx,tx-latency-adjust = <0x0>;
	xlnx,tx-total-bytes-width = <0x4>;
	xlnx,xgmii-interface = <0x1>;
	interrupt-names = "mm2s_ch1_introut", "mm2s_ch2_introut", "mm2s_ch3_introut", "mm2s_ch4_introut", "mm2s_ch5_introut", "mm2s_ch6_introut", "mm2s_ch7_introut", "mm2s_ch8_introut", "mm2s_ch9_introut", "mm2s_ch10_introut", "mm2s_ch11_introut", "mm2s_ch12_introut", "mm2s_ch13_introut", "mm2s_ch14_introut", "mm2s_ch15_introut", "mm2s_ch16_introut", "s2mm_ch1_introut", "s2mm_ch2_introut", "s2mm_ch3_introut", "s2mm_ch4_introut", "s2mm_ch5_introut", "s2mm_ch6_introut", "s2mm_ch7_introut", "s2mm_ch8_introut", "s2mm_ch9_introut", "s2mm_ch10_introut", "s2mm_ch11_introut", "s2mm_ch12_introut", "s2mm_ch13_introut", "s2mm_ch14_introut", "s2mm_ch15_introut", "s2mm_ch16_introut";
	interrupt-parent = <&gic>;
	interrupts = <0 89 4 0 89 4 0 89 4 0 89 4 0 89 4 0 89 4 0 89 4 0 89 4 0 89 4 0 89 4 0 89 4 0 89 4 0 89 4 0 89 4 0 89 4 0 89 4 0 90 4 0 90 4 0 90 4 0 90 4 0 90 4 0 90 4 0 90 4 0 90 4 0 90 4 0 90 4 0 90 4 0 90 4 0 90 4 0 90 4 0 90 4 0 90 4>;

	xxv_ethernet_0_mdio: mdio {
			#address-cells = <1>;
			#size-cells = <0>;
	};
};

• The driver supports random Queue/Channel selection.

• The driver supports channel observer feature through sysfs. This custom feature is useful in multi-core (Observer) system where MCDMA is a shared resource for all cores. MCDMA IP supports a maximum of six cores and 16 Channels can be distributed across each core as a static configuration. The Channel Observer is available for each group and provides the status about the channels in a group being serviced.
• The driver supports per channel weight configuration through sysfs. This custom feature specifies the channel weight i.e number of packets to be sent in one iteration.

• The driver supports Linux multiqueue networking. It uses the alloc_etherdev_mq() function to allocate the subqueues for the device.

  The userspace command 'tc,' part of the iproute2 package, is used to configure qdiscs. To add the MULTIQ qdisc assuming the device is called eth0, run the following command: 

  # tc qdisc add dev eth0 root handle 1: multiq

  The qdisc will allocate the number of bands to equal the number of queues that the device reports, and bring the qdisc online.

  Assuming eth0 has 4 Tx queues, the band mapping would look like:

  band 0 => queue 0
  band 1 => queue 1
  band 2 => queue 2
  band 3 => queue 3

         The behavior of tc filters remains the same. However, a new tc action, skbedit, has been added.
         Assuming we want to route all traffic to a specific host, for example 192.168.0.3, through a specific queue we could use this action and establish a filter such as:

         # tc filter add dev eth0 parent 1: protocol ip prio 1 u32 match ip dst 192.168.0.3 action skbedit queue_mapping 3

        For details refer Linux kernel Documentation/networking/multiqueue.txt

Related device tree information

IEEE 1588 Support

     axi_ethernet_eth_buf: ethernet@40c00000 {
            axistream-connected = <&axi_dma_1>;
            axififo-connected = <&axi_fifo_0>;
            clock-frequency = <100000000>;
            clocks = <&clk_bus_0>;
            compatible = "xlnx,axi-ethernet-1.00.a";
            device_type = "network";
            interrupt-parent = <&microblaze_1_axi_intc>;
            interrupts = <4 2>;
            reg = <0x40c00000 0x40000>;
            xlnx,phy-type = <0x4>;
            xlnx,phyaddr = <0x1>;
            xlnx,rxcsum = <0x0>;
            xlnx,rxmem = <0x8000>;
            xlnx,txcsum = <0x0>;
            xlnx,txmem = <0x8000>;
            phy-handle = <&phy0>;
            mdio {
              #address-cells = <1>;
              #size-cells = <0>;
                  phy0: phy@7 {
                      compatible = "marvell,88e1111";
                     device_type = "ethernet-phy";
                     reg = <7>;
                };
            };
        };

MRMAC 1588 support is tested with Xilinx 1588 Timer-Syncer block details of which can be found here:

https://github.com/Xilinx/linux-xlnx/blob/master/Documentation/devicetree/bindings/ptp/ptp-xilinx.yaml

https://github.com/Xilinx/linux-xlnx/blob/master/drivers/ptp/ptp_xilinx.c

Performance

1G Ethernet with AXIDMA

NOTE- There is ~10% drop (compared to 2019.2) in performance for 1500 MTU.
The drop is due to enable CONFIG_OPTIMIZE_INLINING forcibly” commit in linux kernel.

Kernel and networking stack is full of inline functions and it could be some unoptimized
inline function (could also be dependent on gcc version) leading to a performance drop.

The performance drop is observed on GEM and Xilinx Axi Ethernet MAC’s on Zynq

The plan is to document the performance drop on zynq and initiate the discussion with
the mainline community so that it is analyzed by respective kernel maintainers.

67.53

453

52.86

10G Ethernet with AXIMCDMA

• Set Ethernet MCDMA TX interrupt affinity to core-1

• Run iperf servers on ZynqMP (core2 and core3)

• CPU Utilization reporting

• Run iperf servers on the remote host

Diagnostic and Protocol Tests

Stress Test

ping -f localhost

Performance Test

1588 Testing

ptp4l implements the PTP boundary clock and ordinary clock. When hardware time stamping is enabled, ptp4l synchronizes the PTP hardware clock to the master clock. phc2sys can be used to synchronize the system clock to the PTP hardware clock on the network interface card (NIC).

• None

• Fix xxv mac padding issue.
• Add support for USXGMII IP.
• Update interrupt-names property.
• Code cleanup of tx with no DRE.
• Fix remove() bug for XXV no MDIO.

• None

• Added Support for Ethernet MCDMA Configuration
• Added Support for 1588 in buffered mode configuration.
• Fixed race condition in the transmit path
• Fixed race condition in the random queue selection for Ethernet MCDMA configuration

• None

• Added Support for 10G/25G MAC (PG210)
• Added Support for 2.5G MAC
• Fixed issues with the without DRE DMA based design on zynqMP SOC
• Added Clock Support
• Fixed bug in the rx reject interrupt handling.

• None

• Fix kernel crash on the 64-bit platform reported while testing 1588 on Zynq UltraScale+ MPSOC.

Related Links