Linux AXI Ethernet driver

Axi Ethernet Linux driver for Microblaze, Zynq, Zynq Ultrascale+ MPSoC and Versal


Introduction

Table of Contents

This page gives an overview of Axi Ethernet Linux driver which is available as part of the Linux distribution.
Paths, files, links and documentation on this page are given relative to the Linux kernel source tree.

HW IP features

AXI 1G/2.5G Ethernet Subsystem (PG138)

  • Support for MII, GMII, RGMII, SGMII, and 1000BASE-X PHY interfaces
  • Support for 1000BASE-X and SGMII over select Input/Output (I/O) low voltage differential signaling (LVDS)
  • Support for pause frames for flow control
  • Media Independent Interface Management 
  • Ethernet Audio Video Bridging (AVB) support
  • AXI4-Stream transmit/receive interface
  • Support for 2.5G Ethernet
  • IEEE Standard 1588 Support
  • AXI4-Lite register interface

10 Gigabit Ethernet subsystem (PG157)

  • Designed to 10 Gigabit Ethernet specification IEEE Standard 802.3-2012
  • AXI4-Stream protocol support on client TX and RX interfaces
  • Configured and monitored through an optional AXI4-Lite Management Data interface or using status and configuration vectors
  • Supports 10GBASE-SR, -LR and -ER optical links in Zynq-7000, UltraScale™, Virtex-7, and Kintex-7 devices (LAN mode only)
  • Supports 10GBASE-KR backplane links including Auto-Negotiation (AN), Training and Forward Error Correction (FEC)
  • Supports Deficit Idle Count
  • Comprehensive statistics gathering
  • Supports 802.3 and 802.1Qbb flow control
  • Supports VLAN and jumbo frames
  • Custom Preamble mode
  • Independent TX and RX Maximum Transmission Unit (MTU) frame length
  • Supports high accuracy IEEE Standard 1588-2008 1-step and 2-step timestamping on a 10GBASE-R network interface

10G/25G Ethernet Subsystem (PG210)

  • Designed to the Ethernet requirements for 10/25 Gb/s operation specified by IEEE 802.3 Clause 49, IEEE 802.3by, and the 25G Ethernet Consortium
  • Includes complete Ethernet MAC and PCS/PMA functions or standalone PCS/PMA for 25Gb/s operation
  • Includes complete Ethernet MAC and PCS/PMA functions, standalone MAC or standalone PCS/PMA for 10 Gb/s operation
  • Simple packet-oriented user interface
  • Comprehensive statistics gathering
  • Status signals for all major functional indicators
  • BASE-R PCS sublayer operating at 10.3125Gb/s or 25.78125Gb/s
  • Optional clause 74 BASE-KR FEC sublayer
  • Optional Auto-Negotiation
  • Optional clause 108 25G Reed-Solomon Forward Error Correction (RS-FEC) sublayer
  • Custom Preamble mode
  • Optional IEEE 1588 1-step and 2-step timestamping
  • Runtime switchable between 10G and 25G

USXGMII Ethernet Subsystem (PG251)

  • Designed to meet the USXGMII specification EDCS-1467841 revision 1.4
  • Supports 10M, 100M, 1G, 2.5G, 5G, or 10GE data rates over a 10.3125 Gb/s link
  • Both media access control (MAC) and PCS/PMA functions are included
  • Code replication/removal of lower rates onto the 10GE link
  • Rate adaption onto user clock domain
  • Low data path latency
  • 32-bit AXI4-Stream interface for datapath
  • Optional AXI4-Lite register interface
  • Support for 802.3x and priority-based pause operation
  • Detailed statistics gathering
  • Support for custom preambles
  • Supports deficit idle count (DIC)

MRMAC Ethernet Subsystem (PG314)

  • Hardened Ethernet IP block on Versal.
  • Multi rate Ethernet MAC supporting speeds from 10G to 100G.
    • The driver supports 25GE and 10GE with 1 to 4 lanes.
  • Hardened IP (to be used with Soft DMA and logic for driver subsystems)
  • High performance, low latency.
  • Low data path latency
  • User-side AXI4-Stream interface for data
  • AXI4-Lite register interface
  • Detailed statistics gathering
  • IEEE1588 support

Switchable 1/10/25G IP support (PG292)

  • Designed to the Ethernet requirements for 1/10 Gb/s operation specified by IEEE 802.3 Clause 49 or Clause 36
  • Runtime switchable Ethernet MAC and PCS/ PMA functions for 1/10/25 Gb/s operation.
  • Supports only GTHE3/GTYE3 and GTHE4/GTYE4 transceiver supported devices
  • Include Auto Negotiation (Clause-73)
  • Enable Link Training (Clause 72)


Features supported in the driver

  • Support ethernet IPs- AXI 1G/2.5G Ethernet subsystem (PG138), 10G Ethernet subsystem(PG157), 10G Ethernet Subsystem(PG210), USXGMII(PG251), MRMAC, Switchable 1/10/25G.
  • IEEE 1588 Support for 1G and legacy 10G MAC (PG157), 10G Ethernet subsystem and 25G Ethernet subsystem (PG210) and MRMAC
  • Speed support for 10/100/1000 Mbps for 1G MAC
  • 10G Base-R support for Legacy 10G MAC(PG157) and 10G MAC (PG210)
  • 10G and 25G speed support for MRMAC
  • Support for GMII/RGMII/SGMII/1000Base-X Phy Configurations
  • Supports Independent 4K, 8K, 16K, or 32KB TX and RX frame buffer memory
  • Support for common ethtool queries.
  • NAPI support.
  • Full/Partial Checksum offload support
  • Support for Jumbo Frames
  • Supports AXI DMA and AXI MCDMA dma configuration.
  • Multi-queue support
  • Switchable 1/10/25G IP supports  1G or 10G runtime switchable speeds in the driver.

Missing Features and Known Issues/Limitations in Driver

  • The driver assumes that Axi Ethernet IP is connected to the DMA at the hardware level.
  • The driver doesn't use dma engine framework and contains DMA programming sequence i.e doesn't use separate DMA driver. Hence compatibility string of axidma node (DTS) is set to a dummy device-tree property compatible = "xlnx,eth-dma";
  • The driver doesn't support software time-stamping. It supports only hardware time-stamping. 
  • PTP synchronization along with high speed traffic (iperf or netperf) is not supported as under heavy load,  timestamp in FIFO and DMA data in BD is expected to go out of sync and remain so until the interface is reset.
    • This is a limitation on Soft Ethernet designs which have HW timestamp FIFOs. This issue is not relevant to designs with inline HW timestamping.
  • No support for fixed-link.
  • For 1588 testing the Current driver assumes that AXI Stream FIFO is connected to the MAC TX Time stamp Stream interface at the design level.  For axiethernet 1G/10G subsystem only 2-step PTP is supported.
  • 10G/25G and USXGMII configurations do not support dynamic link status/change in the background as there is no external PHY using PHY framework.
  • Pause frame solution is not supported and hence there could be RX overruns errors in bidirectional throughput.
  • The driver supports MCDMA using kernel config i.e CONFIG_AXIENET_HAS_MCDMA option. So in multi-instance scenario driver will only support a single DMA type i.e 1G + MCDMA and 10G + MCDMA.
  • The driver doesn't support extended multicast and VLAN support. Limited validation of multicast and vlan support. 
  • Runtime Switchable mode.
  • MRMAC speeds 40G/50G/100G are not supported yet.
  • MRMAC multi-lane support is not independent because if common GT reset logic exists in subsystem.
  • AXI Ethernet is only validated on Zynq (1G), ZU+ (1G/2.5G), Versal (1G) and MB (US+ and 7 series, 1G) platforms.
  • USXGMII driver is only validated on ZU+ based platforms.
  • XXV driver is only validated on ZU+, RFSoC and Versal based platforms.
  • MRMAC IP/driver is only available and validated on Versal based platforms
  • On versal support is limited to AXI 1G Ethernet subsystem (without PTP, 2.5G support), XXV Ethernet subsystem (without PTP, validated at 25G) and MRMAC.
  • Switchable 1/10/25G IP support is only validated at 1G and 10G on Zynq Ultrascale+ MPSoC. Dynamic speed switching is possible only if host switches speed and can’t be enforced from driver till 2024.1 release. Speed switching using ethtool is introduced after 2024.1 release.

NOTE: Relevant missing Features and Known Issues/Limitations in IP:

  • Multiple TX and RX channel in MCDMA have common configuration and reset registers and hence cannot be used independently by multiple MACs. For ex., if XXV Ethernet instance 1 uses channel 0-4 of MCDMA and then XXV Ethernet instance 2 uses channels 5-15, then resets during driver initialization and error management effect all channels and both instance need to use common registers. Due to this limitation, multiple MACs cannot be used with a single MCDMA.
  • Ethtool or link/speed information support is available for AXI Ethernet and XXV only, not for MRMAC and 1G/10G/25G switching IP; this limitation is due to phylink framework and non-presence of an external PHY or PCS. However, there is a plan to add this support in an upcoming release.
  • RSFEC configuration is not supported or tested in the driver.
  • AXI Ethernet or any other high speed Ethernet does not have any module support with MCDMA. There are some dependencies.

Missing Features and Known Issues/Limitations in SDT

  • MRMAC connected IP properties (such as GT control GPIOs) and Timer-Syncer IP properties (including associated timestamping related nodes, if any) are not automatically generated in SDT flow. This will be updated in a future release.
  • AXI Ethernet driver in specific MCDMA configuration throws swiotlb full error with jumbo frames. Please refer to 2020.x AR-75128.
  • Default DTG generation for XXV Ethernet designs fails on 2020.2. Please refer to AR-76029 and AR-76457.

Kernel Configuration

The following config options should be enabled in order to build the Axi Ethernet driver
CONFIG_ETHERNET
CONFIG_NET_VENDOR_XILINX
CONFIG_XILINX_AXI_EMAC
CONFIG_AXIENET_HAS_MCDMA (Select this option In the design if Axi Ethernet is configured with Axi MCDMA)
CONFIG_XILINX_PHY (For testing SGMII/1000Base-x Configuration with PCS/PMA Core)




Device-tree

For more details on phy bindings please refer to https://github.com/Xilinx/linux-xlnx/blob/master/Documentation/devicetree/bindings/net/ethernet-phy.yaml
axi_ethernet_eth_buf: ethernet@40c00000 {
axistream-connected = <&axi_dma_1>;
axistream-control-connected = <&axi_dma_1>;
clock-frequency = <100000000>;
clocks = <&clk_bus_0>;
compatible = "xlnx,axi-ethernet-1.00.a";
device_type = "network";
interrupt-parent = <&microblaze_1_axi_intc>;
interrupts = <4 2>;
reg = <0x40c00000 0x40000>;
xlnx,phy-type = <0x4>;
xlnx,phyaddr = <0x1>;
xlnx,rxcsum = <0x0>;
xlnx,rxmem = <0x8000>;
xlnx,txcsum = <0x0>;
phy-handle = <&phy0>;
mdio {
#address-cells = <1>;
#size-cells = <0>;
phy0: phy@7 {
device_type = "ethernet-phy";
reg = <7>;
};
};
};

Soft Ethernet MAC(1G, legacy 10G or 10G/25G MAC, MRMAC) Configured with MCDMA

When Axi Ethernet (10G/25G MAC) configured with MCDMA device-tree node will be like below (to make use of internal MCDMA driver)
 xxv_ethernet_0: ethernet@80020000 {
	axistream-connected = <&axi_dma_hier_axi_mcdma_0>;
	axistream-control-connected = <&axi_dma_hier_axi_mcdma_0>;
	clock-frequency = <100000000>;
	clock-names = "rx_core_clk_0", "dclk", "s_axi_aclk_0";
	clocks = <&misc_clk_0>, <&clk 72>, <&clk 71>;
	compatible = "xlnx,xxv-ethernet-2.5", "xlnx,xxv-ethernet-1.0";
	device_type = "network";
	local-mac-address = [00 0a 35 00 00 00];
	phy-mode = "base-r";
	reg = <0x0 0x80020000 0x0 0x10000>;
	xlnx = <0x0>;
	xlnx,add-gt-cntrl-sts-ports = <0x0>;
	xlnx,anlt-clk-in-mhz = <0x64>;
	xlnx,axis-tdata-width = <0x40>;
	xlnx,axis-tkeep-width = <0x7>;
	xlnx,base-r-kr = "BASE-R";
	xlnx,channel-ids = "1","2","3","4","5","6","7","8","9","a","b","c","d","e","f","10";
	xlnx,clocking = "Asynchronous";
	xlnx,core = "Ethernet MAC+PCS/PMA 64-bit";
	xlnx,data-path-interface = "AXI Stream";
	xlnx,enable-datapath-parity = <0x0>;
	xlnx,enable-pipeline-reg = <0x0>;
	xlnx,enable-preemption = <0x0>;
	xlnx,enable-preemption-fifo = <0x0>;
	xlnx,enable-rx-flow-control-logic = <0x0>;
	xlnx,enable-time-stamping = <0x1>;
	xlnx,enable-tx-flow-control-logic = <0x0>;
	xlnx,enable-vlane-adjust-mode = <0x0>;
	xlnx,family-chk = "zynquplus";
	xlnx,fast-sim-mode = <0x0>;
	xlnx,gt-diffctrl-width = <0x4>;
	xlnx,gt-drp-clk = "100.00";
	xlnx,gt-group-select = "Quad X0Y0";
	xlnx,gt-location = <0x1>;
	xlnx,gt-ref-clk-freq = "156.25";
	xlnx,gt-type = "GTH";
	xlnx,include-auto-neg-lt-logic = "None";
	xlnx,include-axi4-interface = <0x1>;
	xlnx,include-fec-logic = <0x0>;
	xlnx,include-rsfec-logic = <0x0>;
	xlnx,include-shared-logic = <0x1>;
	xlnx,include-user-fifo = <0x1>;
	xlnx,lane1-gt-loc = "X0Y4";
	xlnx,lane2-gt-loc = "NA";
	xlnx,lane3-gt-loc = "NA";
	xlnx,lane4-gt-loc = "NA";
	xlnx,line-rate = <0xa>;
	xlnx,mii-ctrl-width = <0x4>;
	xlnx,mii-data-width = <0x20>;
	xlnx,num-of-cores = <0x1>;
	xlnx,num-queues = /bits/ 16 <0x10>;
	xlnx,ptp-clocking-mode = <0x0>;
	xlnx,ptp-operation-mode = <0x2>;
	xlnx,runtime-switch = <0x0>;
	xlnx,rxmem = <0x40000>;
	xlnx,switch-1-10-25g = <0x0>;
	xlnx,tx-latency-adjust = <0x0>;
	xlnx,tx-total-bytes-width = <0x4>;
	xlnx,xgmii-interface = <0x1>;
	interrupt-names = "mm2s_ch1_introut", "mm2s_ch2_introut", "mm2s_ch3_introut", "mm2s_ch4_introut", "mm2s_ch5_introut", "mm2s_ch6_introut", "mm2s_ch7_introut", "mm2s_ch8_introut", "mm2s_ch9_introut", "mm2s_ch10_introut", "mm2s_ch11_introut", "mm2s_ch12_introut", "mm2s_ch13_introut", "mm2s_ch14_introut", "mm2s_ch15_introut", "mm2s_ch16_introut", "s2mm_ch1_introut", "s2mm_ch2_introut", "s2mm_ch3_introut", "s2mm_ch4_introut", "s2mm_ch5_introut", "s2mm_ch6_introut", "s2mm_ch7_introut", "s2mm_ch8_introut", "s2mm_ch9_introut", "s2mm_ch10_introut", "s2mm_ch11_introut", "s2mm_ch12_introut", "s2mm_ch13_introut", "s2mm_ch14_introut", "s2mm_ch15_introut", "s2mm_ch16_introut";
	interrupt-parent = <&gic>;
	interrupts = <0 89 4 0 89 4 0 89 4 0 89 4 0 89 4 0 89 4 0 89 4 0 89 4 0 89 4 0 89 4 0 89 4 0 89 4 0 89 4 0 89 4 0 89 4 0 89 4 0 90 4 0 90 4 0 90 4 0 90 4 0 90 4 0 90 4 0 90 4 0 90 4 0 90 4 0 90 4 0 90 4 0 90 4 0 90 4 0 90 4 0 90 4 0 90 4>;

	xxv_ethernet_0_mdio: mdio {
			#address-cells = <1>;
			#size-cells = <0>;
	};
};


When Soft Ethernet MAC configured with MCDMA, The driver supports several features of the MCDMA- 
  • The driver supports random Queue/Channel selection.
           Assume in vivado design MCDMA is configured for 16 channels and user don't want Linux driver to use all the 16 channels.
           Below example, pl.dtsi ( AXI ethernet node) marks channel 2, 5 and 10 to be used by Linux driver. 
         
          xlnx,num-queues = <0x3>;
          xlnx,channel-ids = "2","5","10";
          
  • The driver supports channel observer feature through sysfs. This custom feature is useful in multi-core (Observer) system where MCDMA is a shared resource for all cores. MCDMA IP supports a maximum of six cores and 16 Channels can be distributed across each core as a static configuration. The Channel Observer is available for each group and provides the status about the channels in a group being serviced.
  • The driver supports per channel weight configuration through sysfs. This custom feature specifies the channel weight i.e number of packets to be sent in one iteration.
  • The driver supports Linux multiqueue networking. It uses the alloc_etherdev_mq() function to allocate the subqueues for the device.

    The userspace command 'tc,' part of the iproute2 package, is used to configure qdiscs. To add the MULTIQ qdisc assuming the device is called eth0, run the following command: 

    # tc qdisc add dev eth0 root handle 1: multiq

    The qdisc will allocate the number of bands to equal the number of queues that the device reports, and bring the qdisc online.

    Assuming eth0 has 4 Tx queues, the band mapping would look like:

    band 0 => queue 0
    band 1 => queue 1
    band 2 => queue 2
    band 3 => queue 3

         

         The behavior of tc filters remains the same. However, a new tc action, skbedit, has been added.
         Assuming we want to route all traffic to a specific host, for example 192.168.0.3, through a specific queue we could use this action and establish a filter such as:

         # tc filter add dev eth0 parent 1: protocol ip prio 1 u32 match ip dst 192.168.0.3 action skbedit queue_mapping 3

        For details refer Linux kernel Documentation/networking/multiqueue.txt

         
To support USXGMII + MCDMA, use above devicetree as reference and select:
xlnx,phy-type = <0x7>;

Related device tree information

For PHY related DT information, refer to
https://github.com/Xilinx/linux-xlnx/blob/master/Documentation/devicetree/bindings/net/ethernet-phy.yaml

When selecting phy specific settings, make sure to mention interface type, speed (if limited/fixed) and phy address properties.
PHY/Converter devices that may be used with this MAC:
-> Xilinx GMII2RGMII converter (https://github.com/Xilinx/linux-xlnx/blob/master/Documentation/devicetree/bindings/net/xlnx%2Cgmii-to-rgmii.yaml)
-> Xilinx PCS PMA PHY (handled internal to this Ethernet driver)
For information on DT bindings when using this Ethernet driver with AXI DMA driver from dma framework, please refer to the following:
See "AXI 1G/2.5G Ethernet Subsystem + AXIDMA without "axistream-connected" property"
https://github.com/Xilinx/linux-xlnx/blob/master/Documentation/devicetree/bindings/net/xlnx%2Caxi-ethernet.yaml#L268

IEEE 1588 Support

The Existing Axi Ethernet driver in the Xilinx git hub supports 1588 for 1G MAC and legacy 10G MAC and 10G/25G MAC
It does timestamp at the MAC level. 1588 is supported in 7-series and Zynq.
The Current driver assumes that AXI Stream FIFO is connected to the MAC TX Time stamp Stream interface at the design level.

Note:
In order to test the hardware platform, a 1588 timestamp capable timer is required at the h/w level.

Prerequisite for 1588 testing:
Need to enable XILINX_AXI_EMAC_HWTSTAMP in the kernel config which does the timestamping at the MAC level (Tx/Rx)
Users should write their own timer driver for time stamping which does time adjustment/ freq adjustment.

For testing 1588 please refer to Testing tools section below.

1588 Device-tree Example Node
     axi_ethernet_eth_buf: ethernet@40c00000 {
            axistream-connected = <&axi_dma_1>;
            axififo-connected = <&axi_fifo_0>;
            clock-frequency = <100000000>;
            clocks = <&clk_bus_0>;
            compatible = "xlnx,axi-ethernet-1.00.a";
            device_type = "network";
            interrupt-parent = <&microblaze_1_axi_intc>;
            interrupts = <4 2>;
            reg = <0x40c00000 0x40000>;
            xlnx,phy-type = <0x4>;
            xlnx,phyaddr = <0x1>;
            xlnx,rxcsum = <0x0>;
            xlnx,rxmem = <0x8000>;
            xlnx,txcsum = <0x0>;
            xlnx,txmem = <0x8000>;
            phy-handle = <&phy0>;
            mdio {
              #address-cells = <1>;
              #size-cells = <0>;
                  phy0: phy@7 {
                      compatible = "marvell,88e1111";
                     device_type = "ethernet-phy";
                     reg = <7>;
                };
            };
        };

MRMAC 1588 support is tested with Xilinx 1588 Timer-Syncer block details of which can be found here:

https://github.com/Xilinx/linux-xlnx/blob/master/Documentation/devicetree/bindings/ptp/ptp-xilinx.yaml

https://github.com/Xilinx/linux-xlnx/blob/master/drivers/ptp/ptp_xilinx.c

SWITCHABLE 1/10/25G IP

When Axi Ethernet (1G/10G/25G MAC) is configured with DMA device-tree node will be like below

             axi_dma1: dma@80000000 {                                                  
                        #dma-cells = <0x01>;                                    
                        clock-names = "s_axi_lite_aclk\0m_axi_sg_aclk\0m_axi_mm2s_aclk\0m_axi_s2mm_aclk";
                        clocks = <0x03 0x47 0x03 0x47 0x3b 0x3b>;               
                        compatible = "not";                                     
                        interrupt-names = "mm2s_introut", "s2mm_introut";         
                        interrupt-parent = <0x04>;                              
                        interrupts = <0x00 0x59 0x04 0x00 0x5a 0x04>;           
                        reg = <0x00 0x80000000 0x00 0x10000>;                   
                        xlnx,addrwidth = [20];                                  
                        xlnx,include-sg;                                        
                        xlnx,sg-length-width = <0x10>;                          
                        status = "disabled";
						xlnx,include-dre;                                    
                        phandle = <0xac>;                                       
                                                                                
                        dma-channel@80000000 {                                  
                                compatible = "xlnx,axi-dma-mm2s-channel";       
                                dma-channels = <0x01>;                          
                                interrupts = <0x00 0x59 0x04>;                  
                                xlnx,datawidth = <0x40>;                        
                                xlnx,device-id = <0x00>;                        
                                xlnx,include-dre;                               
                        };                                                      
                                                                                
                        dma-channel@80000030 {                                  
                                compatible = "xlnx,axi-dma-s2mm-channel";       
                                dma-channels = <0x01>;                          
                                interrupts = <0x00 0x5a 0x04>;                  
                                xlnx,datawidth = <0x40>;                        
                                xlnx,device-id = <0x00>;                        
                                xlnx,include-dre;                               
                        };                                                      
                };

               ethernet_1_10_25g@80010000 {                                    
                        clock-names = "rx_core_clk_0", "rx_core_clk_1", "dclk", "s_axi_aclk_0", "s_axi_aclk_1";
                        clocks = <0x3b 0x3b 0x03 0x48 0x03 0x47 0x03 0x47>;         
                        compatible = "xlnx,ethernet-1-10-25g-2.7";              
                        reg = <0x00 0x80010000 0x00 0x10000>;                   
                        xlnx,add-gt-cntrl-sts-ports = <0x00>;                   
                        xlnx,anlt-clk-in-mhz = <0x4b>;                          
                        xlnx,axis-tdata-width = <0x40>;                         
                        xlnx,axis-tkeep-width = <0x07>;                         
                        xlnx,base-r-kr = "BASE-R";                              
                        xlnx,clocking = "Asynchronous";                         
                        xlnx,core = "Ethernet MAC+PCS/PMA 64-bit";              
                        xlnx,data-path-interface = "AXI Stream";                
                        xlnx,enable-pipeline-reg = <0x00>;                      
                        xlnx,enable-preemption = <0x00>;                        
                        xlnx,enable-rx-flow-control-logic = <0x00>;             
                        xlnx,enable-time-stamping = <0x00>;                     
                        xlnx,enable-tx-flow-control-logic = <0x00>;             
                        xlnx,enable-vlane-adjust-mode = <0x00>;                 
                        xlnx,family-chk = "zynquplus";                          
                        xlnx,fast-sim-mode = <0x00>;                            
                        xlnx,gt-drp-clk = "50.00";                              
                        xlnx,gt-group-select = "Quad X0Y0";                     
                        xlnx,gt-location = <0x01>;                              
                        xlnx,gt-ref-clk-freq = "156.25";                        
                        xlnx,gt-type = "GTH";                                   
                        xlnx,include-auto-neg-lt-logic = "Include AN Logic";    
                        xlnx,include-axi4-interface = <0x01>;                   
                        xlnx,include-fec-logic = <0x00>;                        
                        xlnx,include-lt-logic = "Include LT Logic";             
                        xlnx,include-rsfec-logic = <0x00>;                      
                        xlnx,include-shared-logic = <0x01>;                     
                        xlnx,include-statistics-counters = <0x01>;              
                        xlnx,include-system-timer-syncers = <0x00>;             
                        xlnx,include-user-fifo = <0x00>;                        
                        xlnx,lane1-gt-loc = "X1Y14";                            
                        xlnx,lane2-gt-loc = "X1Y15";                            
                        xlnx,lane3-gt-loc = "NA";                               
                        xlnx,lane4-gt-loc = "NA";                               
                        xlnx,line-rate = <0x0a>;                                
                        xlnx,num-of-cores = <0x02>;                             
                        xlnx,ptp-clocking-mode = <0x00>;                        
                        xlnx,ptp-operation-mode = <0x02>;                       
                        xlnx,runtime-switch = "1G / 10G";                       
                        xlnx,statistics-counters-size = <0x30>;                 
                        xlnx,statistics-regs-type = <0x00>;                     
                        xlnx,timer-format = <0x00>;                             
                        xlnx,tx-latency-adjust = <0x00>;                        
                        xlnx,tx-total-bytes-width = <0x04>;                     
                        xlnx,xgmii-interface = <0x01>;                          
                        axistream-connected = <&axi_dma1>;                           
                        axistream-control-connected = <&axi_dma1>;                   
                        local-mac-address = [00 0a 35 00 00 00];
                        interrupt-parent = <0x04>;                              
                        interrupt-names = "mm2s_introut", "s2mm_introut";         
                        interrupts = <0x00 0x59 0x04 0x00 0x5a 0x04>;           
                        xlnx,addrwidth = [20];                                  
                        phandle = <0xae>;                                       
                };                                                                



Performance

These benchmark performance numbers were obtained by connecting Xilinx boards to Linux PCs/server machines (Ubuntu/Red Hat Enterprise). 
The tools used are netperf or iperf (Refer to tool information below).
Performance benchmark numbers mentioned in below tables are for reference and dependent on multiple factors i.e setup , vivado design configuration  etc. 
NOTE: CPU utilization reported in below performance tables is an aggregate of all CPU's. i.e on ZynqMP platform, it reports combined utilization of all four A53 cores.

1G Ethernet with AXIDMA

Kernel version: 6.6
AXI 1G/2.5G Ethernet Subsystem : TX and RX full checksum offload enabled.

ZynqMP
Board: ZCU102 board (production silicon) + SFP Module

TCP (Mbps)UDP (Mbps)
MTUTXCPU(%)RXCPU(%)TXCPU(%)RXCPU(%)
1500

941

12.78

843

46.79

957

21.13

951

51.09

8192

989

2.34

989

7.26

779

2.98

992

5.86


Zynq
Board: ZC706 board + SFP Module

NOTE- There is ~10% drop (compared to 2019.2) in performance for 1500 MTU.
The drop is due to enable CONFIG_OPTIMIZE_INLINING forcibly” commit in linux kernel.

Kernel and networking stack is full of inline functions and it could be some unoptimized
inline function (could also be dependent on gcc version) leading to a performance drop.

The performance drop is observed on GEM and Xilinx Axi Ethernet MAC’s on Zynq

The plan is to document the performance drop on zynq and initiate the discussion with
the mainline community so that it is analyzed by respective kernel maintainers.


TCP (Mbps)UDP (Mbps)
MTUTXCPU(%)RXCPU(%)TXCPU(%)RXCPU(%)
1500

740

67.53

537

89.39

453
52.86

456

88.72

8192

977

60.69

732

50.26

743

36.10

643

50.32

Microblaze
Traditionally microblaze designs are not targeted for high performance applications so only functional sanity is done.

10G Ethernet with AXIMCDMA

Kernel version: 6.6

ZynqMP
Board: ZCU102 board (production silicon) + SFP Module


TCP (Gbps)UDP (Gbps)
MTUTXCPU(%)RXCPU(%)TXCPU(%)RXCPU(%)
15002.2950.911.7664.713.0399.931.6570.04
90005.052.353.5653.96.5165.094.6354.28
NOTE: In this design 1588 is not enabled.

Setup Details
Host setup: Dell System Precision Tower 7910 (0619)
Iperf: iperf 3-CURRENT (cJSON 1.5.2)
OS : Ubuntu 22.04.2 LTS (Linux kernel 5.15.0-87-generic)
NIC (10G Solarflare's SFN6322F Dual-Port 10GbE SFP+ Adapter) : Default

Performance benchmarking
NOTE: Better performance numbers are observed with linux v6.6 and flag -P 2 in iperf command.
Pre-requisites:
  • Set Ethernet MCDMA TX interrupt affinity to core-1
root@10g-mcdma-no1588-build:~# echo 2 > /proc/irq/xx/smp_affinity
  • Run iperf servers on ZynqMP (core2 and core3)
root@10g-mcdma-no1588-build:~# taskset -c 2 iperf3 -s -p 5101 &
root@10g-mcdma-no1588-build:~# taskset -c 3 iperf3 -s -p 5102 &
  • CPU Utilization reporting
root@10g-mcdma-no1588-build:~# ./mpstat -P ALL 1 50
  • Run iperf servers on the remote host
server:~# iperf3 -s -p 5101 & ; iperf3 -s -p 5102 & ; iperf3 -s -p 5103 & ; iperf3 -s -p 5104 &

Steps:





Test Procedure

Diagnostic and Protocol Tests

PING
This utility used to test the reachability of a host on an Internet Protocol(IP) network and to measure the round trip time for messages sent from the originating host to a destination computer

How to Run
ping <Remote IP Address>


Telnet
telnet <Server IP Address>


Pkt Generator
Please refer the below link for how to run and various options
https://www.kernel.org/doc/Documentation/networking/pktgen.txt

Stress Test

Iperf with option -d
Run iperf in dual testing mode. This will cause the server to connect back to the client on the port specified in the -L option (or defaults to the port the client connected to the server on). This is done immediately therefore running the tests simultaneously.

./iperf -c <Server IP> -d

Ping flood test
Users can send hundred or more packets per second using -f option. It prints a ‘.’ when a packet is sent, and a backspace is printed when a packet is received
ping -f localhost

Performance Test

Netperf
More information please refer to the below link
http://www.netperf.org/netperf/

How to Run
Server:
netserver

Client:
taskset 2 ./netperf -H <Server IP> -t TCP_STREAM
taskset 2 ./netperf -H <Server IP> -t UDP_STREAM

Iperf
More information please refer to the below link
http://en.wikipedia.org/wiki/Iperf

How to Run
Server:
./iperf -s -u
./iperf -s

Client:
./iperf -c <Server IP> -u -b <bandwidth>
./iperf -c <Server IP>

1588 Testing

ptp4l implements the PTP boundary clock and ordinary clock. When hardware time stamping is enabled, ptp4l synchronizes the PTP hardware clock to the master clock. phc2sys can be used to synchronize the system clock to the PTP hardware clock on the network interface card (NIC).

Build Instructions for ptp4l application

  • Get the source 
     git clone http://git.code.sf.net/p/linuxptp/code linuxptp
  • Set the CROSS_COMPILE environment variable arm toolchain
  • Install the kernel headers

          https://www.kernel.org/doc/Documentation/kbuild/headers_install.txt

  • Include the headers path in makefile

          INC = -I/proj/epdsw/punnaiah/git/test/ethernet/1588/header/include
          CFLAGS = -Wall $(VER) $(incdefs) $(DEBUG) $(INC) $(EXTRA_CFLAGS)

  • run make

Execution steps

In order to perform master-slave sync, run the following:

Master (linux server) : ptp4l -i < interface name> -m
Slave (xilinx board) :   ptp4l -i <interface name> -m -s

NOTE: If intended before synchronization phc2sys -s <devicename> -w & can be run to synchronize the system clock to a PTP hardware clock.

Synchronization is stabilized in a few secs.


Mainline status

The current Axi Ethernet driver is not in sync with mainline. The following components need to be upstreamed:

  • NAPI Support
  • Support for 1588
  • Support for legacy 10G MAC
  • Support for 1G MAC Non-processor Mode
  • Support for 10G/25G MAC with 1588
  • Support for 2.5G MAC with 1588
  • Support for Axi Ethernet with AXI MCDMA Configuration
  • Support for USXGMII IP
  • Support for MRMAC IP
  • Support for 1G/10G/25G IP

Also note that current driver has support for both Ethernet with internal DMA driver and external DMA engine driver.

Change Log

2024.1

  • 6,6 kernel rebase.
  • Added phylink support for XXV.
  • Minor bugfix related to buffer size when DRE disabled.

https://github.com/Xilinx/linux-xlnx/commits/xilinx-v2024.1/drivers/net/ethernet/xilinx

2023.2

  • Acquire ptp device information dynamically
  • Add switchable 1/10/25G MAC support
  • Implement work queue to enable/disable link training.

Commits

https://github.com/Xilinx/linux-xlnx/commits/xilinx-v2023.2/drivers/net/ethernet/xilinx

2023.1

  • 6.1 kernel rebase
  • Added phylink support

Commits:

https://github.com/Xilinx/linux-xlnx/commits/xilinx-v2023.1/drivers/net/ethernet/xilinx

2022.2

  • Bugfix on TX TS FIFO read sequence
  • Upstream fixes for RX ring refill allocation failure handling and removing dependency on phy-node for axienet_mdio_setup function call
  • Minor fixes on code indentation and mixed declarations.


2022.1

  • 5.15 kernel rebase
  • MRMAC bugfix to reporting correct link status at 10G/25G.
  • XXV bugfix to check for new IP register reporting clock stability.
  • Bugfixes for DMA error path handling.
  • Minor cleanup for documentation and GCC warnings.
  • Note: TSN driver support moved to staging - no effect on any of the Ethernet components discussed in this page.

https://github.com/Xilinx/linux-xlnx/commits/xilinx-v2022.1/drivers/net/ethernet/xilinx

2021.2

  • Correct phy-mode handling and pass correct mode to PHY framework.
  • Fix MCMDA crash due to dma map and memory errors by handling the same.
  • Emaclite bug fixes

https://github.com/Xilinx/linux-xlnx/commits/master/drivers/net/ethernet/xilinx

https://github.com/Xilinx/linux-xlnx/commits/xlnx_rebase_v5.10_2021.2/drivers/net/ethernet/xilinx

2021.1

  • 5.10 kernel rebase
  • MRMAC one step support
  • Ethtool stats support

0605a36e0574 net: xilinx: Ethtool statistics support
21d0b2896f3f net: xilinx: axiethernet: Fix compilation error when HW timestamp is enabled
9ee7bae04b55 net: axienet: MRMAC reset sequence fixes
8a885d4a28ef net: axienet: Add one step PDelay support
96bdccafe01a net: xilinx: Add one step 1588 support for MRMAC
b9467dc270a2 net: xilinx: MRMAC TX CMD FIFO update
365e87b2465e net: xilinx: Handle TX TS FIFO timeout error
c155fed4cc52 net: xilinx: Update MRMAC config structure with PTP command length


2020.2

  • Fix kernel crash on running high traffic in 10G MCDMA PTP design
  • Add MRMAC support and related ptp fixes
  • Pass phy interface to of_phy_connect()
  • Fix swiotlb buffer full error
  • In _mcdma.c check if dma_map_single returned a valid address

e966fbb13383 net: xilinx: axiethernet: Add check for transmit data FIFO vacancy
2c44afef9055 net: xilinx: Fix indentation issues
664e0cc2775f net: xilinx: Add HW timestamping support for MRMAC
9036464818ee net: xilinx: Add MRMAC support
b83c55d11cbd net: xilinx: XXV Ethernet HW timestamp fixes
e693fba20632 net: xilinx: Use prandom with range to avoid corner cases
4818d3994645 net: xilinx: axiethernet: Pass phy interface to of_phy_connect()
0245a7128384 net: xilinx: axiethernet: Fix swiotlb buffer full error
8b8ae289f6d3 net: xilinx: Check if dma_map_single returned a valid address.

2020.1

  • Kernel upgrade 5.4 and rebase existing axiethernet patches on top of it.

2019.2

  • Regression fixes i.e crash in axienet_recv , kernel bug in ndo_open and 1588.
  • Enhance error and debug reporting.

Commit IDs:

be66caf net: xilinx: axiethernet: Add error reporting for DMA probe
5917934 net: xilinx: axiethernet: Remove clk_init error message for probe defer
1076feb net: xilinx: axiethernet: Fix kernel bug in ndo_open
fb9853e net: xilinx: axiethernet: Fix crash in axienet_recv
3aecd4f net: xilinx: axiethernet: Fix xxv mac short frame handling
7069a2f net: xilinx: axiethernet: Fixed dev_info message
725b6e3 net: xilinx: axiethernet: Add debug messages for TX timestamp
8f09da4 net: xilinx: axiethernet: Fix xxv mac tx timestamp
9c0e07f net: xilinx: axiethernet: In axienet_device_reset add missing txts fifo reset

2019.1

  • Fix crash in ifconfig down
  • Fix axiethernet register description
  • Check for queue full in transmit path
  • Add 64-bit support
  • Extend clocking support
  • Fix kernel crash on MII ioctl
  • use channel-id for mcdma interrupt names
  • Fix netconsole implementation
  • Trivial code cleanup

Commit IDs:

33ebfdb net: xilinx: axiethernet: Fix crash in ifconfig down
f5b9e58 net: xilinx: axiethernet: Fix axiethernet register description
e491e78 net: xilinx: axiethernet: Check for queue full in transmit path
0ba2b93 net: xilinx: axiethernet: Fix code checker warnings
d4c6c09 net: xilinx: axiethernet: Use %pa format specifier for phys_addr_t type
270968c net: xilinx: axiethernet: Add 64-bit support
d139077 net: xilinx: axiethernet: Extend clocking support
fdce589 net: xilinx: axiethernet: Fix kernel crash on MII ioctl
3f2d6cd net: xilinx: axiethernet: use channel-id for mcdma interrupt names
aaad9c0 net: xilinx: axiethernet: Fix netconsole implementation

2018.3
  • Sync kconfig description.
  • In axienet_skb_tstsmp() failure return TX_BUSY.
  • Add error output on DMA allocation failed.
  • Fix memory leak in axienet bd_free().
  • Refactor and split axidma and mcdma programming in separate sources.
  • Fix dma name buffer size and skb_free in xmit.
  • Format XXV error output.
  • Fix compiler warnings.
Commit IDs:
462eb9d net: xilinx: Sync kconfig description
fbb56dc net: xilinx: axiethernet: Move axienet_bd_free() to xilinx_axienet_dma.c
6d3669b net: xilinx: axiethernet: Refactor mcdma_tx/rx_probes
83241b6 net: xilinx: axiethernet: Reordered ethtool status to print all Tx & Rx stats
1fd503a net: xilinx: axiethernet: Have separate num_queues for tx and rx queues
7758b67net: xilinx: axiethernet: In axienet_skb_tstsmp() correct return NETDEV_TX_BUSY on failure
7107ea5 net: xilinx: axiethernet: Add error output on DMA allocation failed
f16fc2d net: xilinx: axiethernet: Fix memory leak in axienet bd_free()
528a9db net: xilinx: axiethernet: Split AXIMCDMA implementation in separate file
7169681 net: xilinx: axiethernet: Split AXIDMA implementation in separate file
8e2082e net: xilinx: axiethernet: Correct skb free function
3e1c99a net: xilinx: axiethernet: Correct dma name buffer size
aa644d2 net: xilinx: axiethernet: Format XXV MAC error output
dc81819 net: xilinx: axiethernet: Fix compiler warnings
127d458 net: xilinx: axiethernet: Fix compiler warnings
2018.2
  • None

2018.1
  • Fix xxv mac padding issue.
  • Add support for USXGMII IP.
  • Update interrupt-names property.
  • Code cleanup of tx with no DRE.
  • Fix remove() bug for XXV no MDIO.
Commit IDs:
10681b8 net: ethernet: xilinx: update interrupt-names property with ip interupt naming convention
fe44c16 net: ethernet: xilinx: Fix xxv mac padding issue - only pad last element.
cea5c97 net: ethernet: xilinx: axienet cleanup of tx with no dre
f5de0cd net: ethernet: xilinx: only teardown mdio when available
9733c76 net: xilinx: axiethernet: Add USXGMII support

2017.4
  • None

2017.3
  • Added Support for Ethernet MCDMA Configuration
  • Added Support for 1588 in buffered mode configuration.
  • Fixed race condition in the transmit path
  • Fixed race condition in the random queue selection for Ethernet MCDMA configuration
Commit Id's:
2108c9fnet: ethernet: xilinx: Add support for mcdma
2108c9fnet: ethernet: xilinx: Add support for 1588 in buffered mode
2108c9fnet: ethernet: xilinx: Fix race condition in the tx path
2108c9fnet: ethernet: xilinx: Fix race in the random queue selection

2017.2
  • None

2017.1
  • Added Support for 10G/25G MAC (PG210)
  • Added Support for 2.5G MAC
  • Fixed issues with the without DRE DMA based design on zynqMP SOC
  • Added Clock Support
  • Fixed bug in the rx reject interrupt handling.
Commit Id's:
22fa015 net: ethernet : xilinx Add config structure to different axienet macs
e3c3b23 net: ethernet: Add quirk checks in the driver
453042c net: ethernet: Add Support for 10G/25G MAC
4027f19 net: ethernet: Add 1588 support for 10G/25G MAC
f475798 net: ethernet: Fix race condition in the driver for 10G/25G MAC
486d636 net: ethernet: Add support for 2.5G MAC
2857aee net: ethernet: Fix issues in the driver when DRE is not enabled in the h/w
a15cd73 net: ethernet: Add Clock support
9b904af net: ethernet: Fix Bug in rx reject interrupt handling.

2016.4
  • None

2016.3
  • Fix kernel crash on the 64-bit platform reported while testing 1588 on Zynq UltraScale+ MPSOC.
Commit Id's:
ea9f81c net: ethernet: xilinx: Fix kernel crash on 64-bit platform

Related Links

---> http://www.wiki.xilinx.com/Linux+Drivers

© Copyright 2019 - 2022 Xilinx Inc. Privacy Policy