Table of Contents

Introduction

Axi Ethernet Linux driver for Microblaze, Zynq, Zynq Ultrascale+ MPSoC and Versal

This page gives an overview of Axi Ethernet Linux driver which is available as part of the Linux distribution.

Paths, files, links and documentation on this page are given relative to the Linux kernel source tree.

HW IP features

AXI 1G/2.5G Ethernet Subsystem (PG138)

• Support for MII, GMII, RGMII, 1G/2.5G SGMII, and 1000/2500 BASE-X PHY interfaces

• Support for 2.5G Ethernet. In 2.5G mode, only SGMII and 2500BASE-X PHY interfaces are available.

• Support for 1000BASE-X and SGMII over select Input/Output (I/O) low voltage differential signaling (LVDS)

• Support for pause frames for flow control

• Media Independent Interface Management 

• Ethernet Audio Video Bridging (AVB) support

• AXI4-Stream transmit/receive interface

• IEEE Standard 1588 Support

• AXI4-Lite register interface

10 Gigabit Ethernet subsystem (PG157)

• Designed to 10 Gigabit Ethernet specification IEEE Standard 802.3-2012

• AXI4-Stream protocol support on client TX and RX interfaces

• Configured and monitored through an optional AXI4-Lite Management Data interface or using status and configuration vectors

• Supports 10GBASE-SR, -LR and -ER optical links in Zynq-7000, UltraScale™, Virtex-7, and Kintex-7 devices (LAN mode only)

• Supports 10GBASE-KR backplane links including Auto-Negotiation (AN), Training and Forward Error Correction (FEC)

• Supports Deficit Idle Count

• Comprehensive statistics gathering

• Supports 802.3 and 802.1Qbb flow control

• Supports VLAN and jumbo frames

• Custom Preamble mode

• Independent TX and RX Maximum Transmission Unit (MTU) frame length

• Supports high accuracy IEEE Standard 1588-2008 1-step and 2-step timestamping on a 10GBASE-R network interface

10G/25G Ethernet Subsystem (PG210)

• Designed to the Ethernet requirements for 10/25 Gb/s operation specified by IEEE 802.3 Clause 49, IEEE 802.3by, and the 25G Ethernet Consortium

• Includes complete Ethernet MAC and PCS/PMA functions or standalone PCS/PMA for 25Gb/s operation

• Includes complete Ethernet MAC and PCS/PMA functions, standalone MAC or standalone PCS/PMA for 10 Gb/s operation

• Simple packet-oriented user interface

• Comprehensive statistics gathering

• Status signals for all major functional indicators

• BASE-R PCS sublayer operating at 10.3125Gb/s or 25.78125Gb/s

• Optional clause 74 BASE-KR FEC sublayer

• Optional Auto-Negotiation

• Optional clause 108 25G Reed-Solomon Forward Error Correction (RS-FEC) sublayer

• Custom Preamble mode

• Optional IEEE 1588 1-step and 2-step timestamping

• Runtime switchable between 10G and 25G

• Frame pre-emption

USXGMII Ethernet Subsystem (PG251)

• Designed to meet the USXGMII specification EDCS-1467841 revision 1.4

• Supports 10M, 100M, 1G, 2.5G, 5G, or 10GE data rates over a 10.3125 Gb/s link

• Both media access control (MAC) and PCS/PMA functions are included

• Code replication/removal of lower rates onto the 10GE link

• Rate adaption onto user clock domain

• Low data path latency

• 32-bit AXI4-Stream interface for datapath

• Optional AXI4-Lite register interface

• Support for 802.3x and priority-based pause operation

• Detailed statistics gathering

• Support for custom preambles

• Supports deficit idle count (DIC)

MRMAC Ethernet Subsystem (PG314)

• Hardened Ethernet IP block on Versal.

• Multi rate Ethernet MAC supporting speeds from 10G to 100G.

  • The driver supports 25GE and 10GE with 1 to 4 lanes.

• Hardened IP (to be used with Soft DMA and logic for driver subsystems)

• High performance, low latency.

• Low data path latency

• User-side AXI4-Stream interface for data

• AXI4-Lite register interface

• Detailed statistics gathering

• IEEE1588 support

Switchable 1/10/25G IP support (PG292)

• Designed to the Ethernet requirements for 1/10 Gb/s operation specified by IEEE 802.3 Clause 49 or Clause 36

• Runtime switchable Ethernet MAC and PCS/ PMA functions for 1/10/25 Gb/s operation.

• Supports only GTHE3/GTYE3 and GTHE4/GTYE4 transceiver supported devices

• Include Auto Negotiation (Clause-73)

• Enable Link Training (Clause 72)

DCMAC ethernet subsystem (PG369)

• Hardened Ethernet IP block on Versal Premium and Versal HBM (to be used with Soft DMA and logic for driver subsystems).

• 1 x 400GE, 3 x 200GE, 6 x 100GE, or combinations of 100 Gbps, 200 Gbps, and 400 Gbps totaling up to 600 Gbps support.

  • Driver supports 100G with single lane (1 x 100GE).

• Supports 100GBASE-R, 200GBASE-R and 400GBASE-R phy interfaces.

• 40-channel time-sliced MAC capable of 600 Gbps operation.

  • Channelized option for time-sliced applications.

  • Up to 40 channels supported.

  • User-defined bandwidth allocation granularity.

• Pause frame processing, including priority-based flow control.

• Optional built-in RS-FEC functionality.

• IEEE1588 support.

• User-side segmented AXI4-Stream interface.

• AXI4-Lite register interface.

• Detailed statistics gathering.
  
  

Features supported in the driver

Supported Ethernet IPs and their features:

• AXI 1G/2.5G Ethernet subsystem (PG138)

  • Speeds: 10/100/1000/2500 Mbps

  • DMAEngine flow with AXIDMA

  • Full and partial checksum offload

  • Jumbo frame support

  • Independent 4K, 8K, 16K, or 32KB TX and RX frame buffer memory

• 10G Ethernet subsystem (PG157)

  • Jumbo frame support

  • Independent 4K, 8K, 16K, or 32KB TX and RX frame buffer memory

• 10G/25G Ethernet subsystem (PG210)

• USXGMII (PG251)

• MRMAC (PG314)

  • Speeds: 10/25/100G

  • GT narrow and wide mode support

• Switchable 1/10/25G (PG292)

  • Driver supports runtime switching between 1G and 10G speeds

• DCMAC (PG369)

  • 100G single lane support

Generic features:

• GMII, RGMII, SGMII, 1000Base-X, 2500Base-X, 10GBase-R, 25GBase-R, and 100GBase-R PHY configurations

• Supports AXI DMA and AXI MCDMA DMA configurations

• Multi-queue support

• Ethtool support except for MRMAC

• IEEE 1588 support except for USXGMII and DCMAC

• NAPI support in legacy (non-DMAEngine) flow

Missing Features and Known Issues/Limitations in Driver

• Assumes that AXI Ethernet IP is connected to the DMA at the hardware level.

• Supports DMAEngine framework only for AXI1G Ethernet with AXIDMA designs. User can select DMAEngine or legacy flow based on DT changes. For legacy flow, which contains DMA programming sequence i.e., doesn't use separate DMA driver, the compatibility string of DMA node (DTS) is set to a dummy device-tree property compatible = "xlnx,eth-dma".

• The driver doesn't support software time-stamping. It supports only hardware time-stamping.

• PTP synchronization along with high-speed traffic (iperf or netperf) is not supported as under heavy load, timestamp in FIFO and DMA data in BD is expected to go out of sync and remain so until the interface is reset. This is a limitation on Soft Ethernet designs which have HW timestamp FIFOs. This issue is not relevant to designs with inline HW timestamping.

• For 1588 testing, the current driver assumes that AXI Stream FIFO is connected to the MAC TX Time stamp Stream interface at the design level.

• Pause frame solution is not supported and hence there could be RX overrun errors in bidirectional throughput.

• The driver supports MCDMA using kernel config i.e., CONFIG_AXIENET_HAS_MCDMA option. So in multi-instance scenario, the driver will only support a single DMA type i.e., 1G + MCDMA and 10G + MCDMA.

• The driver doesn't support extended multicast and VLAN support. Limited validation of multicast and VLAN support.

• On Versal, support is limited to AXI 1G/2.5G Ethernet subsystem (without PTP), XXV Ethernet subsystem (without PTP, validated at 25G), MRMAC and DCMAC.

• Multiple TX and RX channels in MCDMA have common configuration and reset registers and hence cannot be used independently by multiple MACs. For example, if XXV Ethernet instance 1 uses channels 0-4 of MCDMA and then XXV Ethernet instance 2 uses channels 5-15, then resets during driver initialization and error management affect all channels and both instances need to use common registers. Due to this limitation, multiple MACs cannot be used with a single MCDMA.

• RSFEC configuration is not supported or tested in the driver.

• Does not have any module support with MCDMA. There are some dependencies.

• If using si570 clock generator as the reference for Soft Ethernet designs (for example, on ZCU102), please check the generated frequency on board before and after loading the open source Linux driver for this component. Reprogramming this clock has led to loss of Ethernet functionality in such designs. Workaround is to remove the si570 DT node from the device tree.

• Driver support for PCS/PMA-only configurations is not currently available for any Ethernet IP.

• AXI 1G/2.5G Ethernet subsystem:

  • Fixed-link mode not supported.

  • 2.5G SGMII PHY mode support for AXI 1G/2.5G Ethernet is not available.

  • "max-speed" is a required DT property for AXI 1G/2.5G as it is needed to distinguish 1G and 2.5G speed MAC types. Without this property, driver probe will fail.

  • Validated only on Zynq (1G), ZU+ (1G, 2.5G), Versal (1G, 2.5G), and MB (US+ and 7 series, 1G) platforms.

• 10G Ethernet subsystem

• 10G/25G Ethernet subsystem

  • Dynamic link status/change is not supported in the background as there is no external PHY using PHY framework.

  • XXV Block lock register access in the IP fails (kernel crash with SError) when the reference GT clock is not stable (and link is not up). Since there is no indication to the software on the clock stability, reads to this register cannot be avoided beforehand. If the clock is stable and link is up, there is no effect from this known issue. IP workaround with corresponding driver and DT changes are proposed and documented. Please refer to the “Important AR link” section below.

  • Only validated on ZU+, RFSoC, and Versal based platforms (ZCU102, ZCU670, VCK190).

  • Frame pre-emption is not supported

• USXGMII

  • Dynamic link status/change is not supported in the background as there is no external PHY using PHY framework.

  • Validated on ZU+ based platforms only.

• MRMAC

  • 40G/50G speeds not supported.

  • Multi-lane support is not independent because common GT reset logic exists in subsystem.

  • Ethtool support is not available. This limitation is due to phylink framework and non-presence of an external PHY or PCS. However, there is a plan to add this support in an upcoming release.

  • Only available and validated on Versal based platforms (VCK190, VPK120).

• Switchable 1/10/25G

  • Only 2-step PTP is supported.

  • Driver has workaround to support 1G/10G speed switching. “phy-mode” DT property value should be "internal". It is not automatically generated by SDT. To switch speeds, use the following ethtool command:

    • 1G→10G => $sudo ethtool -s eth1 advertise 0x7C00001C7240

    • 10G→1G => $sudo ethtool -s eth1 advertise 0x100000020000026260

  • Only validated at 1G and 10G on Zynq Ultrascale+ MPSoC (ZCU102) board-to-board setup connected via AFBR-709SMZ SFP module.

• DCMAC

  • Supports only 1x100G configuration.

  • IEEE 1588 support for DCMAC is not available. It will be added in a future release.

  • Validated on Versal premium platform only.

Missing Features and Known Issues/Limitations in SDT

• MRMAC connected IP properties (such as GT control GPIOs) and Timer-Syncer IP properties (including associated timestamping related nodes, if any) are not automatically generated in SDT flow. This will be updated in a future release.

• Support to generate DCMAC and connected nodes (such as GT control and GT reset GPIOs) is not there and will be added in future release. Sample working DCMAC and connected nodes are there in “Device-tree” section:

Important AR links

• AXI Ethernet driver in specific MCDMA configuration throws swiotlb full error with jumbo frames. Please refer to 2020.x AR-75128.

• Default DTG generation for XXV Ethernet designs fails on 2020.2. Please refer to AR-76029 and AR-76457.

• XXV block lock register is not accessible when link is unavailable or unstable, resulting in a Linux driver kernel crash. Please refer AMD Customer Community

Kernel Configuration

The following config options should be enabled in order to build the Axi Ethernet driver
CONFIG_ETHERNET
CONFIG_NET_VENDOR_XILINX
CONFIG_XILINX_AXI_EMAC
CONFIG_AXIENET_HAS_MCDMA (Select this option In the design if Axi Ethernet is configured with Axi MCDMA)
CONFIG_XILINX_PHY (For testing SGMII/1000Base-x Configuration with PCS/PMA Core)

Open

Device-tree

For mor details on Ethernet bindings, please refer to https://github.com/Xilinx/linux-xlnx/blob/master/Documentation/devicetree/bindings/net/xlnx%2Caxi-ethernet.yaml

For more details on phy bindings please refer to https://github.com/Xilinx/linux-xlnx/blob/master/Documentation/devicetree/bindings/net/ethernet-phy.yaml

AXI 1G ethernet with AXIDMA

AXI 1G ethernet with AXIDMA DT for DMAEngine

Soft Ethernet MAC Configured with MCDMA

(1G, legacy 10G or 10G/25G MAC, MRMAC) with MCDMA

When Axi Ethernet (10G/25G MAC) configured with MCDMA device-tree node will be like below (to make use of internal MCDMA driver)

DCMAC with MCDMA

When DCMAC (1x100G) is configured with MCDMA, ethernet node will be like below

When Soft Ethernet MAC configured with MCDMA, The driver supports several features of the MCDMA- 

• The driver supports random Queue/Channel selection.

           Assume in vivado design MCDMA is configured for 16 channels and user don't want Linux driver to use all the 16 channels.
           Below example, pl.dtsi ( AXI ethernet node) marks channel 2, 5 and 10 to be used by Linux driver. 
          xlnx,num-queues = <0x3>;
          xlnx,channel-ids = "2","5","10";

• The driver supports channel observer feature through sysfs. This custom feature is useful in multi-core (Observer) system where MCDMA is a shared resource for all cores. MCDMA IP supports a maximum of six cores and 16 Channels can be distributed across each core as a static configuration. The Channel Observer is available for each group and provides the status about the channels in a group being serviced.

• The driver supports per channel weight configuration through sysfs. This custom feature specifies the channel weight i.e number of packets to be sent in one iteration.

• The driver supports Linux multiqueue networking. It uses the alloc_etherdev_mq() function to allocate the subqueues for the device.

         
To support USXGMII + MCDMA, use above devicetree as reference and select:
xlnx,phy-type = <0x7>;

SWITCHABLE 1/10/25G IP

When Axi Ethernet (1G/10G/25G MAC) is configured with DMA device-tree node will be like below

Related device tree information

For PHY related DT information, refer to

https://github.com/Xilinx/linux-xlnx/blob/master/Documentation/devicetree/bindings/net/ethernet-phy.yaml

When selecting phy specific settings, make sure to mention interface type, speed (if limited/fixed) and phy address properties.
PHY/Converter devices that may be used with this MAC:
-> Xilinx GMII2RGMII converter (https://github.com/Xilinx/linux-xlnx/blob/master/Documentation/devicetree/bindings/net/xlnx%2Cgmii-to-rgmii.yaml)
-> Xilinx PCS PMA PHY (handled internal to this Ethernet driver)

For information on DT bindings when using this Ethernet driver with AXI DMA driver from dma framework, please refer to the following:

See "AXI 1G/2.5G Ethernet Subsystem + AXIDMA without "axistream-connected" property"

https://github.com/Xilinx/linux-xlnx/blob/master/Documentation/devicetree/bindings/net/xlnx%2Caxi-ethernet.yaml#L268

IEEE 1588 Support

The Existing Axi Ethernet driver in the Xilinx git hub supports 1588 for 1G MAC and legacy 10G MAC and 10G/25G MAC
It does timestamp at the MAC level. 1588 is supported in 7-series and Zynq.
The Current driver assumes that AXI Stream FIFO is connected to the MAC TX Time stamp Stream interface at the design level.

Note:
In order to test the hardware platform, a 1588 timestamp capable timer is required at the h/w level.

Prerequisite for 1588 testing:
Need to enable XILINX_AXI_EMAC_HWTSTAMP in the kernel config which does the timestamping at the MAC level (Tx/Rx)
Users should write their own timer driver for time stamping which does time adjustment/ freq adjustment.

For testing 1588 please refer to Testing tools section below.

1588 Device-tree Example Node

MRMAC 1588 support is tested with Xilinx 1588 Timer-Syncer block details of which can be found here:

https://github.com/Xilinx/linux-xlnx/blob/master/Documentation/devicetree/bindings/ptp/ptp-xilinx.yaml

https://github.com/Xilinx/linux-xlnx/blob/master/drivers/ptp/ptp_xilinx.c

Performance

These benchmark performance numbers were obtained by connecting Xilinx boards to Linux PCs/server machines (Ubuntu/Red Hat Enterprise). 
The tools used are netperf or iperf (Refer to tool information below).

Performance benchmark numbers mentioned in below tables are for reference and dependent on multiple factors i.e setup , vivado design configuration  etc. 

NOTE: CPU utilization reported in below performance tables is an aggregate of all CPU's. i.e on ZynqMP platform, it reports combined utilization of all four A53 cores.

1G Ethernet with AXIDMA

Kernel version: 6.12

AXI 1G/2.5G Ethernet Subsystem : TX and RX full checksum offload enabled.
ZynqMP
Board: ZCU102 board (production silicon) + SFP Module

Zynq
Board: ZC706 board + SFP Module

NOTE- There is ~10% drop (compared to 2019.2) in performance for 1500 MTU.
The drop is due to enable CONFIG_OPTIMIZE_INLINING forcibly” commit in linux kernel.

Kernel and networking stack is full of inline functions and it could be some unoptimized
inline function (could also be dependent on gcc version) leading to a performance drop.

The performance drop is observed on GEM and Xilinx Axi Ethernet MAC’s on Zynq

The plan is to document the performance drop on zynq and initiate the discussion with
the mainline community so that it is analyzed by respective kernel maintainers.

Microblaze

Traditionally microblaze designs are not targeted for high performance applications so only functional sanity is done.

10G Ethernet with AXIMCDMA

Kernel version: 6.6

ZynqMP
Board: ZCU102 board (production silicon) + SFP Module