Macb Driver

Cadence Macb Linux Driver for Zynq, Zynq Ultrascale+ MPSoC and Versal

Introduction

This page gives an overview of the macb driver which is available as part of the Zynq, ZynqMP and Versal Linux distribution and in the mainline.
Paths, files, links and documentation on this page are given relative to the Linux kernel source tree.

HW IP features

  • Speed support for 10/100/1000 Mbps
  • MAC loopback and PHY loopback
  • Partial store and forward option
  • Packet buffer option
  • Flow control - TX/RX pause
  • Checksum offload support, CRC checking, FCS stripping
  • Promiscuous mode, Broadcast mode
  • Collision detection and enforcement - this is an IP feature, no SW support required
  • MDIO support for PHY layer management
  • Multicasting support
  • VLAN tagged frames
  • Half duplex support
  • Programmable IPG
  • External FIFO interface
  • Wake on LAN
  • IEEE1588 support for ZynqMP and Versal
  • Jumbo frame size support for ZynqMP and Versal
  • 64 bit addressing for ZynqMP and Versal
  • Priority queue support for ZynqMP and Versal
  • PS SGMII support (hardwired to 1Gbps) is present in ZynqMP

Features supported in driver

(Functional HW IP and stack related features)
  • Speed support for 10/100/1000 Mbps with clock framework
  • Packet buffer option
  • Checksum offload support, CRC checking, FCS stripping
  • MDIO support for PHY layer management
  • Multicasting support
  • Programmable IPG
  • IEEE1588 support for ZynqMP and Versal
  • Jumbo frame size support for ZynqMP and Versal
  • 64 bit addressing for ZynqMP and Versal
  • Priority queue support for ZynqMP and Versal
  • PS SGMII support is present in ZynqMP and supported in the driver
  • This driver can be used with PL SGMII/1000BaseX driver on Zynq and ZynqMP
  • This driver can be used with gmii2rgmii converter driver
  • Support for EthTool queries
  • NAPI support
  • Clock adaptation on Zynq, ZynqMP and Versal
  • Runtime PM and suspend/resume supported on ZynqMP and Versal
  • Partial store and forward
  • Wake on LAN support using ARP on ZynqMP and Versal

Missing Features, Known Issues and Limitations

  • Linux does not support loopback
  • Flow control support is not present in the driver. RX pause frames can be received by the IP but TX pause frame support is not provided.
  • External FIFO interface is not supported by the driver - this implementation is DMA based.
  • No interrupt support for PHY events in driver. The current implementation relies on polling method for phy event
  • No IEEE 1588 support for Zynq as the timestamp implementation in IP is not accurate enough.
    • The timestamp generated on a PTP event is stored in a non-latching register. This means that the timestamp is overwritten whenever a PTP event packet arrives. Hence there is no foolproof way to associate a timestamp with the packet.
    • An application using sync, follow up, pdelay request, pdelay response with a sync cycle of 1 second and NO errors in between might possibly work but it is not reliable because sync will fail the moment there is any deviation: i.e. multiple back to back PTP event packets in the same direction (or) a small sync interval on a high traffic system where the SW is unable to process the timestamp register before it is overwritten.
  • PS SGMII GT initialization is not supported via zynqmp_phy.c - it needs to be explicitly initialized.
  • WOL does not work on warm restart designs because GEM WOL requires an RX BD scratch area that is accessible even during suspend (OCM is used for this) and OCM is secure in this design which is a limitation for this feature.
  • WOL does not work on warm restart designs due to some limitations (2018.1/2/3) - AR-71028
  • PTP time adjustment for a large negative delta fails in 2018.1/2 - AR-71332
  • MACB MDIO bus support - Please find the patches for 2017.1, 2017.2, 2017.3, 2017.4, 2018.1, 2018.2 and 2018.3 at the AR - AR-69132
  • ZynqMP PS SGMII GT initialization and related - AR-68866
  • ZynqMP PS SGMII fixed link - AR-69769
  • TI PHY design on ZynqMP evaluation board has incorrect straps and can be remedied with a SW workaround (already implemented in drivers) - AR-70686
  • PL PCS PMA initialization in fsbl for Zynq and ZynqMP - refer to xapp1026 and xapp130
  • For custom Versal designs using AIE on 2020.1, make sure the low DDR region is accessible to LPD slaves (including GEM) using a workaround (<link to AR>)
  • There is a performance drop of ~100Mbps between 2020.1 (5.4 Linux kernel) and 2019.2 (4.19 Linux kernel) observed on both GEM and Axi Ethernet on Zynq. This is currently suspected to be the result of change in the net framework and there is no workaround yet. Further updates will be documented in AR-75195
  • For full list of ARs, search XKB

Kernel Configuration

The following config options should be enabled in order to build the macb driver
CONFIG_ETHERNET
CONFIG_NET_VENDOR_CADENCE
CONFIG_MACB
CONFIG_NETDEVICES
CONFIG_HAS_DMA


Optional kernel configuration:
CONFIG_MACB_USE_HWSTAMP

Use IEEE 1588 hwstamp (only supported in ZynqMP and Versal) - This config option supports use of 1588 HW TSTAMP support in ZynqMP & Versal and depends on MACB.
This option enables IEEE 1588 Precision Time Protocol (PTP) support for MACB.

Devicetree

Compatible string can be:
  • "cdns,gem" for Zynq
  • "cdns,zynqmp-gem" for ZynqMP. This compatible string enables use of jumbo frame sizes, 1588 and HW timestamping suport and any features exclusive to ZynqMP.
  • "cdns,versal-gem" for Versal. This compatible string enables use of jumbo frame sizes, 1588 and HW timestamping suport, automatic flow control, 802.1AS and any features exclusive to Versal.


For more details on phy bindings please refer "Documentation/devicetree/bindings/net/macb.txt"

Related devicetree information

Ethernet DT

For generic ethernet DT property information, refer to:
https://github.com/Xilinx/linux-xlnx/blob/master/Documentation/devicetree/bindings/net/ethernet.txt

PHY DT

Xilinx converter and PHY DT

PHY/Converter devices that may be used with this MAC:

RGMII Tuning in DT

RGMII tuning is driven in phy framework using "rgmii-id", "rgmii-txid", "rgmii-rxid" properties Make sure to set phy-mode to any of these as per your board requirement.
In addition to enabling tuning, some phys also give control of tuning values via devicetree. Please refer to the devicetree bindings documentation of the phy you use in order to tune these according to your board.

TSU clock in DT

Clock adaption is present by default for both Zynq and ZynqMP. For more details refer to devicetree clock bindings and respective wiki pages.
ZynqMP also has tsu-clk adaption support in addition to all the other reference clocks.

Fixed link DT

This driver can be used for a MAC - MAC fixed link connection. In order to do so, please update the devicetree fixed link node as per
https://github.com/Xilinx/linux-xlnx/blob/master/Documentation/devicetree/bindings/net/fixed-link.txt
and set the phy-mode to "moca" (https://github.com/Xilinx/linux-xlnx/blob/master/include/linux/phy.h)

Common MDIO DT

To use multiple GEM→PHY connections using a common MDIO bus, please use the following devicetree convention:

Where:

→ gem0 is the instance whose phy management is being used (and whose MDC and MDI lines are connected to both PHYs)

→ gem0 is communicating via phya and gem1 is communicating via phyb

Note that gem0 needs to come up before gem1 and stay up (because the MDIO interface is expected to be up first; otherwise, the dependent MAC-PHY link (gem1-phyb) will come up on next ifconfig up/down).

As a result of this gem0's runtime PM will not be effective if gem1 is still active in this configuration.

PS SGMII DTs (ZynqMP only)

→ The DT node for PS SGMII is the same as any other configuration with phy-mode property set to "sgmii" and a phy node as seen below. In this case, the Linux SW(currently phylib, NOT phylink) ensures autonegotiation is performed with the PHY. In addition, PCS block inside of GEM will also negotiate and provide link status information in PCS_status register (to be read twice because of stick bits). 

→ If there is no MDIO access to the SGMII PHY or if SFPs are used, then the phy-mode can be set to sgmii and fixed link node can be used instead of phy node. This means that the Linux SW assumes, there is no PHY or autoneg. PCS block will still attempt autoneg and update PCS_status register (to be read twice because of sticky bits).

→ Alternately, patch in SGMII fixed link AR mentioned above can be used (especially if there is no PHY) with "is-internal-pcspma" property and a fixed link node. In this case, both Linux SW and PCS block do not attempt autoneg and the link status in PCS_status register will always report link up.

Pointers on PHY reset via GPIO

→ For boards which require a PHY reset via GPIO, please see the generic framework provisions here: https://github.com/Xilinx/linux-xlnx/blob/master/Documentation/devicetree/bindings/net/ethernet-phy.yaml#L141

This can be used for multiple PHYs with independent GPIO resets as well.

→ If reset is required before PHY detection, please see the MDIO bus provision here:  https://github.com/Xilinx/linux-xlnx/blob/master/Documentation/devicetree/bindings/net/mdio.yaml#L30

→ When using PHY reset via GPIO, please check manufacturer specific datasheet for the reset polarity, reset assert duration and post de-assert delay for PHY to be functional. These values can then be passed to PHY and MDIO framework via Devicetree documentation above.


Performance

These benchmark performance numbers were obtained by connecting Xilinx boards to Linux PCs/server machines (Ubuntu/Red Hat Enterprise).
The tool used is netperf (Refer to tool information below).
The protocol, MTU size and option to note CPU load can all be selected from netperf/netserver options

Zynq

Board: ZC706
CPU Freq: 666MHz (A9)
Link Speed: 1000Mbps, Full duplex
Linux version: 4.19

TCP (Mbps)UDP(Mbps)
MTUTXCPU(%)RXCPU(%)TXCPU(%)RXCPU(%)
1500728.7697.29548.7095.96565.665.00444.899.55


Linux version: 5.4

NOTE- There is ~10% drop in performance (compared to 2019.2) for 1500 MTU.
The drop is due to this commit enabling CONFIG_OPTIMIZE_INLINING forcibly in linux kernel. It is observed on GEM and Xilinx Axi Ethernet drivers on Zynq.

Kernel and networking stack has a large number of inline functions and it could be some unoptimized inline function (could also be dependent on gcc version) leading to performance drop.

The plan is to document this performance drop on Zynq and initiate a discussion with mainline community so that it is analyzed by respective kernel maintainers.


TCP (Mbps)UDP(Mbps)
MTUTXCPU(%)RXCPU(%)TXCPU(%)RXCPU(%)
1500654.7993.11737.6381.43486.863.5630396.23

ZynqMP

Board: ZCU102
CPU Freq 1100MHz (A53)
Link Speed 1000Mbps, Full duplex
DDR 533MHz
CCU: No
Linux version: 5.4

TCP (Mbps)UDP (Mbps)
MTUTXCPU (%)RXCPU (%)TXCPU (%)RXCPU (%)
1500941.235.0939.1754.94961.520.3961.422.07
8192988.854.9989.0917.01991.95.29985.128.17


Test Procedure

Diagnostic and Protocol Tests

PING

This utility used to test the reachability of a host on an Internet Protocol(IP) network and to measure the round trip time for messages sent from the originating host to a destination computer.
How to run:

WebServer

Connect zynq board to a Linux x86 machine. Ensure that telnet server is running on the Zynq board. It tests for remote access for Zynq board on host machine
Open a web browser on host machine and enter the static IP assigned to zynq board. Webpage is expected to be displayed properly.

Telnet

FTP & TFTP

How to run:
Open a ftp client on the host with the Zynq.
Transfer a big enough file (in MBs) using mput command.
File transfer should be completed without any error.

Pkt Generator

Please refer to link below for how to run and various options
https://www.kernel.org/doc/Documentation/networking/pktgen.txt

Performance Tests

Netperf

How to run:
Server:
Client:
For more information please refer to the link below:
http://www.netperf.org/netperf/

Iperf

How to run:
Server:
Client:
For more information please refer to the link below:
http://en.wikipedia.org/wiki/Iperf

Stress Test

Iperf with option -d

Run iperf in dual testing mode. This will cause the server to connect back to the client on the port specified in the -L option (or defaults to the port the client connected to the server on). This is done immediately therefore running the tests simultaneously.

Ping flood test

Users can send hundred or more packets per second using -f option. It prints a ‘.’ when a packet is sent, and a backspace is printed when a packet is received

PTP

1588 synchronization can be tested on ZynqMP using open source linuxptp application.
http://linuxptp.sourceforge.net/
The setup requires a master with precise clock and timstamping capabilities, typically a NIC or another 1588 capable device.
How to run
master:
slave:

Mainline status

The macb driver is currently at mainline kernel 5.4 with some patches pulled in from later kernels. The patches that not yet in any mainline kernel are as follows:
  • WOL via ARP support (~70 lines)
  • Partial store and forward support (~80 lines)
  • Versal support (~50 lines)
  • Minor differences including mdio phy node support (gmii2rgmii), PCS autoneg and CAPS change, gem_rx_refill skbuff error handling, optimized HW timestamp reading, high DDR handling and other bugfixes (~50 lines altogether).
Any further changes will be upstreamed

PHY details

The following PHYs were tested with ZynqMP GEM:
  • TI DP83867IR
  • TI DP83867E (SGMII)
  • Marvell 88E1112
  • Marvell 88E1510/2
  • Realtek RTL8211
  • Vitesse VSC8211
  • Micrel KSZ9031
  • VSC8531_02

Change Log

2020.2

Summary:

  • Minor bugfix for high memory DMA handling.

Commits:

https://github.com/Xilinx/linux-xlnx/commits/xilinx-v2020.2/drivers/net/ethernet/cadence

2020.1

Summary:

  • Minor bug fixes including
    • Workaround for TSO IP errata on Versal GEM.
    • Handle PHY reset for IPs such as PCS PMA phy.
    • Increase halt timeout to accommodate 10Mbps.

Commits:

https://github.com/Xilinx/linux-xlnx/commits/xilinx-v2020.1/drivers/net/ethernet/cadence

2019.2

Summary:

  • Minor feature updates

Commits:

https://github.com/Xilinx/linux-xlnx/commits/xilinx-v2019.2.01/drivers/net/ethernet/cadence


2019.1

Summary:

  • Bug fixes and enhancements on top of 4.19 kernel

Commits:

https://github.com/Xilinx/linux-xlnx/commits/xilinx-v2019.1/drivers/net/ethernet/cadence

acbe29b net: macb: Apply RXUBR workaround only to versions with errata
7caaf8e net: macb: Fix napi calls in macb_suspend/resume
235128d net: macb: Fix merge error in capabilities in config struct
fee8215 net: macb: Remove repeated tasklet init
95977dd net: macb: Remove unused variables
41f029d net: macb: Remove repeated RBQP assignment
7193098 net: macb: Disable macb pad and fcs for fragmented packets
7d069bc net: macb: Sync RXUBR changes with mainline
d77376c Revert "net: macb: Do not call PM sync in mdio helpers"
d8dd5fe Revert "net: macb: Remove pm calls in mdio timeout function"
f5520f1 macb: Sync PHY reset with mainline
e5c7c2e net: macb: Add separate definition for PPM fraction
24e1145 net: macb: Fix SUBNS increment and increase resolution
b10faf3 net: macb: Optimize reading HW timestamp
e393646 net: macb: Add NULL check for PCLK and HCLK
f22dab1 net: macb: Sync macb_config usage with mainline
9c71c6e net: macb: Change interrupt and napi enable order in open

> 4.19 Mainline fixes pulled in:
af3a961 net: ethernet: cadence: fix socket buffer corruption problem
65f81f2 net: macb: remove unnecessary code
ea79ec6 net: macb: restart tx after tx used bit read
aef9308 net: macb: fix random memory corruption on RX with 64-bit DMA
6b28540 net: macb: fix dropped RX frames due to a race
db8102b net: macb: Check for SKBTX_HW_TSTAMP in macb driver


2018.3

Summary:

  • Bug fixes and minor enhancements

Commits:

9caf0ef net: macb: Fix BUG unregistering invalid mdiobus
86bbb72 net: macb: Use devicetree phy-handle if available
41bc0b5 net: macb: Fix warning about uninitialized phy_node
1e7f173 net: macb: Initialize bool to false instead of zero
c5316f6 net: macb: Remove redundant ptp clock selection
0eddb82 macb: refactor the duplicate code to use a shared function
7b2d994 macb: Do not check for timeout when PHY is idle
c241cc8 net: macb: Fix ptp time adjustment for large negative delta
43c7d00 net: macb: Increment rx bd head after allocating skb and buffer
835b617 net: macb: Process tx timestamp only on ptp packets
d2b7486 net: macb: Do not call PM sync in mdio helpers
f4eb775 net: macb: Fix phy path in macb resume
7b7c156 net: macb: Remove pm calls in mdio timeout function


2018.2
No changes

2018.1
Summary:
  • Use mainline implementation of 64 bit addressing
  • Use mainline implementation of PTP support
  • Add clock management for tsu-clk
  • Add WOL support for ZynqMP
Commits:
Sync with 4.14 mainline kernel
9aa7608 net: macb: Add tsu_clk property and use it
e964800 net: macb: Add WOL support for ZynqMP
53ec68b net: macb: Fix GEM crash when suspend/resume plus down/up is done
2d522a5 net: macb: Remove older MACB_EXT_BD config option
617ea48 net: macb: Correct check for 64 bit addressing
fa19565 net: macb: Add phy suspend and resume
95a33ee net: macb: Set rx mode in resume
d7aa0c8 net: macb: Update macb RX tie-off descriptors
e6fca01 net: macb: Remove unnecessary DBW read back from NWCFG
23d6b42 net: macb: Cleanup empty lines

2017.4
No changes

2017.3
Summary:
  • Added support for partial store and forward
  • Pulled in minor mainline fixes and phy related issues
  • Added support for macb suspend/resume
Commits:
bf85fd4 net: macb: Add support for partial store and forward
f646336 net: macb: Fix gpio for phy reset
a29aa21 net: macb: Fix issues with FPD off
e1a214d net: macb: Misc cleanup

2017.2
Summary:
  • Pulled in a minor mainline fix for mdio bus scan error check
Commits:
4356634 macb: fix mdiobus_scan() error check

2017.1
Summary:
  • Added PM runtime support
  • Added context loss support; Cleanup around clock and suspend, resume paths. Although this support is added in macb driver, there is a know issue at the moment that GEM does not work on resume directly. It is required to bring the interface down and up again.
  • Fixed ptp time adjustment for large negative delta
  • Fix PHY reset and only call GPIOD functions when valid GPIO is present
  • Fixed spinlocks in macb_close around ptp_clock_unregister to avoid kernel panic.
  • Fixed TSU CAPS mask
Related phy driver changes:
  • DP83867: Added a SW workaround for link instability on ZCU102 board.
Commits:
afeaf15 arm64: zynqmp: macb: release spinlock before calling ptp_clock_unregister
36f7baa net: macb: Correct TSU_CAPS mask
27f1c64 macb: fix PHY reset
7613445 net: macb: Only call GPIO functions if there is a valid GPIO
2288919 net: macb: Fix ptp time adjustment for large negative delta
6cbc5cd net: cadence: macb: Fix kernel-doc format
ddd4804 net: macb: fix the clk enable and disable
1b0a659 net: macb: Add runtime support
4dc7d77 net: macb: Add context loss support
b9a2910 net: macb: Fix the double disable of clocks
756de54 net: macb: Cleanup the clock code
2f2bb37 net: macb: Fix unused warning
911b158 net: macb: Enable clocks for the mdio accesses
25f7255 net: macb: Convert the infinite wait loop to a timeout
53ac032 net: macb: Move to runtime_put to cut clocks
d415d56 net: macb: Update the phy write sequence
DP83867 phy driver:
7557928 net: macb: SW workaround for link instability on DP83867

2016.4
Summary:
  • Added support for fixed link
Commits:
59e3534 net: macb: Add support for fixed link

2016.3
Summary:
  • Added support for 64 bit addressing
  • Added support to use gmii2rgmii converter driver
  • Handle HRESP error with SW reset and re-initialization of necessary parameters
  • The above changes are also in mainline
Commits:
b0fbcba net: macb: Handle HRESP error
ff73646 net: macb: Fixed mixed declaration and code warnings
190b6af net: macb: Update TX and RX EXT BD registers only when required
d470dfb net: macb: Correct CAPS masks
6121d00 net: macb: Add support for 64 bit addressing
f9c43e8 net: macb: add support for mdio phy nodes

Related Links