Versal and Versal AI Edge Series Gen 2 SMMU Usage Guidance

Introduction

The SMMU acts just like an MMU for processors; it translates addresses coming from System I/O devices, such as DMA. The SMMU requires masters such as DMAs to use virtual addresses rather than physical addresses. The SMMU is transparent to device drivers in Linux, as the DMA framework knows how to handle it. The SMMU is equivalent to an IOMMU used in other system architectures.

This page is not intended to be a tutorial about the SMMU. Readers should refer to other documents (such as the Versal Technical Reference Manual) for a more detailed understanding of Versal Gen1, together with ARM documents such as the ARM System Memory Management Unit Architecture Specification and the ARM Cortex-A Series Programmers Guide for a more complete understanding of the SMMU operation.

The primary focus of this page is to provide users with information on the usage and how to enable the SSMU for Versal devices from a hardware and software perspective.

Enabling SMMU

By default, SMMU is disabled. The section provides information on enabling SMMU, which covers steps specific to SMMU.

Hardware Requirements

For FPD, changes are not required in the hardware design. Only the LPD masters need the changes outlined below at the hardware design stage.

Routing LPD I/O Master's Traffic Through SMMU (Changes in Hardware Design)

An LPD master's traffic by default takes the LPD path, which skips SMMU.  In order to manage a specific LPD I/O master device through SMMU, its traffic must be routed through SMMU.  The steps to enable that option are shown below:

Design Changes

• To get the Peripheral traffic routing through SMMU, we need to set the routing bit to FPD. This can be done in two ways.

• Generate design using Vivado Integrated Design Environment (IDE) (for details, consult [1])

  • Create Vivado Init file

    $ vi /home/<username>/.Xilinx/Vivado/Vivado_init.tcl Add the below contents to enable isolation-related features, set_param bd.isolation true set_param gui.test isolation_gui

  • Modify Block Design
    Select The "Isolation Editor" Option.
    
    

    Open

    

     

    Enable Coherent & Virtualization under Edit flag
    
    

    Open

    

     

    Only Enabled for GEM (same can be done for other Peripherals)
    
    

    Open

    

  • Generate Device images further with the usual steps. To confirm readback, the register below where the FPD bit should be set.
    
    

•  Another way to enable the SMMU for an LPD DMA master is to add CDO commands that program the relevant registers so that the LPD master’s traffic is routed through FPD, and to ensure that this FPD‑routing setting is enabled for that specific DMA master.

  Versal Gen 1

Versal™ AI Edge Series Gen 2 (Default EDF below write is already taken care)

Software Changes

• Kernel Config :
  CONFIG_ARM_CCI_PMU, CONFIG_ARM_SMMU, CONFIG_ARM_SMMU_V3, etc…

• Device tree node:

  • Make sure to set SMMU status = “ok”. The DMA master needs to have the iommus and dma-coherent properties.

    Versal Gen1 iommu@fd800000 { compatible = "arm,mmu-500"; status = "okay"; reg = <0x00 0xfd800000 0x00 0x40000>; stream-match-mask = <0x7c00>; #iommu-cells = <0x01>; #global-interrupts = <0x01>; interrupts = <0x00 0x6b 0x04 0x00 0x6b 0x04 0x00 0x6b 0x04 0x00 0x6b 0x04 0x00 0x6b 0x04 0x00 0x6b 0x04 0x00 0x6b 0x04 0x00 0x6b 0x04 0x00 0x6b 0x04 0x00 0x6b 0x04 0x00 0x6b 0x04 0x00 0x6b 0x04 0x00 0x6b 0x04 0x00 0x6b 0x04 0x00 0x6b 0x04 0x00 0x6b 0x04 0x00 0x6b 0x04 0x00 0x6b 0x04 0x00 0x6b 0x04 0x00 0x6b 0x04 0x00 0x6b 0x04 0x00 0x6b 0x04 0x00 0x6b 0x04 0x00 0x6b 0x04 0x00 0x6b 0x04 0x00 0x6b 0x04 0x00 0x6b 0x04 0x00 0x6b 0x04 0x00 0x6b 0x04 0x00 0x6b 0x04 0x00 0x6b 0x04 0x00 0x6b 0x04 0x00 0x6b 0x04>; interrupt-parent = <0x4d>; xlnx,ip-name = "psv_fpd_smmutcu"; phandle = <0x5a>; }; dma-controller@ffa80000 { phandle = <0x7f>; compatible = "xlnx,zynqmp-dma-1.0"; status = "okay"; reg = <0x00 0xffa80000 0x00 0x1000>; interrupts = <0x00 0x3c 0x04>; interrupt-parent = <0x59>; clock-names = "clk_main\0clk_apb"; xlnx,dma-type = <0x01>; #dma-cells = <0x01>; xlnx,bus-width = <0x40>; clocks = <0x8f 0x51 0x8f 0x52>; power-domains = <0x8d 0x18224035>; xlnx,is-cache-coherent = <0x01>; xlnx,ip-name = "psv_adma"; xlnx,dma-mode = <0x01>; dma-coherent; iommus = <0x5a 0x210>; //Stream-ID }; ---------------------------------------------------------------------------------------------------------------------------------------------- Versal Gen 2 iommu@ec000000 { phandle = <0x14a>; compatible = "arm,smmu-v3"; status = "ok"; reg = <0x00 0xec000000 0x00 0x40000>; #iommu-cells = <0x01>; interrupt-names = "combined"; interrupt-parent = <0x1b>; interrupts = <0x00 0xa9 0x04>; dma-coherent; }; dma-controller@ebd00000 { phandle = <0x49>; compatible = "amd,versal2-dma-1.0"; status = "okay"; reg = <0x00 0xebd00000 0x00 0x1000>; interrupt-parent = <0x1b>; interrupts = <0x00 0x48 0x04>; clock-names = "clk_main\0clk_apb"; #dma-cells = <0x01>; xlnx,bus-width = <0x40>; clocks = <0xb6 0x51 0xb6 0x52>; power-domains = <0xbc 0x18224035>; xlnx,is-cache-coherent = <0x00>; xlnx,ip-name = "adma"; xlnx,dma-mode = <0x01>; xlnx,zdma-clk-freq-hz = <0x8f0cba9>; iommus = <0x14a 0x210>; //Stream-ID dma-coherent; };

     

    SMMU driver boot logs

    Versal Gen 1 xilinx-vck190-20252:~$ dmesg | grep -e smmu -e iommu [ 1.074217] iommu: Default domain type: Translated [ 1.079050] iommu: DMA domain TLB invalidation policy: strict mode [ 2.097672] arm-smmu fd800000.iommu: probing hardware configuration... [ 2.104276] arm-smmu fd800000.iommu: SMMUv2 with: [ 2.109021] arm-smmu fd800000.iommu: stage 1 translation [ 2.114467] arm-smmu fd800000.iommu: stage 2 translation [ 2.119911] arm-smmu fd800000.iommu: nested translation [ 2.125273] arm-smmu fd800000.iommu: stream matching with 64 register groups [ 2.132473] arm-smmu fd800000.iommu: 32 context banks (0 stage-2 only) [ 2.139153] arm-smmu fd800000.iommu: Supported page sizes: 0x61311000 [ 2.145746] arm-smmu fd800000.iommu: Stage-1: 48-bit VA -> 48-bit IPA [ 2.152335] arm-smmu fd800000.iommu: Stage-2: 48-bit IPA -> 48-bit PA [ 2.159054] arm-smmu fd800000.iommu: preserved 0 boot mappings [ 2.165044] arm-smmu fd800000.iommu: Failed to disable prefetcher for errata workarounds, check SACR.CACHE_LOCK [ 2.868813] xilinx-zynqmp-dma ffa80000.dma-controller: Adding to iommu group 0 [ 2.876458] xilinx-zynqmp-dma ffa90000.dma-controller: Adding to iommu group 1 [ 2.884000] xilinx-zynqmp-dma ffaa0000.dma-controller: Adding to iommu group 2 [ 2.891533] xilinx-zynqmp-dma ffab0000.dma-controller: Adding to iommu group 3 [ 2.899109] xilinx-zynqmp-dma ffac0000.dma-controller: Adding to iommu group 4 [ 2.906639] xilinx-zynqmp-dma ffad0000.dma-controller: Adding to iommu group 5 [ 2.914174] xilinx-zynqmp-dma ffae0000.dma-controller: Adding to iommu group 6 [ 2.921824] xilinx-zynqmp-dma ffaf0000.dma-controller: Adding to iommu group 7 [ 2.929551] zynqmp-qspi f1030000.spi: Adding to iommu group 8 [ 2.977064] macb ff0c0000.ethernet: Adding to iommu group 9 [ 6.238537] macb ff0d0000.ethernet: Adding to iommu group 10 [ 6.389379] dwc3 fe200000.usb: Adding to iommu group 11 [ 6.537812] sdhci-arasan f1050000.mmc: Adding to iommu group 12 --- Versal Gen2 [ 1.512323] iommu: Default domain type: Translated [ 1.517163] iommu: DMA domain TLB invalidation policy: strict mode [ 1.903610] arm-smmu-v3 ec000000.iommu: 65518 shared contexts [ 1.903613] arm-smmu-v3 ec000000.iommu: ias 48-bit, oas 48-bit (features 0x001e1f8f) [ 1.911885] arm-smmu-v3 ec000000.iommu: allocated 65536 entries for cmdq [ 1.919013] arm-smmu-v3 ec000000.iommu: allocated 32768 entries for evtq [ 2.412534] xilinx-zynqmp-dma ebd00000.dma-controller: Adding to iommu group 0 [ 2.419956] xilinx-zynqmp-dma ebd10000.dma-controller: Adding to iommu group 1 [ 2.427301] xilinx-zynqmp-dma ebd20000.dma-controller: Adding to iommu group 2 [ 2.434634] xilinx-zynqmp-dma ebd30000.dma-controller: Adding to iommu group 3 [ 2.441971] xilinx-zynqmp-dma ebd40000.dma-controller: Adding to iommu group 4 [ 2.449310] xilinx-zynqmp-dma ebd50000.dma-controller: Adding to iommu group 5 [ 2.456639] xilinx-zynqmp-dma ebd60000.dma-controller: Adding to iommu group 6 [ 2.463970] xilinx-zynqmp-dma ebd70000.dma-controller: Adding to iommu group 7 [ 2.500562] macb f19e0000.ethernet: Adding to iommu group 8 [ 2.519131] macb f19f0000.ethernet: Adding to iommu group 9

     

SMMU Debug

Below are available mechanisms to observe SMMU behavior during debug, including tracing IOVA (and related PA mappings) and other IOMMU trace events.
These traces also expose runtime device attach/detach to IOMMU domains, which is useful for diagnosing spurious or misbehaving devices. In addition, dma, kmem, swiotlb, and iomap trace events can be leveraged to further root-cause issues during deeper debug.

Kernel Tracing

Event queue

Open

Performance

A detailed study can be checked at [3]

References

[1] https://amd.atlassian.net/wiki/spaces/XPS/pages/903865051

[2] Versal SMMU and CCI configuration

[3] https://amd.atlassian.net/wiki/spaces/XPS/pages/1393332590
[4] Arm® System Memory Management Unit Architecture Specification SMMU architecture version 3