Xen device passthrough examples

Passthrough allows you to give control of physical devices to guests domus. This page has a few examples for device passthroughs.

1 PSUART assignment
- 1.1 IRQS calculation
- 1.2 Setup IOMEM field
2 PCI Passthrough assignment
- 2.1 IRQS calculation
- 2.2 Setup IOMEM field

PSUART assignment

Turn xen.dtb into xen.dts:

dtc -I dtb -O dts xen.dtb > xen.dts

The, edit xen.dts by adding xen,passthrough; under the node of the device to assign, in this case serial@ff000000:

serial@ff010000 {
			compatible = "cdns,uart-r1p12", "xlnx,xuartps";
			status = "okay";
			interrupt-parent = <0xfde8>;
			interrupts = <0x0 0x16 0x4>;
			reg = <0x0 0xff010000 0x1000>;
			device_type = "serial";
			clock-names = "uart_clk", "pclk";
			power-domains = <0x26 0x22>;
			clocks = <0x1e 0x1f>;
			cts-override;
			port-number = <0x1>;
			xen,passthrough; /* add this line */
		};

Convert xen.dts back into xen.dtb:

dtc -I dts -O dtb xen.dts > xen.dtb

To enable the PSUART assignment to a domU. Below is one example for the dom create config file.

In above config file, irqs and iomem entries are added. Here is detailed explanation for these entries:

IRQS calculation

To calculate irqs corresponding to the guest domain, it needs to find the following line in UART settings described in xen.dts file:

take the interrupt which is device specific, in this case interrupt number is 0x16, add 32 which is the base address of interrupt handler.

0x16 = 22
irqs = 22 + 32 = 54

Setup IOMEM field

iomem = [ "0xff010,1" ]
1 page starting at "0xFF010000" base address of UART PS.

Creation of passthrough-serial.dts:

Copy the above device tree code snippet and save it as passthrough-serial.dts and convert the above dtc into dtb using dtc compiler:

Finally, create a new guest using ‘xl create -c configfile'.

If your domU is Linux, make sure to add console=ttyPS1,115200 to its kernel command line. In case of dom0less DomUs, you have to edit the boot.source script to add the command line option, then recreate boot.scr with mkimage.

PCI Passthrough assignment

Turn xen.dtb into xen.dts:

The, edit xen.dts by adding xen,passthrough; under the node of the device to assign, in this case pcie@fd0e0000:

Since the PCIe is a DMA-capable device, it should be included in the SMMU section of the xen.dts. Make sure that the PCIe base address in the IOMMU address map exists in the list of mmu-masters. Please refer the latest version of Zynq Ultrascale + MPSoC TRM (UG1085):
a) Find PCIe master device in the table given in the section AXI Master IDs and pick the corresponding master ID, which is 0011010000.
b) Expand these 10 bits to 12 bits, the top 2 bits are simply indicating which TBUn device is behind, this can be found in the Zynq UltraScale+ MPSoC TRM
Interconnect Functional Description section. For PCIe it is TBU1, i.e. the address will be 010011010000=0x4d0.

Add xen,passthrough and iommus cells in xen.dts. Convert xen.dts back into xen.dtb.

Creation of passthrough-pci.dts:
Please see attached passthrough-pci.dts (Please use for release 2023 onwards)for example of partial device tree file and convert the above dtc into dtb using dtc compiler:

Finally, The next step is to configure guest domain to correspond to the pass-through mode.
The guest configuration file should look as follows for release v2022 or older:

For newer release versions(2023 onwards), please use following guest configs:

The options needed for pass-through mode are defined below:
dtdev - The absolute path of the device to be passed through in the device tree
device_tree - absolute path of partial device tree in the Domain 0
irqs - interrupt number inside interrupt queue for the guest domain (see calculation steps below)
iomem - the number of physical memory pages to be passed to the guest (see calculation steps below)

IRQS calculation

to calculate irqs corresponding to the guest domain, it needs to find the following line in PCIe settings described in xen.dts file:

take the interrupt which is device specific, in this case 0x76, 0x74, 0x73, 0x72 the rest of interrupts appears elsewhere as well, add 32 which is thebase address of interrupt handler.

Setup IOMEM field

iomem = [ "0xfd0e0,1", "0xfd480,1", "0xe0000,1000"]
1 page starting at "0xFD0E0000" base address of AXIPCIE_MAIN register (AXI to PCIe Bridge, Main Control and Status registers), 1 page starting at "0xFD480000" base address of PCIE_ATTRIB register (Register block that holds the attributes of the PCIe Controller, PCIe Attributes) and 1000 page starting at 0xe0000 base address for data.

Note that PCIe is mapped to multiple regions and it all needs to be on one single iomem line.

Check Starting Linux guests with Pass-through networking for another passthrough example.