Xen device passthrough examples

Passthrough allows you to give control of physical devices to guests domus. This page has a few examples for device passthroughs.

PSUART assignment

Turn xen.dtb into xen.dts:

1 dtc -I dtb -O dts xen.dtb > xen.dts

The, edit xen.dts by adding xen,passthrough; under the node of the device to assign, in this case serial@ff000000:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 serial@ff010000 { compatible = "cdns,uart-r1p12", "xlnx,xuartps"; status = "okay"; interrupt-parent = <0xfde8>; interrupts = <0x0 0x16 0x4>; reg = <0x0 0xff010000 0x1000>; device_type = "serial"; clock-names = "uart_clk", "pclk"; power-domains = <0x26 0x22>; clocks = <0x1e 0x1f>; cts-override; port-number = <0x1>; xen,passthrough; /* add this line */ };

Convert xen.dts back into xen.dtb:

1 dtc -I dts -O dtb xen.dts > xen.dtb

To enable the PSUART assignment to a domU. Below is one example for the dom create config file. 

1 2 3 4 irqs = [ 54 ] iomem = [ "0xff010,1" ] device_tree = "/root/passthrough-serial.dtb" dtdev = [ "/amba/serial@ff010000" ]

In above config file, irqs and iomem entries are added. Here is detailed explanation for these entries:

IRQS calculation

To calculate irqs corresponding to the guest domain, it needs to find the following line in UART settings described in xen.dts file:

1 2 3 4 5 serial@ff010000 { compatible = "cdns,uart-r1p12", "xlnx,xuartps"; status = "okay"; interrupt-parent = <0xfde8>; interrupts = <0x0 0x16 0x4>;

take the interrupt which is device specific, in this case interrupt number is 0x16, add 32 which is the base address of interrupt handler.

0x16 = 22
irqs = 22 + 32 = 54

Setup IOMEM field

iomem = [ "0xff010,1" ]
1 page starting at "0xFF010000" base address of UART PS.

Creation of passthrough-serial.dts:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 /dts-v1/; / { #address-cells = <0x2>; #size-cells = <0x2>; passthrough { compatible = "simple-bus"; ranges; #address-cells = <0x2>; #size-cells = <0x2>; pss_ref_clk { compatible = "fixed-clock"; #clock-cells = <0x0>; clock-frequency = <0x1fc9350>; phandle = <0x6>; }; video_clk { compatible = "fixed-clock"; #clock-cells = <0x0>; clock-frequency = <0x19bfcc0>; phandle = <0x7>; }; pss_alt_ref_clk { compatible = "fixed-clock"; #clock-cells = <0x0>; clock-frequency = <0x0>; phandle = <0x8>; }; gt_crx_ref_clk { compatible = "fixed-clock"; #clock-cells = <0x0>; clock-frequency = <0x66ff300>; phandle = <0xa>; }; aux_ref_clk { compatible = "fixed-clock"; #clock-cells = <0x0>; clock-frequency = <0x19bfcc0>; phandle = <0x9>; }; firmware { zynqmp-firmware { compatible = "xlnx,zynqmp-firmware"; method = "smc"; #power-domain-cells = <0x1>; phandle = <0x26>; clock-controller { u-boot,dm-pre-reloc; #clock-cells = <0x1>; compatible = "xlnx,zynqmp-clk"; clocks = <0x6 0x7 0x8 0x9 0xa>; clock-names = "pss_ref_clk", "video_clk", "pss_alt_ref_clk", "aux_ref_clk", "gt_crx_ref_clk"; phandle = <0x3>; }; }; }; serial@ff010000 { compatible = "cdns,uart-r1p12", "xlnx,xuartps"; status = "okay"; interrupt-parent = <0xfde8>; interrupts = <0x0 0x16 0x4>; reg = <0x0 0xff010000 0x0 0x1000>; clock-names = "uart_clk", "pclk"; clocks = <0x3 0x39 0x3 0x1f>; device_type = "serial"; port-number = <0x1>; xen,path = "/amba/serial@ff010000"; xen,reg = <0x0 0xff010000 0x1000 0x0 0xff010000>; }; }; };

Copy the above device tree code snippet and save it as passthrough-serial.dts and convert the above dtc into dtb using dtc compiler:

1 dtc -I dts -O dtb -o passthrough-serial.dtb passthrough-serial.dts

Finally, create a new guest using ‘xl create -c configfile'.

 

If your domU is Linux, make sure to add console=ttyPS1,115200 to its kernel command line. In case of dom0less DomUs, you have to edit the boot.source script to add the command line option, then recreate boot.scr with mkimage.

PCI Passthrough assignment

Turn xen.dtb into xen.dts:

1 dtc -I dtb -O dts xen.dtb > xen.dts

The, edit xen.dts by adding xen,passthrough; under the node of the device to assign, in this case pcie@fd0e0000:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 pcie@fd0e0000 { compatible = "xlnx,nwl-pcie-2.11"; status = "disabled"; #address-cells = <0x3>; #size-cells = <0x2>; #interrupt-cells = <0x1>; msi-controller; device_type = "pci"; interrupt-parent = <0x4>; interrupts = <0x0 0x76 0x4 0x0 0x75 0x4 0x0 0x74 0x4 0x0 0x73 0x4 0x0 0x72 0x4>; interrupt-names = "misc", "dummy", "intx", "msi1", "msi0"; msi-parent = <0x31>; reg = <0x0 0xfd0e0000 0x0 0x1000 0x0 0xfd480000 0x0 0x1000 0x80 0x0 0x0 0x1000000>; reg-names = "breg", "pcireg", "cfg"; ranges = <0x2000000 0x0 0xe0000000 0x0 0xe0000000 0x0 0x10000000 0x43000000 0x6 0x0 0x6 0x0 0x2 0x0>; interrupt-map-mask = <0x0 0x0 0x0 0x7>; bus-range = <0x0 0xff>; interrupt-map = <0x0 0x0 0x0 0x1 0x32 0x1 0x0 0x0 0x0 0x2 0x32 0x2 0x0 0x0 0x0 0x3 0x32 0x3 0x0 0x0 0x0 0x4 0x32 0x4>; power-domains = <0x26 0x3b>; clocks = <0x3 0x17>; xlnx,bar0-enable = <0x0>; xlnx,bar1-enable = <0x0>; xlnx,bar2-enable = <0x0>; xlnx,bar3-enable = <0x0>; xlnx,bar4-enable = <0x0>; xlnx,bar5-enable = <0x0>; xlnx,pcie-mode = "Root Port"; xlnx,tz-nonsecure = <0x0>; phandle = <0x31>; iommus = <0x28 0x4d0>; xen,passthrough = <1>; legacy-interrupt-controller { interrupt-controller; #address-cells = <0x0>; #interrupt-cells = <0x1>; phandle = <0x32>; }; };

Since the PCIe is a DMA-capable device, it should be included in the SMMU section of the xen.dts. Make sure that the PCIe base address in the IOMMU address map exists in the list of mmu-masters. Please refer the latest version of Zynq Ultrascale + MPSoC TRM (UG1085):
a) Find PCIe master device in the table given in the section AXI Master IDs and pick the corresponding master ID, which is 0011010000.
b) Expand these 10 bits to 12 bits, the top 2 bits are simply indicating which TBUn device is behind, this can be found in the Zynq UltraScale+ MPSoC TRM
Interconnect Functional Description section. For PCIe it is TBU1, i.e. the address will be 010011010000=0x4d0.

Add xen,passthrough and iommus cells in xen.dts. Convert xen.dts back into xen.dtb.

Creation of passthrough-pci.dts:
Please see attached passthrough-pci.dts for example of partial device tree file and convert the above dtc into dtb using dtc compiler:

1 dtc -I dts -O dtb -o passthrough-pci.dtb passthrough-pci.dts

Finally, The next step is to configure guest domain to correspond to the pass-through mode.
The guest configuration file should look as follows:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 # Guest name name = "guest0" # Kernel image to boot kernel = "/boot/Image" ramdisk = "/boot/xen-rootfs.cpio.gz" extra = "root=/dev/ram0 init=/bin/sh console=hvc0 rdinit=/sbin/init" # Initial memory allocation (MB) memory = 1024 # Number of VCPUS vcpus = 2 # Passthrough dtdev = [ "/amba/pcie@fd0e0000" ] device_tree = "/etc/xen/domu.dtb" irqs = [ 146, 147, 148, 150 ] iomem = [ "0xfd0e0,1", "0xfd480,1", "0xe0000,1000"]

The options needed for pass-through mode are defined below:
dtdev - The absolute path of the device to be passed through in the device tree
device_tree - absolute path of partial device tree in the Domain 0
irqs - interrupt number inside interrupt queue for the guest domain (see calculation steps below)
iomem - the number of physical memory pages to be passed to the guest (see calculation steps below)

IRQS calculation

  • to calculate irqs corresponding to the guest domain, it needs to find the following line in PCIe settings described in xen.dts file:

1 interrupts = <0x0 0x76 0x4 0x0 0x74 0x4 0x0 0x73 0x4 0x0 0x72 0x4>;
  • take the interrupt which is device specific, in this case 0x76, 0x74, 0x73, 0x72 the rest of interrupts appears elsewhere as well, add 32 which is thebase address of interrupt handler.

1 2 3 4 5 0x72 = 114 irqs[1] = 114 + 32 = 146 irqs[2] = 115 + 32 = 147 irqs[3] = 116 + 32 = 148 irqs[4] = 118 + 32 = 150

Setup IOMEM field

iomem = [ "0xfd0e0,1", "0xfd480,1", "0xe0000,1000"]
1 page starting at "0xFD0E0000" base address of AXIPCIE_MAIN register (AXI to PCIe Bridge, Main Control and Status registers), 1 page starting at "0xFD480000" base address of PCIE_ATTRIB register (Register block that holds the attributes of the PCIe Controller, PCIe Attributes) and 1000 page starting at 0xe0000 base address for data.

Note that PCIe is mapped to multiple regions and it all needs to be on one single iomem line.

Check Starting Linux guests with Pass-through networking for another passthrough example.