Building Xen Hypervisor with Petalinux 2021.2
This page includes information on how to build and deploy Xen on Xilinx boards, including both Xilinx Ultrascale+ and Versal boards.
Table of Contents
Overview
The guide below shows you how to build Xen, boot Xen and then run some example configurations. The steps below use PetaLinux and assume you have some knowledge of using PetaLinux. Before starting you need to create a PetaLinux project. It is assumed that a default PetaLinux reference design is used unchanged in these instructions. The default PetaLinux configuration has images ready to do boot Xen, these are the pre-built images. You can use those or you can manually edit recipes and build Xen yourself. The pre-built images can be found in this directory (inside a PetaLinux project) pre-built/linux/xen. You can either use the pre-builts or follow the next section to configure and build Xen yourself. If you are using the pre-builts you can skip to the booting Xen section for your release version.
Configuring and building XEN from source using PetaLinux
First let's create a Petalinux project
petalinux-create -t project --template zynqMP --force -n xilinx-zcu
Then let's enable Xen to be built by default.
$ petalinux-config -c rootfs
Petalinux Package Groups ---> packagegroup-petalinux-xen ---> [*] packagegroup-petalinux-xen Filesystem Packages ---> console ---> tools ---> xen ---> [*] xen-tools
$ petalinux-config
Image Packaging Configuration ---> Root filesystem type (INITRAMFS) ---> (X) INITRD Image Packaging Configuration ---> Update/edit the name petalinux-initramfs-image to petalinux-image-minimal
NOTE: This means that any images built will NOT have the rootFS in the Image that is built by PetaLinux. This means you will need to edit any scripts or configuration that expects the rootFS to be included. This includes the Xen configs mentioned later.
You can still use the prebuilt Image file which does still include the rootFS to boot DomU.
We also want to edit the device tree to build in the extra Xen related configs.
Edit this file
project-spec/meta-user/recipes-bsp/device-tree/files/system-user.dtsi
It should look like this for harware:
/include/ "system-conf.dtsi" /include/ "xen.dtsi" / { };
/include/ "system-conf.dtsi" /include/ "xen.dtsi" / { cpus { cpu@1 { device_type = "none"; }; cpu@2 { device_type = "none"; }; cpu@3 { device_type = "none"; }; }; };
Also edit this file:
project-spec/meta-user/recipes-bsp/device-tree/device-tree.bbappend
The file should look like this:
FILESEXTRAPATHS_prepend := "${THISDIR}/files:" SRC_URI += "file://system-user.dtsi" SRC_URI += "file://xen.dtsi"
Then run petaliux-build:
$ petalinux-build
TFTP Booting Xen and Dom0
Run Xen dom0 on QEMU
To use the prebuilt Xen run:$ petalinux-boot --qemu --prebuilt 2 --qemu-args "-net nic -net nic -net nic -net nic -net user,tftp=pre-built/linux/xen"
$ petalinux-boot --qemu --u-boot --qemu-args "-net nic -net nic -net nic -net nic -net user,tftp=images/linux"
Run Xen dom0 on HW
To use the prebuilt Xen on hardware:$ petalinux-boot --jtag --prebuilt 2
$ petalinux-boot --jtag --u-boot
Hit any key to stop autoboot:
u-boot> setenv serverip 10.0.2.2 u-boot> setenv serverip 10.0.2.15
Now to download and boot Xen, if running on QEMU, use xen-qemu.dtb otherwise use xen.dtb. Example:
TFTPing Xen using ImageBuilder
ImageBuilder is a set of Open Source community scripts to automatically configure a Xen system with Dom0 and multiple Dom0-less VMs for booting. ImageBuilder can generate a U-Boot script that loads all of the binaries automatically and boot the full system quickly. ImageBuilder is available here. The main script is scripts/uboot-script-gen and its usage is described in details on the Xen Project wikipage.
Petalinux prebuilt binaries can be used in a config file as follows for uboot-script-gen:
MEMORY_START="0x0" MEMORY_END="0x80000000" DEVICE_TREE="xen.dtb" XEN="xen" DOM0_KERNEL="xen-Image" DOM0_RAMDISK="xen-rootfs.cpio.gz" NUM_DOMUS=0 UBOOT_SOURCE="boot.source" UBOOT_SCRIPT="boot.scr"
Now uboot-script-gen can be used to generate boot.scr:
$ bash ./scripts/uboot-script-gen -c config -d . -t tftp
Boot the system with the following uboot command (assuming the tftp serverip is 10.0.2.2, which is typically the value for QEMU):
u-boot> setenv serverip 10.0.2.2 u-boot> tftpb 0xC00000 boot.scr; source 0xC00000
The Xen and Dom0 command line are generated by uboot-script-gen. If you would like to change anything, for instance increase the dom0 memory allocation, it is always possible by editing boot.source. Simply do the following:
- edit boot.source, change dom0_mem to dom0_mem=2G
- regenerate boot.scr with the following command: mkimage -A arm64 -T script -C none -a 0xC00000 -e 0xC00000 -d boot.source boot.scr
SD Booting Xen and Dom0
To boot Xen from an SD card you need to copy the following files to the boot partition of the SD card:- boot.scr
- xen-Image
- the compiled device tree file renamed to system.dtb (xen.dtb or xen-qemu.dtb for QEMU from the pre-built images, system.dtb from a Petalinux build)
- xen
- xen-rootfs.cpio.gz)
When using the pre-built images from the BSP, copy these files from <project-dir>/pre-built/linux/xen.
Booting with ImageBulider
ImageBuilder's script uboot-script-gen can be used to generate a uboot script that loads all the binaries automatically from MMC. Call uboot-script-gen with the following command, assuming that $sdbootdev is 0 and $partid is 1:$ bash ./scripts/uboot-script-gen -c config -d . -t "load mmc 0:1"
$ mmc dev $sdbootdev &&&& mmcinfo; load mmc $sdbootdev:$partid 0xC00000 boot.scr; source 0xC00000
In case of qemu, one needs to generate the sd.img using the following command.
sudo bash -x ./scripts/disk_image -c config -d . -t sd -w tmp -a 512 -o sd.img
And then boot with the following command
petalinux-boot --qemu --u-boot --qemu-args "-net nic -net nic -net nic -net nic -net user,tftp=pre-built/linux/xen -drive file=pre-built/linux/xen/sd.img,if=sd,format=raw,index=1 -boot mode=1"
Starting simple additional guests
If running on QEMU, we'll need to setup a port mapping for port 22 (SSH) in our VM.In this example, we forward the hosts port 2222 to the VM's port 22.
$ petalinux-boot --qemu --u-boot --qemu-args "-net nic -net nic -net nic -net nic -net user,tftp=pre-built/linux/xen,hostfwd=tcp:127.0.0.1:2222-10.0.2.15:22"
Once you hit the u-boot prompt, follow the steps in the earlier section on how to run Xen dom0.
When dom0 has finished booting, we'll need to copy a guest Image into dom0's filesystem.
We'll use the base prebuilt PetaLinux Image as our domU guest.
If running on QEMU, we use scp's -P option to connect to our hosts port 2222 where QEMU will forward the connection to the guests port 22:
To target QEMU run the following on the host:
scp -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -P 2222 pre-built/linux/xen/xen-Image root@localhost:/boot/Image
scp -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no pre-built/linux/xen/xen-Image root@<board-ip>:/boot/Image
The xen-image-minimal rootFS includes some prepared configurations that you can use. These are located in '/etc/xen/'
# cd /etc/xen
scp -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -P 2222 pre-built/linux/xen/xen-rootfs.cpio.gz root@localhost:/boot/
name = "guest0" kernel = "/boot/Image" ramdisk = "/boot/xen-rootfs.cpio.gz" extra = "console=hvc0 rdinit=/sbin/init root=/dev/ram0" memory = 1024 vcpus = 2
# xl create -c example-simple.cfg
At any time you can leave the console of the guest and get back to dom0 by pressing ctrl+].
Once at the dom0 prompt you can list the guests from dom0:
# xl list
# xl console guest0
# xl create example-simple.cfg name=\"guest1\" # xl create example-simple.cfg name=\"guest2\" root@plnx_aarch64:/etc/xen# xl list Name ID Mem VCPUs State Time(s) Domain-0 0 512 1 r----- 79.8 Domain-0 0 512 1 r----- 79.8 guest0 1 256 2 ------ 93.7 guest1 2 256 2 ------ 26.6 guest2 3 256 2 ------ 1.8
# xl destroy guest0
CPU Pinning
The following will only work on QEMU with multi-core enabled or on real HW.When running multiple guests with multiple Virtual CPUs, Xen will schedule the various vCPUs onto real physical CPUs.
The rules and considerations taken in scheduling decisions depend on the chosen scheduler and the configuration.
To avoid having multiple vCPUs share a single pCPU, it is possible to pin a vCPU onto a pCPU and to give it exclusive access.
To create a simple guest with one Virtual CPU pinned to Physical CPU #3, you can do the following:
xl create example-simple.cfg 'name="g0"' 'vcpus="1"' 'cpus="3"'
Another way to pin virtual CPUs on to Physical CPUs is to create dedicated cpu-pools.
This has the advantage of isolating the scheduling instances.
By default a single cpu-pool named Pool-0 exists. It contains all the physical cpus.
We'll now create our pool named rt using the credit2 scheduler.
xl cpupool-create 'name="rt"' 'sched="credit"' xl cpupool-cpu-remove Pool-0 3 xl cpupool-cpu-add rt 3
xl create /etc/xen/example-simple.cfg 'vcpus="1"' 'pool="rt"' 'cpus="3"' 'name="g0"'
Starting Linux guests with Para-Virtual networking (PV network)
The default IP addresses assigned by QEMUs builtin DHCP server start from 10.0.2.15 and count upwards.
Dom0 will be assigned 10.0.2.15, the next guest 10.0.2.16 and so on.
So here's the command line that maps host port 2222 to dom0 port 22 and 2322 to domUs port 22.
$ petalinux-boot --qemu --u-boot --qemu-args "-net nic -net nic -net nic -net nic -net user,tftp=pre-built/linux/xen,hostfwd=tcp:127.0.0.1:2222-10.0.2.15:22,hostfwd=tcp:127.0.0.1:2322-10.0.2.16:22"
Now, follow the instructions from section 1 on how to boot Xen dom0.
Once you are at the dom0 prompt and have copied a domU image (see earlier steps) we'll need to setup the networking.
In this example, we will configure the guests to directly join the external network by means of a bridge.
First of all, we need to de-configure the default setup.
Kill the dhcp client for eth0:
# killall -9 udhcpc
# ip addr show dev eth0
# ip addr del 10.0.2.15/24 dev eth0
# brctl addbr xenbr0 # brctl addif xenbr0 eth0 # /sbin/udhcpc -i xenbr0 -b
udhcpc (v1.24.1) started [ 165.460858] xenbr0: port 1(eth0) entered blocking state [ 165.461819] xenbr0: port 1(eth0) entered forwarding state Sending discover... Sending select for 10.0.2.15... Lease of 10.0.2.15 obtained, lease time 86400 /etc/udhcpc.d/50default: Adding DNS 10.0.2.3
# cd /etc/xen
# xl create -c example-pvnet.cfg
Now we'll ssh into the domU from the host running Para-Virtual networking:
$ ssh -p 2322 root@localhost
Starting Linux guests with Pass-through networking
It is possible to directly assign the network peripheral to a domU on both ZU+ and Versal. The following example is for ZU+.
Turn xen.dtb into xen.dts:
dtc -I dtb -O dts xen.dtb > xen.dts
The, edit xen.dts by adding xen,passthrough; under the node of the device to assign, in this case ethernet@ff0e0000:
ethernet@ff0e0000 { compatible = "cdns,zynqmp-gem"; status = "enabled"; interrupt-parent = <0x2>; interrupts = <0x0 0x3f 0x4 0x0 0x3f 0x4>; reg = <0x0 0xff0e0000 0x0 0x1000>; clock-names = "pclk", "hclk", "tx_clk"; #address-cells = <0x1>; #size-cells = <0x0>; #stream-id-cells = <0x1>; iommus = <0x6 0x77>; power-domains = <0x10>; clocks = <0xd 0xd 0xd>; phy-mode = "rgmii-id"; xlnx,ptp-enet-clock = <0x0>; local-mac-address = [00 0a 35 00 02 90]; phy-handle = <0x11>; linux,phandle = <0x22>; phandle = <0x22>; xen,passthrough; phy@c { reg = <0xc>; ti,rx-internal-delay = <0x8>; ti,tx-internal-delay = <0xa>; ti,fifo-depth = <0x1>; linux,phandle = <0x11>; phandle = <0x11>; }; };
Convert xen.dts back into xen.dtb:
dtc -I dts -O dtb xen.dts > xen.dtb
Use this xen-rootfs-custom.cpio.gz for dom0 as it has /boot/xen-Image and /boot/xen-rootfs.cpio.gz
The following is the config file to be used with imagebuilder/scripts/uboot-script-gen
MEMORY_START="0x0" MEMORY_END="0x80000000" DEVICE_TREE="xen.dtb" XEN="xen" DOM0_KERNEL="xen-Image" DOM0_RAMDISK="xen-rootfs-custom.cpio.gz" NUM_DOMUS=0 UBOOT_SOURCE="boot.source" UBOOT_SCRIPT="boot.scr"
When dom0 boots up, one can confirm that ethernet is not up.
root@xilinx-zcu102-2021_2:/etc/xen# ifconfig lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:65536 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
Edit /etc/xen/example-passnet.cfg to make it as follows
name = "guest0" kernel = "/boot/xen-Image" ramdisk = "/boot/xen-rootfs.cpio.gz" extra = "console=hvc0 rdinit=/sbin/init root=/dev/ram0 init=/bin/sh" memory = 1024 vcpus = 2 dtdev = [ "/axi/ethernet@ff0e0000" ] device_tree = "/etc/xen/passthrough-example-part.dtb" irqs = [ 95 ] iomem = [ "0xff0e0,1" ]
Then, boot the guest0
xl create -c example-passnet.cfg
And confirm that guest0 can access ethernet
root@xilinx-zcu102-2021_2:~# ifconfig eth0 Link encap:Ethernet HWaddr 00:0A:35:00:22:01 inet addr:10.0.2.16 Bcast:10.0.2.255 Mask:255.255.255.0 inet6 addr: fe80::20a:35ff:fe00:2201/64 Scope:Link inet6 addr: fec0::20a:35ff:fe00:2201/64 Scope:Site UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:9 errors:0 dropped:0 overruns:0 frame:0 TX packets:134 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:1824 (1.7 KiB) TX bytes:36092 (35.2 KiB) Interrupt:13 lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:65536 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
Starting a Guest with a Passthrough SD Card
It is possible to directly assign the MMC controller to a domU on both ZU+ and Versal. The following example is for ZU+ only. It allows a Xen DomU full unmediated access to a SD card plugged into the board.
Add the following lines to the xl config file for the VM:
name = "guest0" ramdisk = "/boot/xen-rootfs.cpio.gz" kernel = "/boot/Image" extra = "console=hvc0 rdinit=/sbin/init" memory = 1024 vcpus = 2 dtdev = [ "/axi/mmc@ff170000" ] device_tree = "/etc/xen/passthrough-example-part_mmc.dtb" irqs = [ 81 ] iomem = [ "0xff170,1", "0xff110,1", "0xff120,1", "0xff130,1", "0xff140,1" ]
The partial device tree is here https://github.com/Xilinx/xen-passthrough-device-trees/blob/master/device-trees-2021.2/mmc%40ff170000.dts
Also, you need to add xen,passthrough; under the mmc node in the host device tree so that it doesn't get automatically assigned to dom0:
mmc@ff170000 { [...] clock-frequency = <0xb2cbcae>; xlnx,mio_bank = <0x1>; phandle = <0x24>; xen,passthrough; /* add this line */ };
Finally, make sure to run the following commands at boot from xsdb to configure the system so that normal-world MMC DMA goes via the SMMU:
mwr -force 0xff180408 0x20 mwr -force 0xFF240000 0x100492 mwr -force 0xFF240004 0x100492
Dom0less
For MMC assignment to a dom0less guest, use the partial device tree :https://github.com/Xilinx/xen-passthrough-device-trees/blob/master/device-trees-2021.2/mmc%40ff170000.dts
To use it, the following are the contents of the ImageBuilder config:
MEMORY_START="0x0" MEMORY_END="0x80000000" DEVICE_TREE="xen.dtb" XEN="xen" DOM0_KERNEL="xen-Image" DOM0_RAMDISK="xen-rootfs-custom.cpio.gz" DOMU_KERNEL[0]="xen-Image" DOMU_RAMDISK[0]="xen-rootfs-custom.cpio.gz" DOMU_PASSTHROUGH_DTB[0]="passthrough-example-part-dom0less.dtb" NUM_DOMUS=1 UBOOT_SOURCE="boot-dom0less.source" UBOOT_SCRIPT="boot-dom0less.scr"
Regenerate the boot.scr and boot.source scripts as usual with ImageBuilder’s uboot-script-gen.
As for the traditional Dom0 case, it is also necessary to add xen,passthrough; under the mmc node in the host device tree, and issue the three special writes at boot time to configure the system so that normal-world MMC DMA goes via the SMMU.
Starting a guest with a passthrough SATA disk
It is possible to directly assign the SATA controller to a domU on both ZU+ and Versal. The following example is for ZU+ only. It allows a Xen DomU full unmediated access to any SATA disks connected to it.
Add the following lines to the xl config file for the VM:
# Device passthroughs dtdev = [ "/axi/ahci@fd0c0000" ] device_tree = "/path/to/passthrough-example-sata.dtb" irqs = [ 165 ] iomem = [ "0xfd0c0,2" ]
Where passthrough-example-sata.dts is https://github.com/Xilinx/xen-passthrough-device-trees/blob/master/device-trees-2021.2/ahci%40fd0c0000.dts
Then, you need to add xen,passthrough; under the ahci@fd0c0000 node in the host device tree so that it doesn't get automatically assigned to dom0:.
It is also necessary to add the four SMIDs of the SATA controller to device tree: 0x4c0, 0x4c1, 0x4c2, 0x4c3:
[...] smmu@fd800000 { mmu-masters = <... 0x60 0x4c0 0x4c1 0x4c2 0x4c3>; } [...] ahci@fd0c0000 { [...] #stream-id-cells = <0x04>; iommus = <0x28 0x4c0>, <0x28 0x4c1>, <0x28 0x4c2>, <0x28 0x4c3>; phy-names = "sata-phy"; phys = <0x35 0x1 0x1 0x1 0x7735940>; xlnx,tz-nonsecure-sata0 = <0x0>; xlnx,tz-nonsecure-sata1 = <0x0>; xen,passthrough; /* add this line */ };
Finally, make sure to run the following commands at boot from xsdb to configure the system so that normal-world SATA DMA goes via the SMMU:
mask_write 0xFD690020 0x0000000F 0x0000000F
Starting a guest with passthrough UART1 serial
It is possible to assign serial@ff010000 to a domU.
For this, one needs to use the following imagebuilder config file (called config_serial0_xen)
MEMORY_START="0x0" MEMORY_END="0x80000000" DEVICE_TREE="xen.dtb" XEN="xen" DOM0_KERNEL="xen-Image" DOM0_RAMDISK="xen-rootfs.cpio.gz" XEN_CMD="console=dtuart dtuart=serial0 dom0_mem=2G dom0_max_vcpus=1 bootscr0000" NUM_DOMUS=0 UBOOT_SOURCE="boot.source" UBOOT_SCRIPT="boot.scr"
And run "bash <path_to>/imagebuilder/scripts/uboot-script-gen -c config_serial0_xen -d . -t tftpb"
Use the https://github.com/Xilinx/xen-passthrough-device-trees/blob/master/device-trees-2021.2/serial%40ff010000.dts for a dom0less domU. For a regular domU, add the following lines to the domU config file:
name = "linux_serial1" kernel = "/boot/Image" ramdisk = "/etc/xen/xen-rootfs-custom.cpio.gz" extra = "earlycon console=ttyPS0,115200 clk_ignore_unused rdinit=/sbin/init root=/dev/ram0 init=/bin/sh" memory = 1024 vcpus = 1 device_tree = "/etc/xen/serial@ff010000.dtb" cpus = [1] irqs = [ 54 ] iomem = [ "0xff010,1", "0xff110,1", "0xff120,1", "0xff130,1", "0xff140,1" ]
If your domU is Linux, make sure to add console=ttyPS0,115200 to its kernel command line. In case of dom0less DomUs, you have to edit the boot.source script to add the command line option, then recreate boot.scr with mkimage.
Starting a guest with passthrough UART0 serial
It is possible to assign serial@ff000000 to a domU.
For this, you need to assign serial1 (ie serial@0xff010000 to Xen and Dom0). Use https://github.com/Xilinx/xen-passthrough-device-trees/blob/master/device-trees-2021.2/serial%40ff000000.dts for the dts
Use the following imagebuilder config file.
MEMORY_START="0x0" MEMORY_END="0x80000000" DEVICE_TREE="xen.dtb" XEN="xen" DOM0_KERNEL="xen-Image" DOM0_RAMDISK="xen-rootfs.cpio.gz" XEN_CMD="console=dtuart dtuart=serial1 dom0_mem=2G dom0_max_vcpus=1 bootscrub=0 vwfi=native sched=null" NUM_DOMUS=0 UBOOT_SOURCE="boot.source" UBOOT_SCRIPT="boot.scr"
And run "bash <path_to>/imagebuilder/scripts/uboot-script-gen -c config_serial1_xen -d . -t tftpb"
For the regular domU, add the following lines to the domU config file.
name = "linux_serial1" kernel = "/boot/Image" ramdisk = "/etc/xen/xen-rootfs-custom.cpio.gz" extra = "earlycon console=ttyPS0,115200 clk_ignore_unused rdinit=/sbin/init root=/dev/ram0 init=/bin/sh" memory = 1024 vcpus = 1 device_tree = "/etc/xen/serial@ff000000.dtb" cpus = [1] irqs = [ 53 ] iomem = [ "0xff000,1", "0xff110,1", "0xff120,1", "0xff130,1", "0xff140,1" ]
© Copyright 2019 - 2022 Xilinx Inc. Privacy Policy