Building Xen Hypervisor with Petalinux 2019.2
Table of Contents
Overview
The guide below shows you how to build Xen, boot Xen and then run some example configurations on ZU+. The steps below use PetaLinux and assume you have some knowledge of using PetaLinuxThe default PetaLinux configuration has images ready to do boot Xen, these are the pre-built images. You can use those or you can manually edit receipes and build Xen yourself. The pre-built images can be found in this directory (inside a PetaLinux project) pre-built/linux/images/ and prefixed with "xen-". You can either use the pre-builts or follow the next section to configure and build Xen yourself. If you are using the pre-builts you can skip to the booting Xen section for your release version.
Configuring and building XEN from source using PetaLinux
First let's enable Xen to be built by default.$ petalinux-config -c rootfs
Petalinux Package Groups ---> packagegroup-petalinux-xen ---> [*] packagegroup-petalinux-xen
Now we need to change the rootFS to be an INITRD
$ petalinux-config
Image Packaging Configuration ---> Root filesystem type (INITRAMFS) ---> (X) INITRD
You can still use the prebuilt Image file which does still include the rootFS to boot DomU.
We also want to edit the device tree to build in the extra Xen related configs.
Edit this file
project-spec/meta-user/recipes-bsp/device-tree/files/system-user.dtsi
It should look like this for harware:
/include/ "system-conf.dtsi" /include/ "xen.dtsi" / { };
/include/ "system-conf.dtsi" /include/ "xen.dtsi" / { cpus { cpu@1 { device_type = "none"; }; cpu@2 { device_type = "none"; }; cpu@3 { device_type = "none"; }; }; };
Also edit this file:
project-spec/meta-user/recipes-bsp/device-tree/device-tree.bbappend
The file should look like this:
FILESEXTRAPATHS_prepend := "${THISDIR}/files:" SRC_URI += "file://system-user.dtsi" SRC_URI += "file://xen.dtsi"
Then run petaliux-build:
$ petalinux-build
TFTP Booting Xen and Dom0
Run Xen dom0 on QEMU:
To use the prebuilt Xen run:$ petalinux-boot --qemu --prebuilt 2 --qemu-args "-net nic -net nic -net nic -net nic -net user,tftp=pre-built/linux/images"
$ petalinux-boot --qemu --u-boot
Run Xen dom0 on HW:
To use the prebuilt Xen on hardware:$ petalinux-boot --jtag --prebuilt 2
$ petalinux-boot --jtag --u-boot
Hit any key to stop autoboot:
u-boot> setenv serverip 10.0.2.2
Now to download and boot Xen, if running on QEMU, use xen-qemu.dtb otherwise use xen.dtb. Example:
TFTPing Xen using ImageBuilder
ImageBuilder is a set of Open Source community scripts to automatically configure a Xen system with Dom0 and multiple Dom0-less VMs for booting. ImageBuilder can generate a U-Boot script that loads all of the binaries automatically and boot the full system quickly. ImageBuilder is available here. The main script is scripts/uboot-script-gen and its usage is described in details on the Xen Project wikipage.
uboot-script-gen takes raw binaries as input (not uboot binaries). Petalinux pre-build images provide u-boot binaries for Xen and the Dom0 rootfs. First, they need to be converted into raw binaries:
dd if=xen.ub of=xen bs=64 skip=1 dd if=xen-rootfs.cpio.gz.u-boot of=xen-rootfs.cpio.gz bs=64 skip=1
Then, they can be used in a config file as follows for uboot-script-gen:
MEMORY_START="0x0" MEMORY_END="0x80000000" DEVICE_TREE="xen.dtb" XEN="xen" DOM0_KERNEL="xen-Image" DOM0_RAMDISK="xen-rootfs.cpio.gz" NUM_DOMUS=0 UBOOT_SOURCE="boot.source" UBOOT_SCRIPT="boot.scr"
Now uboot-script-gen can be used to generate boot.scr:
$ bash ./scripts/uboot-script-gen -c config -d . -t tftp
Boot the system with the following uboot command (assuming the tftp serverip is 10.0.2.2, which is typically the value for QEMU):
u-boot> setenv serverip 10.0.2.2 u-boot> tftpb 0xC00000 boot.scr; source 0xC00000
The Xen and Dom0 command line are generated by uboot-script-gen. If you would like to change anything, for instance increase the dom0 memory allocation, it is always possible by editing boot.source. Simply do the following:
- edit boot.source, change dom0_mem to dom0_mem=2G
- regenerate boot.scr with the following command: mkimage -A arm64 -T script -C none -a 0xC00000 -e 0xC00000 -d boot.source boot.scr
TFTPing Xen from pre-built images manually
$ tftpb 1280000 xen-qemu.dtb; tftpb 0x80000 xen-Image; tftpb 1400000 xen.ub; tftpb 9000000 xen-rootfs.cpio.gz.u-boot; bootm 1400000 9000000 1280000
Booting with larger binaries and Workaround for ZCU104
Any time you update binaries and they potentially increase in size, please make sure that they don't overlap. For instance, if the kernel Image is loaded at 0x80000 and the dtb at 0x1280000, it means that the maximum size of the kernel is 12MB. More than that and the loading addresses need to be updated to avoid overlaps. Moreover, both address and size of the kernel Image are also specified under the /chosen node in the dtb file.
The ZCU104 BSP comes with a larger Dom0 rootfs (xen-rootfs.cpio.gz.u-boot file). U-boot has issues booting the rootfs as a u-boot image, it needs to be loaded as a raw binary instead. Thus, the following workaround is required for ZCU104.
Convert the xen-rootfs back to a normal binary:
$ dd if=xen-rootfs.cpio.gz.u-boot of=xen-rootfs.cpio.gz. bs=64 skip=1
Add the following node under /chosen in device tree xen-qemu.dts (you can convert a device tree binary to source and back using dtc):
dom0-ramdisk { compatible = "xen,linux-initrd", "xen,multiboot-module"; reg = <0x0 0x9000000 152373016>; };
Making sure that the size of the ramdisk (152373016 in the example) matches the actual size of xen-rootfs.cpio.gz.u-boot. Also add root=/dev/ram0 to the dom0 command line on device tree (/chosen/xen,dom0-bootargs).
Boot the system with the following command:
u-boot> tftpb 1280000 xen-qemu.dtb; tftpb 0x80000 xen-Image; tftpb 1400000 xen.ub; tftpb 9000000 xen-rootfs.cpio.gz; bootm 1400000 - 1280000
TFTPing Xen from your own images
u-boot> tftpb 1280000 system.dtb; tftpb 0x80000 Image; tftpb 1400000 xen.ub; tftpb 9000000 rootfs.cpio.gz.u-boot; bootm 1400000 9000000 1280000
[. . .] BOOTP broadcast 1 DHCP client bound to address 10.0.2.15 (2 ms) Hit any key to stop autoboot: 0 ZynqMP> setenv serverip 10.0.2.2 ZynqMP> tftpb 1280000 xen-qemu.dtb; tftpb 0x80000 Image; tftpb 1400000 xen.ub; tftpb 9000000 rootfs.cpio.gz.u-boot; bootm 1400000 9000000 1280000 Using ethernet@ff0e0000 device TFTP from server 10.0.2.2; our IP address is 10.0.2.15 Filename 'xen-qemu.dtb'. Load address: 0x1280000 Loading: ### 18.3 MiB/s done Bytes transferred = 38412 (960c hex) Using ethernet@ff0e0000 device TFTP from server 10.0.2.2; our IP address is 10.0.2.15 Filename 'Image'. Load address: 0x80000 Loading: ################################################################# ################################################################# ################################################################# ################################################################# ################################################################# ################################################################# ################################################################# ################################################################# ################################################################# ################################################################# ################################################################# ################################################################# [. . .] Bytes transferred = 49329157 (2f0b405 hex) ## Booting kernel from Legacy Image at 01400000 ... Image Name: Image Type: AArch64 Linux Kernel Image (uncompressed) Data Size: 754000 Bytes = 736.3 KiB Load Address: 06000000 Entry Point: 06000000 Verifying Checksum ... OK ## Loading init Ramdisk from Legacy Image at 09000000 ... Image Name: petalinux-user-image-zcu102-zynq Image Type: AArch64 Linux RAMDisk Image (uncompressed) Data Size: 49329093 Bytes = 47 MiB Load Address: 00000000 Entry Point: 00000000 Verifying Checksum ... OK ## Flattened Device Tree blob at 01000000 Booting using the fdt blob at 0x1000000 Loading Kernel Image ... OK Loading Ramdisk to 030f4000, end 05fff3c5 ... OK Loading Device Tree to 0000000007ff3000, end 0000000007fff60b ... OK Starting kernel ... Xen 4.9.2-pre (XEN) Xen version 4.9.2-pre (@) (aarch64-xilinx-linux-gcc (GCC) 7.2.0) debug=n Mon Mar 12 16:17:51 EDT 2018 (XEN) Latest ChangeSet: Sat Feb 24 07:47:11 2018 -0800 git:8c11d16-dirty (XEN) Processor: 410fd034: "ARM Limited", variant: 0x0, part 0xd03, rev 0x4 (XEN) 64-bit Execution: (XEN) Processor Features: 0000000000002222 0000000000000000 (XEN) Exception Levels: EL3:64+32 EL2:64+32 EL1:64+32 EL0:64+32 (XEN) Extensions: FloatingPoint AdvancedSIMD (XEN) Debug Features: 0000000010305106 0000000000000000 (XEN) Auxiliary Features: 0000000000000000 0000000000000000 (XEN) Memory Model Features: 0000000000001122 0000000000000000 (XEN) ISA Features: 0000000000011120 0000000000000000 (XEN) 32-bit Execution: (XEN) Processor Features: 00001231:00011011 (XEN) Instruction Sets: AArch32 A32 Thumb Thumb-2 ThumbEE Jazelle (XEN) Extensions: GenericTimer Security (XEN) Debug Features: 03010066 (XEN) Auxiliary Features: 00000000 (XEN) Memory Model Features: 10101105 40000000 01260000 02102211 (XEN) ISA Features: 02101110 13112111 21232042 01112131 00011142 00011121 (XEN) Generic Timer IRQ: phys=30 hyp=26 virt=27 Freq: 50000 KHz (XEN) GICv2 initialization: (XEN) gic_dist_addr=00000000f9010000 (XEN) gic_cpu_addr=00000000f9020000 (XEN) gic_hyp_addr=00000000f9040000 (XEN) gic_vcpu_addr=00000000f9060000 (XEN) gic_maintenance_irq=25 (XEN) GICv2: Adjusting CPU interface base to 0xf902f000 (XEN) GICv2: 192 lines, 4 cpus (IID 00000000). (XEN) Using scheduler: SMP Credit Scheduler (credit) (XEN) Allocated console ring of 16 KiB. (XEN) Bringing up CPU1 (XEN) Bringing up CPU2 (XEN) Bringing up CPU3 (XEN) Brought up 4 CPUs (XEN) P2M: 40-bit IPA with 40-bit PA and 8-bit VMID (XEN) P2M: 3 levels with order-1 root, VTCR 0x80023558 /amba@0/smmu0@0xFD800000: Decode error: write to 6c=0 (XEN) I/O virtualisation enabled (XEN) - Dom0 mode: Relaxed (XEN) Interrupt remapping enabled (XEN) *** LOADING DOMAIN 0 *** (XEN) Loading kernel from boot module @ 0000000000080000 (XEN) Loading ramdisk from boot module @ 00000000030f4000 (XEN) Allocating 1:1 mappings totalling 768MB for dom0: (XEN) BANK[0] 0x00000020000000-0x00000040000000 (512MB) (XEN) BANK[1] 0x00000060000000-0x00000070000000 (256MB) (XEN) Grant table range: 0x0000087fe00000-0x0000087fe5b000 (XEN) Loading zImage from 0000000000080000 to 0000000020080000-0000000023180000 (XEN) Loading dom0 initrd from 00000000030f4000 to 0x0000000028200000-0x000000002b10b3c5 (XEN) Allocating PPI 16 for event channel interrupt (XEN) Loading dom0 DTB to 0x0000000028000000-0x0000000028008f30 (XEN) Std. Loglevel: Errors and warnings (XEN) Guest Loglevel: Nothing (Rate-limited: Errors and warnings) (XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch input to Xen) (XEN) Freed 280kB init memory. (XEN) d0v0: vGICD: unhandled word write 0xffffffff to ICACTIVER4 (XEN) d0v0: vGICD: unhandled word write 0xffffffff to ICACTIVER8 (XEN) d0v0: vGICD: unhandled word write 0xffffffff to ICACTIVER12 (XEN) d0v0: vGICD: unhandled word write 0xffffffff to ICACTIVER16 (XEN) d0v0: vGICD: unhandled word write 0xffffffff to ICACTIVER20 (XEN) d0v0: vGICD: unhandled word write 0xffffffff to ICACTIVER0 [ 0.000000] Booting Linux on physical CPU 0x0 [. . .] Starting syslogd/klogd: done Starting /usr/sbin/xenstored... Setting domain 0 name, domid and JSON config... Done setting up Dom0 Starting xenconsoled... Starting QEMU as disk backend for dom0 Starting domain watchdog daemon: xenwatchdogd startup [done] Starting tcf-agent: OK PetaLinux 2018.3 xilinx-zcu102-2018_3 /dev/hvc0 xilinx-zcu102-2018_3 login:
Login using 'root' as the username and password.
SD Booting Xen and Dom0
To boot Xen from an SD card you need to copy the following files to the boot partition of the SD card:- BOOT.bin
- Image
- the compiled device tree file renamed to system.dtb (xen.dtb or xen-qemu.dtb for QEMU from the pre-built images, system.dtb from a Petalinux build)
- xen.ub
- rootfs.cpio.gz.u-boot (Only if using initrd instead of initramfs for the rootfs)
When using the pre-built images from the BSP, copy these files from <project-dir>/pre-built/linux/images. The prebuilt images are built to support both Linux without Xen and with Xen such that some of the Xen based image file names are different than in a normal Petalinux build. The prebuilt Linux kernel image includes an initramfs rootfs. Petalinux builds (rather than prebuilt images) require an initrd rootfs such that another file for the rootfs must also be used as described below.
Booting with ImageBulider
ImageBuilder's script uboot-script-gen can be used to generate a uboot script that loads all the binaries automatically from MMC. First, convert the xen and xen-rootfs binaries into raw binaries and write an appropriate config file, see TFTPing Xen using ImageBuilder above. Then, call uboot-script-gen with the following command, assuming that $sdbootdev is 0 and $partid is 1:$ bash ./scripts/uboot-script-gen -c config -d . -t "load mmc 0:1"
$ mmc dev $sdbootdev &&&& mmcinfo; load mmc $sdbootdev:$partid 0xC00000 boot.scr; source 0xC00000
Booting with RootFS in Kernel (initramfs) manually
This method allows the use of a Linux kernel with an initramfs (such as the prebuilt image). Boot the SD card on hardware or QEMU and stop the u-boot autoboot. At the u-boot prompt run:mmc dev $sdbootdev &&&& mmcinfo; load mmc $sdbootdev:$partid 1280000 system.dtb &&&& load mmc $sdbootdev:$partid 0x80000 Image; fdt addr 1280000; load mmc $sdbootdev:$partid 1400000 xen.ub; bootm 1400000 - 1280000
Booting RootFS mounted on RAM (initrd) manually
This method is required when using a Linux image which is initrd based and does not include a rootfs. The rootfs.cpio.gz.u-boot file will be loaded in memory from u-boot. Then boot the SD card on hardware or QEMU and stop the u-boot autoboot. At the u-boot prompt run:mmc dev $sdbootdev &&&& mmcinfo; load mmc $sdbootdev:$partid 1280000 system.dtb &&&& load mmc $sdbootdev:$partid 0x80000 Image; fdt addr 1280000; load mmc $sdbootdev:$partid 1400000 xen.ub; load mmc $sdbootdev:$partid 9000000 rootfs.cpio.gz.u-boot; bootm 1400000 9000000 1280000
Starting simple additional guests
If running on QEMU, we'll need to setup a port mapping for port 22 (SSH) in our VM.In this example, we forward the hosts port 2222 to the VM's port 22.
$ petalinux-boot --qemu --u-boot --qemu-args "-net nic -net nic -net nic -net nic -net user,tftp=images/linux,hostfwd=tcp:127.0.0.1:2222-10.0.2.15:22"
Once you hit the u-boot prompt, follow the steps in the earlier section on how to run Xen dom0.
When dom0 has finished booting, we'll need to copy a guest Image into dom0's filesystem.
We'll use the base prebuilt PetaLinux Image as our domU guest.
If running on QEMU, we use scp's -P option to connect to our hosts port 2222 where QEMU will forward the connection to the guests port 22:
To target QEMU run the following on the host:
scp -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -P 2222 pre-built/linux/images/Image root@localhost:/boot/
scp -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no pre-built/linux/images/Image root@<board-ip>:/boot/
The xen-image-minimal rootFS includes some prepared configurations that you can use. These are located in '/etc/xen/'
# cd /etc/xen
# xl create -c example-simple.cfg
At any time you can leave the console of the guest and get back to dom0 by pressing ctrl+].
Once at the dom0 prompt you can list the guests from dom0:
# xl list
# xl console guest0
# xl create example-simple.cfg name=\"guest1\" # xl create example-simple.cfg name=\"guest2\" root@plnx_aarch64:/etc/xen# xl list Name ID Mem VCPUs State Time(s) Domain-0 0 512 1 r----- 79.8 Domain-0 0 512 1 r----- 79.8 guest0 1 256 2 ------ 93.7 guest1 2 256 2 ------ 26.6 guest2 3 256 2 ------ 1.8
# xl destroy guest0
CPU Pinning
The following will only work on QEMU with multi-core enabled or on real HW.When running multiple guests with multiple Virtual CPUs, Xen will schedule the various vCPUs onto real physical CPUs.
The rules and considerations taken in scheduling decisions depend on the chosen scheduler and the configuration.
To avoid having multiple vCPUs share a single pCPU, it is possible to pin a vCPU onto a pCPU and to give it exclusive access.
To create a simple guest with one Virtual CPU pinned to Physical CPU #3, you can do the following:
xl create example-simple.cfg 'name="g0"' 'vcpus="1"' 'cpus="3"'
Another way to pin virtual CPUs on to Physical CPUs is to create dedicated cpu-pools.
This has the advantage of isolating the scheduling instances.
By default a single cpu-pool named Pool-0 exists. It contains all the physical cpus.
We'll now create our pool named rt using the credit2 scheduler.
xl cpupool-create 'name="rt"' 'sched="credit"' xl cpupool-cpu-remove Pool-0 3 xl cpupool-cpu-add rt 3
xl create /etc/xen/example-simple.cfg 'vcpus="1"' 'pool="rt"' 'cpus="3"' 'name="g0"'
Starting Linux guests with Para-Virtual networking
This time we will run QEMU slightly different. We'll create two port mappings. One for dom0's SSH port and another for the Para-Virtual domU.The default IP addresses assigned by QEMUs builtin DHCP server start from 10.0.2.15 and count upwards.
Dom0 will be assigned 10.0.2.15, the next guest 10.0.2.16 and so on.
So here's the command line that maps host port 2222 to dom0 port 22 and 2322 to domUs port 22.
$ petalinux-boot --qemu --u-boot --qemu-args "-net nic -net nic -net nic -net nic -net user,tftp=./images/linux/,hostfwd=tcp:127.0.0.1:2222-10.0.2.15:22,hostfwd=tcp:127.0.0.1:2322-10.0.2.16:22"
Now, follow the instructions from section 1 on how to boot Xen dom0.
Once you are at the dom0 prompt and have copied a domU image (see earlier steps) we'll need to setup the networking.
In this example, we will configure the guests to directly join the external network by means of a bridge.
First of all, we need to de-configure the default setup.
Kill the dhcp client for eth0:
# killall -9 udhcpc
# ip addr show dev eth0
# ip addr del 10.0.2.15/24 dev eth0
# brctl addbr xenbr0 # brctl addif xenbr0 eth0 # /sbin/udhcpc -i xenbr0 -b
udhcpc (v1.24.1) started [ 165.460858] xenbr0: port 1(eth0) entered blocking state [ 165.461819] xenbr0: port 1(eth0) entered forwarding state Sending discover... Sending select for 10.0.2.15... Lease of 10.0.2.15 obtained, lease time 86400 /etc/udhcpc.d/50default: Adding DNS 10.0.2.3
# cd /etc/xen
# xl create -c example-pvnet.cfg
Now we'll ssh into the domU from the host running Para-Virtual networking:
$ ssh -p 2322 root@localhost
Starting Linux guests with Pass-through networking
The difficulty with using pass through networking is that the steps above use Dom0 networking to load the DomU boot image onto the guest. This won't work with pass through networking as Dom0 never has any networking avaliable.You will need to find a way to get the kernel and rootFS (the pre-built Image file) onto the guest. The steps below are used to get the Image file onto a SD card image and attach it to QEMU. Similar steps can be followed for hardware, excpet just copy the Image file to a formated SD card and insert it into the board.
Create and format the file we will be using on your host:
$ dd if=/dev/zero of=qemu_sd.img bs=128M count=1 $ mkfs.vfat -F 32 qemu_sd.img
NOTE: We are using the pre-built Image which contains a kernel and rootFS. If you use the Image you built above then no rootFS is included. You will need to copy the rootFS onto the SD card and edit the Xen config file later to specify a rootFS.
$ mcopy -i qemu_sd.img ./pre-built/linux/images/Image ::/
Now boot QEMU with this extra option appened inside the --qemu-args: "-drive file=qemu_sd.img,if=sd,format=raw,index=1"
The full command should look something like this for your prebuilt images:
petalinux-boot --qemu --prebuilt 2 --qemu-args "-net nic -net nic -net nic -net nic -net user,tftp=./pre-built/linux/images/,hostfwd=tcp:127.0.0.1:2222-10.0.2.15:22,hostfwd=tcp:127.0.0.1:2322-10.0.2.16:22 -drive file=qemu_sd.img,if=sd,format=raw,index=1"
petalinux-boot --qemu --u-boot --qemu-args "-net nic -net nic -net nic -net nic -net user,tftp=./images/linux/,hostfwd=tcp:127.0.0.1:2222-10.0.2.15:22,hostfwd=tcp:127.0.0.1:2322-10.0.2.16:22 -drive file=qemu_sd.img,if=sd,format=raw,index=1"
After loading the device tree to memory you will need to run this: fdt addr $fdt_addr && fdt resize 128; fdt set /amba/ethernet@ff0e0000 status "disabled" && fdt set /amba/ethernet@ff0e0000 xen,passthrough "1"
The full command for booting prebuilt images you built is shown below:
u-boot> tftpb 1280000 xen-qemu.dtb; fdt addr 1280000 &&&& fdt resize 128; fdt set /amba/ethernet@ff0e0000 status "disabled" &&&& fdt set /amba/ethernet@ff0e0000 xen,passthrough "1" &&&& tftpb 0x80000 xen-Image; tftpb 1400000 xen.ub; tftpb 9000000 xen-rootfs.cpio.gz.u-boot; bootm 1400000 9000000 1280000
u-boot> tftpb 1280000 system.dtb; fdt addr 1280000 &&&& fdt resize 128; fdt set /amba/ethernet@ff0e0000 status "disabled" &&&& fdt set /amba/ethernet@ff0e0000 xen,passthrough "1" &&&& tftpb 0x80000 Image; tftpb 1400000 xen.ub; tftpb 9000000 rootfs.cpio.gz.u-boot; bootm 1400000 9000000 1280000
Once you have logged onto the system mount the SD card and copy the image.
mount /dev/mmcblk0 /mnt/ cp /mnt/Image /boot/
cd /etc/xen
xl create -c example-passnet.cfg
© Copyright 2019 - 2022 Xilinx Inc. Privacy Policy