Building the Xen Hypervisor with PetaLinux 2016.4 and newer
Table of Contents
Overview
The guide below shows you how to build Xen, boot Xen and then run some example configurations on ZU+. The steps below use PetaLinux and assume you have some knowledge of using PetaLinux.Before starting you need to create a PetaLinux project. It is assumed that a default PetaLinux reference design is used unchanged in these instructions.
The default PetaLinux configuration has images ready to do boot Xen, these are the pre-built images. You can use those or you can manually edit receipes and build Xen yourself. The pre-built images can be found in this directory (inside a PetaLinux project) pre-built/linux/images/ and prefixed with "xen-". You can either use the pre-builts or follow the next section to configure and build Xen yourself. If you are using the pre-builts you can skip to the booting Xen section for your release version.
Remove or comment out '#' the following line
The second file that needs to be edited is:
Add the extra line at the end:
NOTE: You might need to have already run the build command below to be able to edit this file.
NOTE: As of PetaLinux 2016.4 the petalinux-build command no longer builds Xen by default, so the above command is required
You have to manually copy the xen-qemu.dtb from the pre-built directory:
You will also need to manually copy the recently built Xen images
NOTE: If you re-build anything else after Xen the following images will be over-written
To use the Xen you built yourself run:
To use the Xen you built yourself run:
You should eventually see something similar to this, when you do press any key to stop the autoboot.
If u-boot wasn't able to get an IP address from the DHCP server you may need to manually set the serverip (it's typically 10.0.2.2 for QEMU):
Now to download and boot Xen, if running on QEMU, use xen-qemu.dtb otherwise use xen.dtb. Example:
Below is an example of what you will see.
Login using 'root' as the username
In this example, we forward the hosts port 2222 to the VM's port 22.
Once you hit the u-boot prompt, follow the steps in the earlier section on how to run Xen dom0.
When dom0 has finished booting, we'll need to copy a guest Image into dom0's filesystem.
We'll use the base prebuilt PetaLinux Image as our domU guest.
If running on QEMU, we use scp's -P option to connect to our hosts port 2222 where QEMU will forward the connection to the guests port 22:
To target QEMU run the following on the host:
If running on hardware run the following on the host:
If you would prefer to load DomU's kernel to the guest via SD card, you can follow the instructions in the "Starting Linux guests with Pass-through networking" section.
The xen-image-minimal rootFS includes some prepared configurations that you can use. These are located in '/etc/xen/'
To start a simple guest run the following from the dom0 prompt
You'll see another instance of Linux booting up.
At any time you can leave the console of the guest and get back to dom0 by pressing ctrl+].
Once at the dom0 prompt you can list the guests from dom0:
To get back to the guests console:
You can create further guests by for example running:
To destroy a guest:
xl destroy guest0
The following will only work on QEMU with multi-core enabled or on real HW.
When running multiple guests with multiple Virtual CPUs, Xen will schedule the various vCPUs onto real physical CPUs.
The rules and considerations taken in scheduling decisions depend on the chosen scheduler and the configuration.
To avoid having multiple vCPUs share a single pCPU, it is possible to pin a vCPU onto a pCPU and to give it exclusive access.
To create a simple guest with one Virtual CPU pinned to Physical CPU #3, you can do the following:
Another way to pin virtual CPUs on to Physical CPUs is to create dedicated cpu-pools.
This has the advantage of isolating the scheduling instances.
By default a single cpu-pool named Pool-0 exists. It contains all the physical cpus.
We'll now create our pool named rt using the credit2 scheduler.
Now we are ready to create a guest with a single vcpu pinned to physical CPU #3.
xl create /etc/xen/example-simple.cfg 'vcpus="1"' 'pool="rt"' 'cpus="3"' 'name="g0"'
The default IP addresses assigned by QEMUs builtin DHCP server start from 10.0.2.15 and count upwards.
Dom0 will be assigned 10.0.2.15, the next guest 10.0.2.16 and so on.
So here's the command line that maps host port 2222 to dom0 port 22 and 2322 to domUs port 22.
Now, follow the instructions from section 1 on how to boot Xen dom0.
Once you are at the dom0 prompt and have copied a domU image we'll need to setup the networking.
In this example, we will configure the guests to directly join the external network by means of a bridge.
First of all, we need to de-configure the default setup.
Kill the dhcp client for eth0:
List and remove existing addresses from eth0:
In our example the address is 10.0.2.15/24:
Then, create the bridge and start DHCP on it for dom0:
You should see something like the following:
Similar to before we will use the pre-defined examples in '/etc/xen/'
You should see a new linux instance boot up.
Now we'll ssh into the domU from the host running Para-Virtual networking:
You will need to find a way to get the kernel and rootFS (the pre-built Image file) onto the guest. The steps below are used to get the Image file onto a SD card image and attach it to QEMU. Similar steps can be followed for hardware, excpet just copy the Image file to a formated SD card and insert it into the board.
Create and format the file we will be using on your host:
Copy the Image file onto the card.
NOTE: We are using the pre-built Image which contains a kernel and rootFS. If you use the Image you built above then no rootFS is included. You will need to copy the rootFS onto the SD card and edit the Xen config file later to specify a rootFS.
Now boot QEMU with this extra option appened inside the --qemu-args: -drive file=qemu_sd.img,if=sd,format=raw,index=1
The full command should look something like this for your prebuilt images:
The full command should look something like this for your own images:
Then boot Dom0 following the steps above, with one difference. You will need to make sure that you tell Xen about the network passthrough. To do this you will need to edit the device tree. We are going to use u-boot to edit the device tree.
After loading the device tree to memory you will need to run this: fdt addr $fdt_addr && fdt resize 128; fdt set /amba/ethernet@ff0e0000 status "disabled" && fdt set /amba/ethernet@ff0e0000 xen,passthrough "1"
The full command for booting prebuilt images you built is shown below:
The full command for booting images you built is shown below:
NOTE: If running on hardware you will need to make a change to allow the DMA transactions. See here for more details: Passthrough Network Example
Once you have logged onto the system mount the SD card and copy the image.
Similar to before we will use another pre-defined examples in '/etc/xen/'
The default PetaLinux configuration has images ready to do boot Xen, these are the pre-built images. You can use those or you can manually edit receipes and build Xen yourself. The pre-built images can be found in this directory (inside a PetaLinux project) pre-built/linux/images/ and prefixed with "xen-". You can either use the pre-builts or follow the next section to configure and build Xen yourself. If you are using the pre-builts you can skip to the booting Xen section for your release version.
Configuring and building XEN from source using PetaLinux 2016.4
To build the XEN tools and XEN hypervisor you will need to edit the following two files:$ vim project-spec/meta-plnx-generated/recipes-core/images/petalinux-user-image.bb
EXTRA_USERS_PARAMS = "usermod -P root root;"*
$ vim build/conf/local.conf
EXTRA_IMAGE_FEATURES = "debug-tweaks"
$ petalinux-build -c xen-image-minimal
You have to manually copy the xen-qemu.dtb from the pre-built directory:
$ cp ./pre-built/linux/images/xen-qemu.dtb ./images/linux/
You will also need to manually copy the recently built Xen images
$ cp build/tmp/deploy/images/plnx_aarch64/xen.ub ./images/linux/ $ cp build/tmp/deploy/images/plnx_aarch64/Image ./images/linux/xen-Image $ cp build/tmp/deploy/images/plnx_aarch64/xen-image-minimal-plnx_aarch64.cpio.gz.u-boot ./images/linux/xen-rootfs.cpio.gz.u-boot
NOTE: If you re-build anything else after Xen the following images will be over-written
TFTP Booting Xen and Dom0 2016.4
Run Xen dom0 on QEMU:
To use the prebuilt Xen run:$ petalinux-boot --qemu --prebuilt 2 --qemu-args "-net nic -net nic -net nic -net nic -net user,tftp=pre-built/linux/images"
$ petalinux-boot --qemu --u-boot
Run Xen dom0 on HW:
To use the prebuilt Xen on hardware:$ petalinux-boot --jtag --prebuilt 2
$ petalinux-boot --jtag --u-boot
You should eventually see something similar to this, when you do press any key to stop the autoboot.
Hit any key to stop autoboot:
If u-boot wasn't able to get an IP address from the DHCP server you may need to manually set the serverip (it's typically 10.0.2.2 for QEMU):
$ setenv serverip 10.0.2.2
Now to download and boot Xen, if running on QEMU, use xen-qemu.dtb otherwise use xen.dtb. Example:
$ tftpb 4000000 xen-qemu.dtb $ tftpb 0x80000 xen-Image $ tftpb 6000000 xen.ub $ tftpb 0x1000000 xen-rootfs.cpio.gz.u-boot $ bootm 6000000 0x1000000 4000000
ZynqMP> tftpb 4000000 xen-qemu.dtb Using ethernet@ff0e0000 device TFTP from server 10.0.2.2; our IP address is 10.0.2.15 Filename 'xen-qemu.dtb'. Load address: 0x4000000 Loading: ###### 4 MiB/s done Bytes transferred = 29417 (72e9 hex) ZynqMP> tftpb 0x80000 xen-Image Using ethernet@ff0e0000 device TFTP from server 10.0.2.2; our IP address is 10.0.2.15 Filename 'xen-Image'. Load address: 0x80000 Loading: ################################################################# ################################################################# ################################################################# [...] ################################################################# ############################################################## 5.4 MiB/s done Bytes transferred = 12297216 (bba400 hex) ZynqMP> tftpb 6000000 xen.ub Using ethernet@ff0e0000 device TFTP from server 10.0.2.2; our IP address is 10.0.2.15 Filename 'xen.ub'. Load address: 0x6000000 Loading: ################################################################# ################################################################# ############ 4.2 MiB/s done Bytes transferred = 722528 (b0660 hex) ZynqMP> tftpb 0x1000000 xen-rootfs.cpio.gz.u-boot Using ethernet@ff0e0000 device TFTP from server 10.0.2.2; our IP address is 10.0.2.15 Filename 'xen-rootfs.cpio.gz.u-boot'. Load address: 0x1000000 Loading: ################################################################# ################################################################# ################################################################# ################################################################# ################################################################# [...] ################################################################# ################################################################# ##### 5.2 MiB/s done Bytes transferred = 36571940 (22e0b24 hex) ZynqMP> bootm 6000000 0x1000000 4000000 ## Booting kernel from Legacy Image at 06000000 ... Image Name: Image Type: AArch64 Linux Kernel Image (uncompressed) Data Size: 722464 Bytes = 705.5 KiB Load Address: 05000000 Entry Point: 05000000 Verifying Checksum ... OK ## Loading init Ramdisk from Legacy Image at 01000000 ... Image Name: xen-image-minimal-plnx_aarch64-2 Image Type: AArch64 Linux RAMDisk Image (gzip compressed) Data Size: 36571876 Bytes = 34.9 MiB Load Address: 00000000 Entry Point: 00000000 Verifying Checksum ... OK ## Flattened Device Tree blob at 04000000 Booting using the fdt blob at 0x4000000 Loading Kernel Image ... OK Loading Ramdisk to 05d1f000, end 07fffae4 ... OK Loading Device Tree to 0000000005d14000, end 0000000005d1e2e8 ... OK Starting kernel ... [...] Starting OpenBSD Secure Shell server: sshd generating ssh RSA key... generating ssh ECDSA key... generating ssh DSA key... generating ssh ED25519 key... done. Starting syslogd/klogd: done Starting /usr/sbin/xenstored... Setting domain 0 name, domid and JSON config... Done setting up Dom0 Starting xenconsoled... Starting QEMU as disk backend for dom0 Starting domain watchdog daemon: xenwatchdogd startup [done] PetaLinux 2016.4 plnx_aarch64 /dev/hvc0 plnx_aarch64 login:
Login using 'root' as the username
Starting simple additional guests (PetaLinux 2016.4 or later)
If running on QEMU, we'll need to setup a port mapping for port 22 (SSH) in our VM.In this example, we forward the hosts port 2222 to the VM's port 22.
$ petalinux-boot --qemu --u-boot --qemu-args "-net nic -net nic -net nic -net nic -net user,tftp=images/linux,hostfwd=tcp:127.0.0.1:2222-10.0.2.15:22"
Once you hit the u-boot prompt, follow the steps in the earlier section on how to run Xen dom0.
When dom0 has finished booting, we'll need to copy a guest Image into dom0's filesystem.
We'll use the base prebuilt PetaLinux Image as our domU guest.
If running on QEMU, we use scp's -P option to connect to our hosts port 2222 where QEMU will forward the connection to the guests port 22:
To target QEMU run the following on the host:
scp -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -P 2222 images/linux/Image root@localhost:/boot/
scp -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no images/linux/Image root@<board-ip>:/boot/
The xen-image-minimal rootFS includes some prepared configurations that you can use. These are located in '/etc/xen/'
$ cd /etc/xen
To start a simple guest run the following from the dom0 prompt
xl create -c example-simple.cfg
You'll see another instance of Linux booting up.
At any time you can leave the console of the guest and get back to dom0 by pressing ctrl+].
Once at the dom0 prompt you can list the guests from dom0:
xl list
To get back to the guests console:
xl console guest0
You can create further guests by for example running:
xl create example-simple.cfg name=\"guest1\" xl create example-simple.cfg name=\"guest2\" root@plnx_aarch64:/etc/xen# xl list Name ID Mem VCPUs State Time(s) Domain-0 0 512 1 r----- 79.8 Domain-0 0 512 1 r----- 79.8 guest0 1 256 2 ------ 93.7 guest1 2 256 2 ------ 26.6 guest2 3 256 2 ------ 1.8
To destroy a guest:
xl destroy guest0
CPU Pinning
The following will only work on QEMU with multi-core enabled or on real HW.
When running multiple guests with multiple Virtual CPUs, Xen will schedule the various vCPUs onto real physical CPUs.
The rules and considerations taken in scheduling decisions depend on the chosen scheduler and the configuration.
To avoid having multiple vCPUs share a single pCPU, it is possible to pin a vCPU onto a pCPU and to give it exclusive access.
To create a simple guest with one Virtual CPU pinned to Physical CPU #3, you can do the following:
xl create example-simple.cfg 'name="g0"' 'vcpus="1"' 'cpus="3"'
Another way to pin virtual CPUs on to Physical CPUs is to create dedicated cpu-pools.
This has the advantage of isolating the scheduling instances.
By default a single cpu-pool named Pool-0 exists. It contains all the physical cpus.
We'll now create our pool named rt using the credit2 scheduler.
xl cpupool-create 'name="rt"' 'sched="credit"' xl cpupool-cpu-remove Pool-0 3 xl cpupool-cpu-add rt 3
Now we are ready to create a guest with a single vcpu pinned to physical CPU #3.
xl create /etc/xen/example-simple.cfg 'vcpus="1"' 'pool="rt"' 'cpus="3"' 'name="g0"'
Starting Linux guests with Para-Virtual networking (PetaLinux 2016.4 or later)
This time we will run QEMU slightly different. We'll create two port mappings. One for dom0's SSH port and another for the Para-Virtual domU.The default IP addresses assigned by QEMUs builtin DHCP server start from 10.0.2.15 and count upwards.
Dom0 will be assigned 10.0.2.15, the next guest 10.0.2.16 and so on.
So here's the command line that maps host port 2222 to dom0 port 22 and 2322 to domUs port 22.
petalinux-boot --qemu --u-boot --qemu-args "-net nic -net nic -net nic -net nic -net user,tftp=./images/linux/,hostfwd=tcp:127.0.0.1:2222-10.0.2.15:22,hostfwd=tcp:127.0.0.1:2322-10.0.2.16:22"
Now, follow the instructions from section 1 on how to boot Xen dom0.
Once you are at the dom0 prompt and have copied a domU image we'll need to setup the networking.
In this example, we will configure the guests to directly join the external network by means of a bridge.
First of all, we need to de-configure the default setup.
Kill the dhcp client for eth0:
# killall -9 udhcpc
List and remove existing addresses from eth0:
# ip addr show dev eth0
# ip addr del 10.0.2.15/24 dev eth0
Then, create the bridge and start DHCP on it for dom0:
# brctl addbr xenbr0 # brctl addif xenbr0 eth0 # /sbin/udhcpc -i xenbr0 -b
You should see something like the following:
udhcpc (v1.24.1) started [ 186.459495] xenbr0: port 1(eth0) entered blocking state [ 186.461194] xenbr0: port 1(eth0) entered forwarding state Sending discover... Sending select for 10.0.2.15... Lease of 10.0.2.15 obtained, lease time 86400 /etc/udhcpc.d/50default: Adding DNS 10.0.2.3
Similar to before we will use the pre-defined examples in '/etc/xen/'
$ cd /etc/xen
# xl create -c example-pvnet.cfg
You should see a new linux instance boot up.
Now we'll ssh into the domU from the host running Para-Virtual networking:
ssh -p 2322 root@localhost
Starting Linux guests with Pass-through networking (PetaLinux 2017.1)
The difficulty with using pass through networking is that the steps above use Dom0 networking to load the DomU boot image onto the guest. This won't work with pass through networking as Dom0 never has any networking avaliable.You will need to find a way to get the kernel and rootFS (the pre-built Image file) onto the guest. The steps below are used to get the Image file onto a SD card image and attach it to QEMU. Similar steps can be followed for hardware, excpet just copy the Image file to a formated SD card and insert it into the board.
Create and format the file we will be using on your host:
$ dd if=/dev/zero of=qemu_sd.img bs=128M count=1 $ mkfs.vfat -F 32 qemu_sd.img
NOTE: We are using the pre-built Image which contains a kernel and rootFS. If you use the Image you built above then no rootFS is included. You will need to copy the rootFS onto the SD card and edit the Xen config file later to specify a rootFS.
$ mcopy -i qemu_sd.img ./pre-built/linux/images/Image ::/
The full command should look something like this for your prebuilt images:
petalinux-boot --qemu --u-boot --qemu-args "-net nic -net nic -net nic -net nic -net user,tftp=./pre-built/linux/images/,hostfwd=tcp:127.0.0.1:2222-10.0.2.15:22,hostfwd=tcp:127.0.0.1:2322-10.0.2.16:22 -drive file=qemu_sd.img,if=sd,format=raw,index=1"
petalinux-boot --qemu --u-boot --qemu-args "-net nic -net nic -nic -net nic -net user,tftp=./images/linux/,hostfwd=tcp:127.0.0.1:2222-10.0.2.15:22,hostfwd=tcp:127.0.0.1:2322-10.0.2.16:22 -drive file=qemu_sd.img,if=sd,format=raw,index=1"
After loading the device tree to memory you will need to run this: fdt addr $fdt_addr && fdt resize 128; fdt set /amba/ethernet@ff0e0000 status "disabled" && fdt set /amba/ethernet@ff0e0000 xen,passthrough "1"
The full command for booting prebuilt images you built is shown below:
$ tftpb 4000000 xen-qemu.dtb; fdt addr 4000000 &&&& fdt resize 128; fdt set /amba/ethernet@ff0e0000 status "disabled" &&&& fdt set /amba/ethernet@ff0e0000 xen,passthrough "1" &&&& tftpb 0x80000 xen-Image; tftpb 6000000 xen.ub; tftpb 0x1000000 xen-rootfs.cpio.gz.u-boot; bootm 6000000 0x1000000 4000000
$ tftpb 4000000 system.dtb; fdt addr 4000000 &&&& fdt resize 128; fdt set /amba/ethernet@ff0e0000 status "disabled" &&&& fdt set /amba/ethernet@ff0e0000 xen,passthrough "1" &&&& tftpb 0x80000 Image; tftpb 6000000 xen.ub; tftpb 0x1000000 rootfs.cpio.gz.u-boot; bootm 6000000 0x1000000 4000000
Once you have logged onto the system mount the SD card and copy the image.
# mount /dev/mmcblk0 /mnt/ # cp /mnt/Image /boot/
$ cd /etc/xen
# xl create -c example-passnet.cfg
© Copyright 2019 - 2022 Xilinx Inc. Privacy Policy