...
Before starting you need to create a PetaLinux project. It is assumed that a default PetaLinux reference design is used unchanged in these instructions.
The default PetaLinux configuration has images ready to do boot Xen, these are the pre-built images. You can use those or you can manually edit receipes and build Xen yourself. The pre-built images can be found in this directory (inside a PetaLinux project) pre-built/linux/images/ and prefixed with "xen-". You can either use the pre-builts or follow the next section to configure and build Xen yourself. If you are using the pre-builts you can skip to the booting Xen section for your release version.
Configuring and building XEN from source using PetaLinux
First let's enable Xen to be built by default.
| Code Block |
|---|
|
$ petalinux-config -c rootfs |
Now let's enable Xen:
| Code Block |
|---|
|
Petalinux Package Groups ---> packagegroup-petalinux-xen ---> [*] packagegroup-petalinux-xen |
Now we need to change the rootFS to be an INITRD
| Code Block |
|---|
|
$ petalinux-config |
And change
| Code Block |
|---|
|
Image Packaging Configuration ---> Root filesystem type (INITRAMFS) ---> (X) INITRD |
NOTE: This means that any images built will NOT have the rootFS in the Image that is built by PetaLinux. This means you will need to edit any scripts or configuration that expects the rootFS to be included. This includes the Xen configs mentioned later.You can still use the prebuilt Image file which does still include the rootFS to boot DomU.
We also want to edit the device tree to build in the extra Xen related configs.
Edit this file
| Code Block |
|---|
|
project-spec/meta-user/recipes-bsp/device-tree/files/system-user.dtsi |
and add this line: /include/ "xen.dtsi".
It should look like this for harware:
| Code Block |
|---|
|
/include/ "system-conf.dtsi"
/include/ "xen.dtsi"
/ { }; |
or like this for QEMU:
| Code Block |
|---|
|
/include/ "system-conf.dtsi"
/include/ "xen.dtsi"
/ {
cpus {
cpu@1 {
device_type = "none";
};
cpu@2 {
device_type = "none";
};
cpu@3 {
device_type = "none";
};
};
}; |
NOTE: There is a bug on QEMU where the CPUs running in SMP sometimes cause hangs. To avoid this we only tell Xen about a single CPU.
Also edit this file:
| Code Block |
|---|
|
project-spec/meta-user/recipes-bsp/device-tree/device-tree.bbappend |
and add this line to it: file://xen.dtsi.
The file should look like this:
| Code Block |
|---|
|
FILESEXTRAPATHS_prepend := "${THISDIR}/files:"
SRC_URI += "file://system-user.dtsi"
SRC_URI += "file://xen.dtsi"
|
Then run petaliux-build:
| Code Block |
|---|
|
$ petalinux-build |
TFTP Booting Xen and Dom0
Run Xen dom0 on QEMU:
To use the prebuilt Xen run:
| Code Block |
|---|
|
$ petalinux-boot --qemu --prebuilt 2 --qemu-args "-net nic -net nic -net nic -net nic -net user,tftp=pre-built/linux/images" |
To use the Xen you built yourself run:
| Code Block |
|---|
|
$ petalinux-boot --qemu --u-boot |
Run Xen dom0 on HW:
To use the prebuilt Xen on hardware:
| Code Block |
|---|
|
$ petalinux-boot --jtag --prebuilt 2 |
To use the Xen you built yourself run:
| Code Block |
|---|
|
$ petalinux-boot --jtag --u-boot |
You should eventually see something similar to this, when you do press any key to stop the autoboot.
| Code Block |
|---|
|
Hit any key to stop autoboot: |
If u-boot wasn't able to get an IP address from the DHCP server you may need to manually set the serverip (it's typically 10.0.2.2 for QEMU):
| Code Block |
|---|
|
$ setenv serverip 10.0.2.2 |
Now to download and boot Xen, if running on QEMU, use xen-qemu.dtb otherwise use xen.dtb. Example:
TFTPing Xen from pre-built images
| Code Block |
|---|
|
$ tftpb 1280000 xen-qemu.dtb; tftpb 0x80000 xen-Image; tftpb 1400000 xen.ub; tftpb 9000000 xen-rootfs.cpio.gz.u-boot; bootm 1400000 9000000 1280000 |
Booting with larger binaries and Workaround for ZCU104
Any time you update binaries and they potentially increase in size, please make sure that they don't overlap. For instance, if the kernel Image is loaded at 0x80000 and the dtb at 0x1280000, it means that the maximum size of the kernel is 12MB. More than that and the loading addresses need to be updated to avoid overlaps. Moreover, both address and size of the kernel Image are also specified under the /chosen node in the dtb file.
The ZCU104 BSP comes with a larger Dom0 rootfs (xen-rootfs.cpio.gz.u-boot file). U-boot has issues booting the rootfs as a u-boot image, it needs to be loaded as a raw binary instead. Thus, the following workaround is required for ZCU104.
Convert the xen-rootfs back to a normal binary:
| Code Block |
|---|
|
$ dd if=xen-rootfs.cpio.gz.u-boot of=xen-rootfs.cpio.gz. bs=64 skip=1 |
Add the following node under /chosen in device tree xen-qemu.dts (you can convert a device tree binary to source and back using dtc):
...
RootFS in Kernel (initramfs)
This method allows the use of a Linux kernel with an initramfs (such as the prebuilt image). Boot the SD card on hardware or QEMU and stop the u-boot autoboot. At the u-boot prompt run:
| Code Block |
|---|
|
mmc dev $sdbootdev &&&& mmcinfo; load mmc $sdbootdev:$partid 1280000 system.dtb &&&& load mmc $sdbootdev:$partid 0x80000 Image; fdt addr 1280000; load mmc $sdbootdev:$partid 1400000 xen.ub; bootm 1400000 - 1280000 |
This would also allow a rootfs mounted on the SD card to be used. It requires that you extract the rootFS (cpio file) to the root partition on the SD card and setup the kernel to expect the rootfs on the SD card in the kernel command line.
RootFS mounted on RAM (initrd)
This method is required when using a Linux image which is initrd based and does not include a rootfs. The rootfs.cpio.gz.u-boot file will be loaded in memory from u-boot. Then boot the SD card on hardware or QEMU and stop the u-boot autoboot. At the u-boot prompt run:
| Code Block |
|---|
|
mmc dev $sdbootdev &&&& mmcinfo; load mmc $sdbootdev:$partid 1280000 system.dtb &&&& load mmc $sdbootdev:$partid 0x80000 Image; fdt addr 1280000; load mmc $sdbootdev:$partid 1400000 xen.ub; load mmc $sdbootdev:$partid 9000000 rootfs.cpio.gz.u-boot; bootm 1400000 9000000 1280000 |
Starting simple additional guests
If running on QEMU, we'll need to setup a port mapping for port 22 (SSH) in our VM.
In this example, we forward the hosts port 2222 to the VM's port 22.
| Code Block |
|---|
|
$ petalinux-boot --qemu --u-boot --qemu-args "-net nic -net nic -net nic -net nic -net user,tftp=images/linux,hostfwd=tcp:127.0.0.1:2222-10.0.2.15:22" |
Once you hit the u-boot prompt, follow the steps in the earlier section on how to run Xen dom0.
When dom0 has finished booting, we'll need to copy a guest Image into dom0's filesystem.
We'll use the base prebuilt PetaLinux Image as our domU guest.
If running on QEMU, we use scp's -P option to connect to our hosts port 2222 where QEMU will forward the connection to the guests port 22:
To target QEMU run the following on the host:
| Code Block |
|---|
|
scp -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -P 2222 pre-built/linux/images/Image root@localhost:/boot/ |
If running on hardware run the following on the host:
| Code Block |
|---|
|
scp -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no pre-built/linux/images/Image root@<board-ip>:/boot/ |
If you would prefer to load DomU's kernel to the guest via SD card, you can follow the instructions in the "Starting Linux guests with Pass-through networking" section.
The xen-image-minimal rootFS includes some prepared configurations that you can use. These are located in '/etc/xen/'
To start a simple guest run the following from the dom0 prompt
| Code Block |
|---|
|
# xl create -c example-simple.cfg |
You'll see another instance of Linux booting up.
At any time you can leave the console of the guest and get back to dom0 by pressing
ctrl+].
Once at the dom0 prompt you can list the guests from dom0:
To get back to the guests console:
| Code Block |
|---|
|
# xl console guest0 |
You can create further guests by for example running:
| Code Block |
|---|
|
# xl create example-simple.cfg name=\"guest1\"
# xl create example-simple.cfg name=\"guest2\"
root@plnx_aarch64:/etc/xen# xl list
Name ID Mem VCPUs State Time(s)
Domain-0 0 512 1 r----- 79.8
Domain-0 0 512 1 r----- 79.8
guest0 1 256 2 ------ 93.7
guest1 2 256 2 ------ 26.6
guest2 3 256 2 ------ 1.8 |
To destroy a guest:
| Code Block |
|---|
|
# xl destroy guest0 |
CPU Pinning
The following will only work on QEMU with multi-core enabled or on real HW.
When running multiple guests with multiple Virtual CPUs, Xen will schedule the various vCPUs onto real physical CPUs.
The rules and considerations taken in scheduling decisions depend on the chosen scheduler and the configuration.
To avoid having multiple vCPUs share a single pCPU, it is possible to pin a vCPU onto a pCPU and to give it exclusive access.
To create a simple guest with one Virtual CPU pinned to Physical CPU #3, you can do the following:
| Code Block |
|---|
|
xl create example-simple.cfg 'name="g0"' 'vcpus="1"' 'cpus="3"' |
Another way to pin virtual CPUs on to Physical CPUs is to create dedicated cpu-pools.
This has the advantage of isolating the scheduling instances.
By default a single cpu-pool named Pool-0 exists. It contains all the physical cpus.
We'll now create our pool named rt using the credit2 scheduler.
| Code Block |
|---|
|
xl cpupool-create 'name="rt"' 'sched="credit"'
xl cpupool-cpu-remove Pool-0 3
xl cpupool-cpu-add rt 3 |
Now we are ready to create a guest with a single vcpu pinned to physical CPU #3.
| Code Block |
|---|
|
xl create /etc/xen/example-simple.cfg 'vcpus="1"' 'pool="rt"' 'cpus="3"' 'name="g0"' |
Starting Linux guests with Para-Virtual networking
This time we will run QEMU slightly different. We'll create two port mappings. One for dom0's SSH port and another for the Para-Virtual domU.
The default IP addresses assigned by QEMUs builtin DHCP server start from 10.0.2.15 and count upwards.
Dom0 will be assigned 10.0.2.15, the next guest 10.0.2.16 and so on.
So here's the command line that maps host port 2222 to dom0 port 22 and 2322 to domUs port 22.
| Code Block |
|---|
|
$ petalinux-boot --qemu --u-boot --qemu-args "-net nic -net nic -net nic -net nic -net user,tftp=./images/linux/,hostfwd=tcp:127.0.0.1:2222-10.0.2.15:22,hostfwd=tcp:127.0.0.1:2322-10.0.2.16:22" |
Now, follow the instructions from section 1 on how to boot Xen dom0.
Once you are at the dom0 prompt and have copied a domU image (see earlier steps) we'll need to setup the networking.
In this example, we will configure the guests to directly join the external network by means of a bridge.
First of all, we need to de-configure the default setup.
Kill the dhcp client for eth0:
| Code Block |
|---|
|
# killall -9 udhcpc |
List and remove existing addresses from eth0:
| Code Block |
|---|
|
# ip addr show dev eth0 |
In our example the address is 10.0.2.15/24:
| Code Block |
|---|
|
# ip addr del 10.0.2.15/24 dev eth0 |
Then, create the bridge and start DHCP on it for dom0:
| Code Block |
|---|
|
# brctl addbr xenbr0
# brctl addif xenbr0 eth0
# /sbin/udhcpc -i xenbr0 -b |
You should see something like the following:
| Code Block |
|---|
|
udhcpc (v1.24.1) started
[ 165.460858] xenbr0: port 1(eth0) entered blocking state
[ 165.461819] xenbr0: port 1(eth0) entered forwarding state
Sending discover...
Sending select for 10.0.2.15...
Lease of 10.0.2.15 obtained, lease time 86400
/etc/udhcpc.d/50default: Adding DNS 10.0.2.3 |
Similar to before we will use the pre-defined examples in '/etc/xen/'
The start DomU
| Code Block |
|---|
|
# xl create -c example-pvnet.cfg |
You should see a new linux instance boot up.
Now we'll ssh into the domU from the host running Para-Virtual networking:
| Code Block |
|---|
|
$ ssh -p 2322 root@localhost |
Starting Linux guests with Pass-through networking
The difficulty with using pass through networking is that the steps above use Dom0 networking to load the DomU boot image onto the guest. This won't work with pass through networking as Dom0 never has any networking avaliable.
You will need to find a way to get the kernel and rootFS (the pre-built Image file) onto the guest. The steps below are used to get the Image file onto a SD card image and attach it to QEMU. Similar steps can be followed for hardware, excpet just copy the Image file to a formated SD card and insert it into the board.
Create and format the file we will be using on your host:
| Code Block |
|---|
|
$ dd if=/dev/zero of=qemu_sd.img bs=128M count=1 $ mkfs.vfat -F 32 qemu_sd.img |
Copy the Image file onto the card.
NOTE: We are using the pre-built Image which contains a kernel and rootFS. If you use the Image you built above then no rootFS is included. You will need to copy the rootFS onto the SD card and edit the Xen config file later to specify a rootFS.
| Code Block |
|---|
|
$ mcopy -i qemu_sd.img ./pre-built/linux/images/Image ::/ |
Now boot QEMU with this extra option appened inside the --qemu-args: "-drive file=qemu_sd.img,if=sd,format=raw,index=1"
The full command should look something like this for your prebuilt images:
| Code Block |
|---|
|
petalinux-boot --qemu --prebuilt 2 --qemu-args "-net nic -net nic -net nic -net nic -net user,tftp=./pre-built/linux/images/,hostfwd=tcp:127.0.0.1:2222-10.0.2.15:22,hostfwd=tcp:127.0.0.1:2322-10.0.2.16:22 -drive file=qemu_sd.img,if=sd,format=raw,index=1" |
The full command should look something like this for your own images:
| Code Block |
|---|
|
petalinux-boot --qemu --u-boot --qemu-args "-net nic -net nic -net nic -net nic -net user,tftp=./images/linux/,hostfwd=tcp:127.0.0.1:2222-10.0.2.15:22,hostfwd=tcp:127.0.0.1:2322-10.0.2.16:22 -drive file=qemu_sd.img,if=sd,format=raw,index=1" |
Then boot Dom0 following the steps above, with one difference. You will need to make sure that you tell Xen about the network passthrough. To do this you will need to edit the device tree. We are going to use u-boot to edit the device tree.
After loading the device tree to memory you will need to run this: fdt addr $fdt_addr && fdt resize 128; fdt set /amba/ethernet@ff0e0000 status "disabled" && fdt set /amba/ethernet@ff0e0000 xen,passthrough "1"
The full command for booting prebuilt images you built is shown below:
| Code Block |
|---|
|
$ tftpb 1280000 xen-qemu.dtb; fdt addr 1280000 &&&& fdt resize 128; fdt set /amba/ethernet@ff0e0000 status "disabled" &&&& fdt set /amba/ethernet@ff0e0000 xen,passthrough "1" &&&& tftpb 0x80000 xen-Image; tftpb 1400000 xen.ub; tftpb 9000000 xen-rootfs.cpio.gz.u-boot; bootm 1400000 9000000 1280000 |
The full command for booting images you built is shown below:
| Code Block |
|---|
|
$ tftpb 1280000 system.dtb; fdt addr 1280000 &&&& fdt resize 128; fdt set /amba/ethernet@ff0e0000 status "disabled" &&&& fdt set /amba/ethernet@ff0e0000 xen,passthrough "1" &&&& tftpb 0x80000 Image; tftpb 1400000 xen.ub; tftpb 9000000 rootfs.cpio.gz.u-boot; bootm 1400000 9000000 1280000 |
NOTE: If running on hardware you will need to make a change to allow the DMA transactions. See here for more details: Passthrough Network ExampleOnce you have logged onto the system mount the SD card and copy the image.
| Code Block |
|---|
|
mount /dev/mmcblk0 /mnt/ cp /mnt/Image /boot/ |
Similar to before we will use another pre-defined examples in '/etc/xen/'
| Code Block |
|---|
|
xl create -c example-passnet.cfg |
