Table of Contents

Building the Xen Image

Main use of Yocto for Xen is to build custom images. if you want pre-built images please check this page for instruction.

Below are the steps to build the Xen Image using Yocto:

Install the repo

Repo is a tool that enables the management of many git repositories given a single manifest file. The repo will fetch the git repositories specified in the manifest and, by doing so, sets up a Yocto Project build environment for you.

# Download the Repo script.
curl -k https://storage.googleapis.com/git-repo-downloads/repo > repo
# Generally, curl is install at /usr/bin/curl. If it is installed at custom location please point it to correct location.
  
# Make it executable.
chmod a+x repo

# If it is correctly installed, you should see a Usage message when invoked with the help flag.
repo --help
# If system complaints about repo not found. Please run with './repo --help'

Fetch all sources

# initialize the repo.
repo init -u https://github.com/Xilinx/yocto-manifests.git -b <release-version>

# repo sync to get all sources
repo sync

Example of release-version: rel-v2020.2, rel-v2020.1 and rel-v2019.2 etc.

Source environment

# source the environment to build using bitbake.
source setupsdk

Add option to generate rootfs image with Yocto build

This option will generate the rootfs in cpio.gz format which will be used in creating boot-scripts step. Rootfs built will contain the useful Xen tools like xl.

# Edit conf/local.conf and Add the below line
IMAGE_FSTYPES += "cpio.gz"

Add option to generate xen.dtsi

For some Yocto versions, system device tree for xen has not included “xen.dtsi”. To enable this it’s needed to add a user configuration

# Edit recipes-bsp/device-tree/device-tree.bbappend
ENABLE_XEN_DTSI="1"

Build using bitbake

Next, we will build the xen-image-minimal image. This will produce Xen, Linux kernel for Xen and rootfs. If only Xen needs to be build, it can done using MACHINE=vck190-versal bitbake xen.

# For building:
$ MACHINE=vck190-versal bitbake xen-image-minimal
# Please use appropriate yocto machine name to build for a target hardware. Examples:
#MACHINE=vck190-versal
#MACHINE=zcu102-zynqmp
#MACHINE=zcu104-zynqmp
#MACHINE=ultra96-zynqmp

If user wants to compile Xen from external source i.e. wants to modify Xen source code and rebuild the images. Please do the following:

# Before 2022.1 release, edit conf/local.conf as follows:
INHERIT += "externalsrc"
EXTERNALSRC_pn-xen = "/path_to_xen_source/"
EXTERNALSRC_BUILD_pn-xen = "/path_to_xen_source/"
EXTERNALSRC_pn-xen-tools = "/path_to_xen_source/"
EXTERNALSRC_BUILD_pn-xen-tools = "/path_to_xen_source/"

# After 2022.1 release, edit conf/local.conf as follows:
INHERIT+="externalsrc"
EXTERNALSRC:pn-xen = "/path_to_xen_source/"
EXTERNALSRC_BUILD:pn-xen = "/path_to_xen_source/"
EXTERNALSRC:pn-xen-tools = "/path_to_xen_source/"
EXTERNALSRC_BUILD:pn-xen-tools = "/path_to_xen_source/"

# After local.conf is updated, clean the Xen build
MACHINE=vck190-versal bitbake xen-image-minimal -c cleanall xen-tools

# Rebuild using:
MACHINE=vck190-versal bitbake xen-image-minimal

This step can take a lot of time depending upon host machine processing power. It will also required significant disk space, please see Yocto for more details.

Diagnosing Xen Build issue:

If build complains about "error: liblzma: Memory allocation failed", change the memory settings on your machine:
```
sudo sysctl -w vm.overcommit_memory=0
sudo sysctl -w vm.overcommit_ratio=50
```

In case you get the following error

ayankuma@xcbayankuma40x:/scratch$ repo sync
  File "/scratch/.repo/repo/main.py", line 79
    file=sys.stderr)
        ^
SyntaxError: invalid syntax

You need to add python3 before 'repo'. The command will then be as follows :-

# initialize the repo.
python3 repo init -u https://github.com/Xilinx/yocto-manifests.git -b <release-version>

# repo sync to get all sources
python3 repo sync

If builds complains about "Checkpolicy" or flask, do the following:

# edit  yocto_xen/source/meta-virtualization/recipes-extended/xen/xen.inc
# to the DEPENDS variable, add this line below gnu-efi
     checkpolicy-native \
 
# Save and bitbake again

If build fails due to "qemu-xen-dir-find failed" issue: Always clean the xen repo before invoking "MACHINE=zcu102-zynqmp bitbake xen-image-minimal" ie do the following
```
cd /path_to_xen_source/
git clean -fdx
git reset --hard
```

If you see do_package_qa issues like below

Initialising tasks: 100% |############################################################################################| Time: 0:00:03
Sstate summary: Wanted 7 Found 0 Missed 7 Current 2070 (0% match, 99% complete)
NOTE: Executing Tasks
ERROR: xen-tools-4.14+stable999-r0 do_package_qa: QA Issue: xen-tools-pygrub: /work/cortexa72-cortexa53-xilinx-linux/xen-tools/4.14+s
table999-r0/packages-split/xen-tools-pygrub/usr/lib/xen/bin/pygrub maximum shebang size exceeded, the maximum size is 128. [shebang-s
ize]
ERROR: xen-tools-4.14+stable999-r0 do_package_qa: QA Issue: /usr/lib/xen/bin/pygrub contained in package xen-tools-pygrub requires /s
cratch/ayankuma/xilinx_yocto/build/tmp/work/cortexa72-cortexa53-xilinx-linux/xen-tools/4.14+stable999-r0/recipe-sysroot-native/usr/bi
n/python3-native/python3, but no providers found in RDEPENDS_xen-tools-pygrub? [file-rdeps]
ERROR: xen-tools-4.14+stable999-r0 do_package_qa: QA run found fatal errors. Please consider fixing them.
ERROR: Logfile of failure stored in: /scratch/ayankuma/xilinx_yocto/build/tmp/work/cortexa72-cortexa53-xilinx-linux/xen-tools/4.14+s$
able999-r0/temp/log.do_package_qa.11224
ERROR: Task (/scratch/ayankuma/xilinx_yocto/sources/core/../meta-virtualization/recipes-extended/xen/xen-tools_4.14.bb:do_package_qa)
failed with exit code '1'

The solution would be to remove pygrub. Under xen-tools.inc: do_install()
```
rm -r ${D}${bindir}/pygrub
rm -r ${D}${libdir}/xen/bin/pygrub
```

During the boot, if you see issues for launch-xenstore as below

Public key portion is:
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCGljmUgRv8Kri5kDW/iCBElkjPFNs8qUSqOPvxjxbKXR5ZFZg4TjFc3LbHUjZFuOIqCPqWv82qp77rdwnc1dGpKvD7F/FHt9oPJgsFukurhaXhs99P+Z5BLZInj7ygM8V0u6pkZ7u5QyaLDpff03T0yt1BwiLFFHAcWnspJCljzvOIXVGh7RdZ9ob39gA74VpxM+yAxfIafM2xBySYkK56BIL+Jzc810HTSKbP7h5jEYSDBkCa2Or3QEbXth+DAZpapV5no70FcgXWFarevCFIrLgw7T3z8eILg3YPyaSPuWSbLOzUYNGh/RYMMUgUjtAuS4+cm6u3N05O1xT2tJGf root@zcu102-zynqmp
Fingerprint: sha1!! f3:af:73:ac:22:15:0b:b0:97:90:19:75:27:21:e0:ef:c5:cd:c8:99
dropbear.
Starting internet superserver: inetd.
Starting syslogd/klogd: done
Starting /usr/sbin/xenstored.../etc/xen/scripts/launch-xenstore: line 86: prlimit: command not found
..............................
Could not start /usr/sbin/xenstored
Starting domain watchdog daemon: xenwatchdogd startup

The solution would be to add util-linux-prlimit and util-linux to the list of installed images.

You can do either the below change

diff --git a/recipes-extended/xen/xen-tools.inc b/recipes-extended/xen/xen-tools.inc
index 722c503..791f6fb 100644
--- a/recipes-extended/xen/xen-tools.inc
+++ b/recipes-extended/xen/xen-tools.inc
@@ -30,7 +30,7 @@ GLIBC32_class-native = ""
 DEPENDS += "${GLIBC32}"

 RDEPENDS_${PN} = "\
-    bash perl xz \
+    bash perl xz util-linux-prlimit util-linux \
     ${PN}-console \
     ${PN}-libxenguest \
     ${PN}-libxenlight \

Or a proper solution would be to do the following

diff --git a/recipes-extended/images/xen-image-minimal.bb b/recipes-extended/images/xen-image-minimal.bb
index 6733801..ee5bf02 100644
--- a/recipes-extended/images/xen-image-minimal.bb
+++ b/recipes-extended/images/xen-image-minimal.bb
@@ -9,6 +9,8 @@ XEN_KERNEL_MODULES ?= "kernel-module-xen-blkback kernel-module-xen-gntalloc \
                       "

 IMAGE_INSTALL += " \
+    util-linux \
+    util-linux-prlimit \
     packagegroup-core-boot \
     packagegroup-core-ssh-openssh \
     ${XEN_KERNEL_MODULES} \

During boot, if you see this error
1. ```
Setting domain 0 name, domid and JSON config...
FATAL: Failed to open connection to gnttab: No such file or directory
cannot set domid for Dom0
```
2. Solution :- The kernel image is missing some of the XEN configs. You may use the kernel configuration attached here Building Xen Hypervisor through Yocto Flow

If you see errors like the the following

| ERROR: pkg-config binary 'aarch64-xilinx-linux-pkg-config' not found                                                    
|                                                                                                                         
| make: Entering directory '/scratch/ayankuma/upstream_xen/xen/tools/qemu-xen-build'

The remedy is as follows :-

In xen/tools/Makefile, disable building subdir-install-qemu-xen-dir as follows

-subdir-install-qemu-xen-dir: subdir-all-qemu-xen-dir
-	cd emu-xen-build; \
+#subdir-install-qemu-xen-dir: subdir-all-qemu-xen-dir
+#	cd emu-xen-build; \
	$(MAKE) install

Or errors related to "unistd.h" not present or gcc related errors
1. cd <to_dir>/xen
2. git clean -fdx
3. git clean -fdx
4. git reset --hard HEAD
5. cd <to_dir>/xen/tools/qemu-xen-dir-remote
6. git clean -fdx
7. And then in Yocto , do the following
  1. MACHINE=zcu102-zynqmp bitbake -c clean xen-image-minimal
  2. MACHINE=zcu102-zynqmp bitbake xen-image-minimal

If you see errors like the following,

ERROR: xen-tools-4.14+stable999-r0 do_compile: oe_runmake failed
ERROR: xen-tools-4.14+stable999-r0 do_compile: Execution of '/scratch/ayankuma/xilinx_yocto/build/tmp/work/cortexa72-cortexa53-xilinx-linux/xen-tools/4.14+stable999-r0/temp/run.do_compile.23363' failed with exit code 1:
make -C tools/include install                                                                                                                                 
make[1]: Entering directory '/scratch/ayankuma/upstream_xen/xen/tools/include'                                                                                
/scratch/ayankuma/upstream_xen/xen/tools/include/../../tools/Rules.mk:174: *** You have to run ./configure before building or installing the tools.  Stop.

rm -rf tmp
Repeat the steps 5.b

Also, if you see errors related to "version-going-backwards"

You need to remove "version-going-backwards" from ERROR_QA in core/meta/classes/insane.bbclass

Thus core/meta/classes/insane.bbclass will have the following for ERROR_QA

ERROR_QA ?= "dev-so debug-deps dev-deps debug-files arch pkgconfig la \
            perms dep-cmp pkgvarcheck perm-config perm-line perm-link \
            split-strip packages-list pkgv-undefined var-undefined \
            expanded-d invalid-chars \
            license-checksum dev-elf file-rdeps configure-unsafe \
            configure-gettext perllocalpod shebang-size \
            already-stripped installed-vs-shipped ldflags compile-host-path \
            install-host-path pn-overrides unknown-configure-option \
            useless-rpaths rpaths staticdev \
            "

For release 2022.1 and 2022.2, there is an issue with task do_write_xen_qemuboot_dtb for xen-image-minimal build and this results in following error:

| Multiarch not setup properly.
| WARNING: /scratch/devops/yocto/yocto_test/build/tmp/work/vck190_versal-xilinx-linux/xen-image-minimal/1.0-r0/temp/run.do_write_xen_qemuboot_dtb.14323:165 exit 1 from 'qemu-system-aarch64-multiarch -device qemu-xhci -device usb-tablet -device usb-kbd -machine virt,gic-version=3 -machine virtualization=true ${QB_CPU} -m 512 -nographic -serial mon:stdio -machine "dumpdtb=/scratch/devops/yocto/yocto_test/build/tmp/work/vck190_versal-xilinx-linux/xen-image-minimal/1.0-r0/xen-image-minimal-1.0/qemu-dumped.dtb"'
| WARNING: Backtrace (BB generated script):
|       #1: generate_xen_qemuboot_dtb, /scratch/devops/yocto/yocto_test/build/tmp/work/vck190_versal-xilinx-linux/xen-image-minimal/1.0-r0/temp/run.do_write_xen_qemuboot_dtb.14323, line 165
|       #2: do_write_xen_qemuboot_dtb, /scratch/devops/yocto/yocto_test/build/tmp/work/vck190_versal-xilinx-linux/xen-image-minimal/1.0-r0/temp/run.do_write_xen_qemuboot_dtb.14323, line 154
|       #3: main, /scratch/devops/yocto/yocto_test/build/tmp/work/vck190_versal-xilinx-linux/xen-image-minimal/1.0-r0/temp/run.do_write_xen_qemuboot_dtb.14323, line 325
ERROR: Task (/scratch/devops/yocto/yocto_test/sources/core/../meta-virtualization/recipes-extended/images/xen-image-minimal.bb:do_write_xen_qemuboot_dtb) failed with exit code '1'

Please run below steps and apply the attached 0001-Fix-for-xen-image-minimal.patch :

# If git tool is installed, user can apply patch using git:
cd {Yocto_dir}/sources/meta-virtualization
git am 0001-Fix-for-xen-image-minimal.patch

# If git tools isn't installed, user can edit following file:
vim {Yocto_dir}/sources/meta-virtualization/recipes-extended/images/xen-image-minimal.bb and add below line:
deltask do_write_xen_qemuboot_dtb

# Finally, re-run the bitbake
MACHINE=vck190-versal bitbake xen-image-minimal

If there are error about missing pmu-rom.elf like below:

runqemu - ERROR - Failed to run qemu:
Error: Missing PMU ROM: /scratch/devops/yocto/yocto_xen/build/tmp/deploy/images/zcu102-zynqmp/pmu-rom.elf
See "meta-xilinx/README.qemu.md" for more information on accquiring the PMU ROM.

Please copy a pmu-rom.elf under pre-built/linux/images folder of a BSP(you can download from here) or can be build using yocto like below:

 MACHINE=zcu102-zynqmp bitbake petalinux-image-minimal

Creating boot scripts

To create boot scripts, we use Imagebuilder which generates a U-Boot script that loads all the necessary binaries and automatically adds the required entries to device tree at boot time. In order to use it, we will need to write a config file first.

First, cd to <current directory>/tmp/deploy/images/$(machine_name). Example: build/tmp/deploy/images/vck190-versal.

Next, create a file named "config" with the contents as illustrated below and save the file.

MEMORY_START="0x0"
MEMORY_END="0x80000000"

DEVICE_TREE="system.dtb"
XEN="xen"
DOM0_KERNEL="Image"
DOM0_RAMDISK="xen-image-minimal-vck190-versal.cpio.gz"

NUM_DOMUS=0

UBOOT_SOURCE="boot_xen.source"
UBOOT_SCRIPT="boot_xen.scr"

Save the config file.

Now uboot-script-gen can be used to generate boot.scr:

bash uboot-script-gen -c config -d . -t tftp

Running Xen on QEMU for Versal

We will need two terminals to boot Linux on multi arch QEMU. Open a terminal and do the following

cd {Yocto_dir}
source build
# Create a tmp directory For QEMU.
mkdir tmp

# NOTE: Please check if the qemu sd image is of size 2. If size is not power of 2, we need to resize it to next power of 2 size in MB. Example: xen-image-minimal-vck190-versal.wic.qemu-sd size is 768MB, we need to resize it to 1024MB or 1GB. Please do the following:
/scratch/devops/yocto/yocto_test/build/tmp/work/x86_64-linux/qemu-xilinx-helper-native/1.0-r0/recipe-sysroot-native/usr/bin/qemu-img resize tmp/deploy/images/vck190-versal/xen-image-minimal-vck190-versal.wic.qemu-sd 1G

# Launch QEMU for VCK190-Versal
tmp/work/x86_64-linux/qemu-xilinx-helper-native/1.0-r0/recipe-sysroot-native/usr/bin/qemu-system-aarch64 \
-drive if=sd,index=1,file=tmp/deploy/images/vck190-versal/xen-image-minimal-vck190-versal.wic.qemu-sd,format=raw \
-serial null -serial null -serial mon:stdio -display none -boot mode=5 \
-hw-dtb tmp/deploy/images/vck190-versal/qemu-hw-devicetrees/multiarch/board-versal-ps-vck190.dtb \
-net nic -net user,tftp=tmp/deploy/images/vck190-versal \
-machine arm-generic-fdt -m 8G -machine-path tmp_path

# Launch QEMU for ZCU102-Zyqnmp
tmp/work/x86_64-linux/qemu-xilinx-helper-native/1.0-r0/recipe-sysroot-native/usr/bin/qemu-system-aarch64 \
-drive if=sd,index=1,file=tmp/deploy/images/vck190-versal/xen-image-minimal-vck190-versal.wic.qemu-sd,format=raw \
-serial null -serial null -serial mon:stdio -display none -boot mode=5 \
-hw-dtb tmp/deploy/images/vck190-versal/qemu-hw-devicetrees/multiarch/board-versal-ps-vck190.dtb \
-net nic -net user,tftp=tmp/deploy/images/vck190-versal \
-machine arm-generic-fdt -m 8G -machine-path tmp_path

Open another terminal and do the following:

tmp/work/x86_64-linux/qemu-xilinx-helper-native/1.0-r0/recipe-sysroot-native/usr/bin/qemu-system-microblazeel \
-M microblaze-fdt \
-device loader,file=tmp/deploy/images/vck190-versal/BOOT-vck190-versal_bh.bin,addr=0xF201E000,force-raw \
-device loader,addr=0xf0000000,data=0xba020004,data-len=4 -device loader,addr=0xf0000004,data=0xb800fffc,data-len=4 \
-device loader,file=tmp/deploy/images/vck190-versal/pmc_cdo.bin,addr=0xf2000000,force-raw \
-device loader,file=tmp/deploy/images/vck190-versal/plm-vck190-versal.elf,cpu-num=1 \
-device loader,addr=0xF1110624,data=0x0,data-len=4 \
-device loader,addr=0xF1110620,data=0x1,data-len=4 \
-hw-dtb tmp/deploy/images/vck190-versal/qemu-hw-devicetrees/multiarch/board-versal-pmc-vc-p-a2197-00.dtb \
-display none -machine-path tmp_path/

On first terminal, QEMU boot will start.

Running Xen on QEMU for ZynqMP

MACHINE=zcu102-zynqmp runqemu xen-image-minimal

Once you get to the u-boot prompt use the below commands to TFTP boot Xen.

dhcp
tftpb 0xC00000 boot_xen.scr; source 0xC00000

This should start the QEMU boot.

After you enter the above command, the QEMU boot sequence loads the Xen images and boots Linux. At the prompt login, enter root or petalinux(for 2021.2 release and later) as the username and root as the password.

To quit the emulation, press CTRL+A followed by X.