Table of Contents

Building the Xen Image

Main use of Yocto for Xen is to build custom images. if you want pre-built images please check this page for instruction.

Below are the steps to build the Xen Image using Yocto:

Install the repo

Repo is a tool that enables the management of many git repositories given a single manifest file. The repo will fetch the git repositories specified in the manifest and, by doing so, sets up a Yocto Project build environment for you.

Fetch all sources

Example of release-version: rel-v2020.2, rel-v2020.1 and rel-v2019.2 etc.

Source environment

Add option to generate rootfs image with Yocto build

This option will generate the rootfs in cpio.gz format which will be used in creating boot-scripts step. Rootfs built will contain the useful Xen tools like xl.

Add option to generate xen.dtsi

For some Yocto versions, system device tree for xen has not included “xen.dtsi”. To enable this it’s needed to add a user configuration

Build using bitbake

Next, we will build the xen-image-minimal image. This will produce Xen, Linux kernel for Xen and rootfs. If only Xen needs to be build, it can done using MACHINE=vck190-versal bitbake xen.

If user wants to compile Xen from external source i.e. wants to modify Xen source code and rebuild the images. Please do the following:

Diagnosing Xen Build issue:

1. If build complains about "error: liblzma: Memory allocation failed", change the memory settings on your machine:

   sudo sysctl -w vm.overcommit_memory=0 sudo sysctl -w vm.overcommit_ratio=50

2. In case you get the following error

   ayankuma@xcbayankuma40x:/scratch$ repo sync File "/scratch/.repo/repo/main.py", line 79 file=sys.stderr) ^ SyntaxError: invalid syntax

   You need to add python3 before 'repo'. The command will then be as follows :-

   # initialize the repo. python3 repo init -u https://github.com/Xilinx/yocto-manifests.git -b <release-version> # repo sync to get all sources python3 repo sync

3. If builds complains about "Checkpolicy" or flask, do the following:

   # edit  yocto_xen/source/meta-virtualization/recipes-extended/xen/xen.inc # to the DEPENDS variable, add this line below gnu-efi      checkpolicy-native \   # Save and bitbake again

4. If build fails due to "qemu-xen-dir-find failed" issue: Always clean the xen repo before invoking "MACHINE=zcu102-zynqmp bitbake xen-image-minimal" ie do the following

   cd /path_to_xen_source/ git clean -fdx git reset --hard

5. If you see do_package_qa issues like below

   1. Initialising tasks: 100% |############################################################################################| Time: 0:00:03 Sstate summary: Wanted 7 Found 0 Missed 7 Current 2070 (0% match, 99% complete) NOTE: Executing Tasks ERROR: xen-tools-4.14+stable999-r0 do_package_qa: QA Issue: xen-tools-pygrub: /work/cortexa72-cortexa53-xilinx-linux/xen-tools/4.14+s table999-r0/packages-split/xen-tools-pygrub/usr/lib/xen/bin/pygrub maximum shebang size exceeded, the maximum size is 128. [shebang-s ize] ERROR: xen-tools-4.14+stable999-r0 do_package_qa: QA Issue: /usr/lib/xen/bin/pygrub contained in package xen-tools-pygrub requires /s cratch/ayankuma/xilinx_yocto/build/tmp/work/cortexa72-cortexa53-xilinx-linux/xen-tools/4.14+stable999-r0/recipe-sysroot-native/usr/bi n/python3-native/python3, but no providers found in RDEPENDS_xen-tools-pygrub? [file-rdeps] ERROR: xen-tools-4.14+stable999-r0 do_package_qa: QA run found fatal errors. Please consider fixing them. ERROR: Logfile of failure stored in: /scratch/ayankuma/xilinx_yocto/build/tmp/work/cortexa72-cortexa53-xilinx-linux/xen-tools/4.14+s$ able999-r0/temp/log.do_package_qa.11224 ERROR: Task (/scratch/ayankuma/xilinx_yocto/sources/core/../meta-virtualization/recipes-extended/xen/xen-tools_4.14.bb:do_package_qa) failed with exit code '1'

   2. The solution would be to remove pygrub. Under xen-tools.inc: do_install()

      rm -r ${D}${bindir}/pygrub rm -r ${D}${libdir}/xen/bin/pygrub

6. During the boot, if you see issues for launch-xenstore as below

   1. Public key portion is: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCGljmUgRv8Kri5kDW/iCBElkjPFNs8qUSqOPvxjxbKXR5ZFZg4TjFc3LbHUjZFuOIqCPqWv82qp77rdwnc1dGpKvD7F/FHt9oPJgsFukurhaXhs99P+Z5BLZInj7ygM8V0u6pkZ7u5QyaLDpff03T0yt1BwiLFFHAcWnspJCljzvOIXVGh7RdZ9ob39gA74VpxM+yAxfIafM2xBySYkK56BIL+Jzc810HTSKbP7h5jEYSDBkCa2Or3QEbXth+DAZpapV5no70FcgXWFarevCFIrLgw7T3z8eILg3YPyaSPuWSbLOzUYNGh/RYMMUgUjtAuS4+cm6u3N05O1xT2tJGf root@zcu102-zynqmp Fingerprint: sha1!! f3:af:73:ac:22:15:0b:b0:97:90:19:75:27:21:e0:ef:c5:cd:c8:99 dropbear. Starting internet superserver: inetd. Starting syslogd/klogd: done Starting /usr/sbin/xenstored.../etc/xen/scripts/launch-xenstore: line 86: prlimit: command not found .............................. Could not start /usr/sbin/xenstored Starting domain watchdog daemon: xenwatchdogd startup

   2. The solution would be to add util-linux-prlimit and util-linux to the list of installed images.

      1. You can do either the below change

         diff --git a/recipes-extended/xen/xen-tools.inc b/recipes-extended/xen/xen-tools.inc index 722c503..791f6fb 100644 --- a/recipes-extended/xen/xen-tools.inc +++ b/recipes-extended/xen/xen-tools.inc @@ -30,7 +30,7 @@ GLIBC32_class-native = "" DEPENDS += "${GLIBC32}" RDEPENDS_${PN} = "\ - bash perl xz \ + bash perl xz util-linux-prlimit util-linux \ ${PN}-console \ ${PN}-libxenguest \ ${PN}-libxenlight \

      2. Or a proper solution would be to do the following

         diff --git a/recipes-extended/images/xen-image-minimal.bb b/recipes-extended/images/xen-image-minimal.bb index 6733801..ee5bf02 100644 --- a/recipes-extended/images/xen-image-minimal.bb +++ b/recipes-extended/images/xen-image-minimal.bb @@ -9,6 +9,8 @@ XEN_KERNEL_MODULES ?= "kernel-module-xen-blkback kernel-module-xen-gntalloc \ " IMAGE_INSTALL += " \ + util-linux \ + util-linux-prlimit \ packagegroup-core-boot \ packagegroup-core-ssh-openssh \ ${XEN_KERNEL_MODULES} \

7. During boot, if you see this error

   1. Setting domain 0 name, domid and JSON config... FATAL: Failed to open connection to gnttab: No such file or directory cannot set domid for Dom0

   2. Solution :- The kernel image is missing some of the XEN configs. You may use the kernel configuration attached here Building Xen Hypervisor through Yocto Flow

8. If you see errors like the the following

   1. | ERROR: pkg-config binary 'aarch64-xilinx-linux-pkg-config' not found | | make: Entering directory '/scratch/ayankuma/upstream_xen/xen/tools/qemu-xen-build'

      1. The remedy is as follows :-

         1. In xen/tools/Makefile, disable building subdir-install-qemu-xen-dir as follows

         2. -subdir-install-qemu-xen-dir: subdir-all-qemu-xen-dir - cd emu-xen-build; \ +#subdir-install-qemu-xen-dir: subdir-all-qemu-xen-dir +# cd emu-xen-build; \ $(MAKE) install

   2. Or errors related to "unistd.h" not present or gcc related errors

      1. cd <to_dir>/xen

      2. git clean -fdx

      3. git clean -fdx

      4. git reset --hard HEAD

      5. cd <to_dir>/xen/tools/qemu-xen-dir-remote

      6. git clean -fdx

      7. And then in Yocto , do the following

         1. MACHINE=zcu102-zynqmp bitbake -c clean  xen-image-minimal

         2. MACHINE=zcu102-zynqmp bitbake  xen-image-minimal

9. If you see errors like the following,

   1. ERROR: xen-tools-4.14+stable999-r0 do_compile: oe_runmake failed ERROR: xen-tools-4.14+stable999-r0 do_compile: Execution of '/scratch/ayankuma/xilinx_yocto/build/tmp/work/cortexa72-cortexa53-xilinx-linux/xen-tools/4.14+stable999-r0/temp/run.do_compile.23363' failed with exit code 1: make -C tools/include install make[1]: Entering directory '/scratch/ayankuma/upstream_xen/xen/tools/include' /scratch/ayankuma/upstream_xen/xen/tools/include/../../tools/Rules.mk:174: *** You have to run ./configure before building or installing the tools. Stop.

   2. rm -rf tmp

   3. Repeat the steps 5.b

10. Also, if you see errors related to "version-going-backwards"  

    1. You need to remove "version-going-backwards" from ERROR_QA in core/meta/classes/insane.bbclass

    2. Thus  core/meta/classes/insane.bbclass will have the following for ERROR_QA

       ERROR_QA ?= "dev-so debug-deps dev-deps debug-files arch pkgconfig la \ perms dep-cmp pkgvarcheck perm-config perm-line perm-link \ split-strip packages-list pkgv-undefined var-undefined \ expanded-d invalid-chars \ license-checksum dev-elf file-rdeps configure-unsafe \ configure-gettext perllocalpod shebang-size \ already-stripped installed-vs-shipped ldflags compile-host-path \ install-host-path pn-overrides unknown-configure-option \ useless-rpaths rpaths staticdev \ "

11. For release 2022.1 and 2022.2, there is an issue with task do_write_xen_qemuboot_dtb for xen-image-minimal build and this results in following error:

    | Multiarch not setup properly. | WARNING: /scratch/devops/yocto/yocto_test/build/tmp/work/vck190_versal-xilinx-linux/xen-image-minimal/1.0-r0/temp/run.do_write_xen_qemuboot_dtb.14323:165 exit 1 from 'qemu-system-aarch64-multiarch -device qemu-xhci -device usb-tablet -device usb-kbd -machine virt,gic-version=3 -machine virtualization=true ${QB_CPU} -m 512 -nographic -serial mon:stdio -machine "dumpdtb=/scratch/devops/yocto/yocto_test/build/tmp/work/vck190_versal-xilinx-linux/xen-image-minimal/1.0-r0/xen-image-minimal-1.0/qemu-dumped.dtb"' | WARNING: Backtrace (BB generated script): | #1: generate_xen_qemuboot_dtb, /scratch/devops/yocto/yocto_test/build/tmp/work/vck190_versal-xilinx-linux/xen-image-minimal/1.0-r0/temp/run.do_write_xen_qemuboot_dtb.14323, line 165 | #2: do_write_xen_qemuboot_dtb, /scratch/devops/yocto/yocto_test/build/tmp/work/vck190_versal-xilinx-linux/xen-image-minimal/1.0-r0/temp/run.do_write_xen_qemuboot_dtb.14323, line 154 | #3: main, /scratch/devops/yocto/yocto_test/build/tmp/work/vck190_versal-xilinx-linux/xen-image-minimal/1.0-r0/temp/run.do_write_xen_qemuboot_dtb.14323, line 325 ERROR: Task (/scratch/devops/yocto/yocto_test/sources/core/../meta-virtualization/recipes-extended/images/xen-image-minimal.bb:do_write_xen_qemuboot_dtb) failed with exit code '1'

    Please run below steps and apply the attached  :

    # If git tool is installed, user can apply patch using git: cd {Yocto_dir}/sources/meta-virtualization git am 0001-Fix-for-xen-image-minimal.patch # If git tools isn't installed, user can edit following file: vim {Yocto_dir}/sources/meta-virtualization/recipes-extended/images/xen-image-minimal.bb and add below line: deltask do_write_xen_qemuboot_dtb # Finally, re-run the bitbake MACHINE=vck190-versal bitbake xen-image-minimal

12. If there are error about missing pmu-rom.elf like below: 

    runqemu - ERROR - Failed to run qemu: Error: Missing PMU ROM: /scratch/devops/yocto/yocto_xen/build/tmp/deploy/images/zcu102-zynqmp/pmu-rom.elf See "meta-xilinx/README.qemu.md" for more information on accquiring the PMU ROM.

    Please copy a pmu-rom.elf under pre-built/linux/images folder of a BSP(you can download from here) or can be build using yocto like below:

    MACHINE=zcu102-zynqmp bitbake petalinux-image-minimal

Creating boot scripts

To create boot scripts, we use Imagebuilder which generates a U-Boot script that loads all the necessary binaries and automatically adds the required entries to device tree at boot time. In order to use it, we will need to write a config file first.

First, cd to <current directory>/tmp/deploy/images/$(machine_name). Example: build/tmp/deploy/images/vck190-versal.

Next, create a file named "config" with the contents as illustrated below and save the file.

Save the config file. 

Now uboot-script-gen can be used to generate boot.scr:

Running Xen on QEMU for Versal

We will need two terminals to boot Linux on multi arch QEMU. Open a terminal and do the following

Open another terminal and do the following:

On first terminal, QEMU boot will start.

Running Xen on QEMU for ZynqMP

Once you get to the u-boot prompt use the below commands to TFTP boot Xen.

This should start the QEMU boot.

After you enter the above command, the QEMU boot sequence loads the Xen images and boots Linux. At the prompt login, enter root or petalinux(for 2021.2 release and later) as the username and root as the password.