Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents

Table of Contents
excludeTable of Contents

Building the Xen Image

Main use of Yocto for Xen is to build custom images. if you want pre-built images please check this page for instruction.

Below are the steps to build the Xen Image using Yocto:

Install the repo

Repo is a tool that enables the management of many git repositories given a single manifest file. The repo will fetch the git repositories specified in the manifest and, by doing so, sets up a Yocto Project build environment for you.

Code Block
languagebash
themeMidnight
# Download the Repo script.
curl -k https://storage.googleapis.com/git-repo-downloads/repo > repo
# Generally, curl is install at /usr/bin/curl. If it is installed at custom location please point it to correct location.
  
# Make it executable.
chmod a+x repo

# If it is correctly installed, you should see a Usage message when invoked with the help flag.
repo --help
# If system complaints about repo not found. Please run with './repo --help' 

Fetch all sources

Code Block
languagebash
themeMidnight
# initialize the repo.
repo init -u https://github.com/Xilinx/yocto-manifests.git -b <release-version>

# repo sync to get all sources
repo sync

Example of release-version: rel-v2020.2, rel-v2020.1 and rel-v2019.2 etc.


Source environment

Code Block
languagebash
themeMidnight
# source the environment to build using bitbake.
source setupsdk

Add option to generate rootfs image with Yocto build

This option will generate the rootfs in cpio.gz format which will be used in creating boot-scripts step. Rootfs built will contain the useful Xen tools like xl.

Code Block
languagebash
themeMidnight
# Edit conf/local.conf and Add the below line
IMAGE_FSTYPES += "cpio.gz"

Add option to generate xen.dtsi

For some Yocto versions, system device tree for xen has not included “xen.dtsi”. To enable this it’s needed to add a user configuration

Code Block
languagebash
# Edit recipes-bsp/device-tree/device-tree.bbappend
ENABLE_XEN_DTSI="1"

Build using bitbake

Next, we will build the xen-image-minimal image. This will produce Xen, Linux kernel for Xen and rootfs. If only Xen needs to be build, it can done using MACHINE=vck190-versal bitbake xen.
Code Block
themeMidnight
# For building:
$ MACHINE=vck190-versal bitbake xen-image-minimal
# Please use appropriate yocto machine name to build for a target hardware. Examples:
#MACHINE=vck190-versal
#MACHINE=zcu102-zynqmp
#MACHINE=zcu104-zynqmp
#MACHINE=ultra96-zynqmp

If user wants to compile Xen from external source i.e. wants to modify Xen source code and rebuild the images. Please do the following:

Code Block
languagebash
# Before 2022.1 release, edit conf/local.conf as follows:
INHERIT += "externalsrc"
EXTERNALSRC_pn-xen = "/path_to_xen_source/"
EXTERNALSRC_BUILD_pn-xen = "/path_to_xen_source/"
EXTERNALSRC_pn-xen-tools = "/path_to_xen_source/"
EXTERNALSRC_BUILD_pn-xen-tools = "/path_to_xen_source/"

# After 2022.1 release, edit conf/local.conf as follows:
INHERIT+="externalsrc"
EXTERNALSRC:pn-xen = "/path_to_xen_source/"
EXTERNALSRC_BUILD:pn-xen = "/path_to_xen_source/"
EXTERNALSRC:pn-xen-tools = "/path_to_xen_source/"
EXTERNALSRC_BUILD:pn-xen-tools = "/path_to_xen_source/"

# After local.conf is updated, clean the Xen build
MACHINE=vck190-versal bitbake xen-image-minimal -c cleanall xen-tools

# Rebuild using:
MACHINE=vck190-versal bitbake xen-image-minimal


Warning

This step can take a lot of time depending upon host machine processing power. It will also required significant disk space, please see Yocto for more details.

Diagnosing Xen Build issue:


  1. If build complains about "error: liblzma: Memory allocation failed", change the memory settings on your machine:

    Code Block
    languagebash
    sudo sysctl -w vm.overcommit_memory=0
    sudo sysctl -w vm.overcommit_ratio=50


  2. In case you get the following error

    Code Block
    languagebash
    ayankuma@xcbayankuma40x:/scratch$ repo sync
      File "/scratch/.repo/repo/main.py", line 79
        file=sys.stderr)
            ^
    SyntaxError: invalid syntax


    You need to add python3 before 'repo'. The command will then be as follows :-

    Code Block
    languagebash
    # initialize the repo.
    python3 repo init -u https://github.com/Xilinx/yocto-manifests.git -b <release-version>
    
    # repo sync to get all sources
    python3 repo sync


  3. If builds complains about "Checkpolicy" or flask, do the following:

    Code Block
    languagebash
    # edit  yocto_xen/source/meta-virtualization/recipes-extended/xen/xen.inc
    # to the DEPENDS variable, add this line below gnu-efi
         checkpolicy-native \
     
    # Save and bitbake again


  4. If build fails due to "qemu-xen-dir-find failed" issue: Always clean the xen repo before invoking "MACHINE=zcu102-zynqmp bitbake xen-image-minimal" ie do the following

    Code Block
    languagebash
    cd /path_to_xen_source/
    git clean -fdx
    git reset --hard


  5. If you see do_package_qa issues like below

    1. Code Block
      languagebash
      Initialising tasks: 100% |############################################################################################| Time: 0:00:03
      Sstate summary: Wanted 7 Found 0 Missed 7 Current 2070 (0% match, 99% complete)
      NOTE: Executing Tasks
      ERROR: xen-tools-4.14+stable999-r0 do_package_qa: QA Issue: xen-tools-pygrub: /work/cortexa72-cortexa53-xilinx-linux/xen-tools/4.14+s
      table999-r0/packages-split/xen-tools-pygrub/usr/lib/xen/bin/pygrub maximum shebang size exceeded, the maximum size is 128. [shebang-s
      ize]
      ERROR: xen-tools-4.14+stable999-r0 do_package_qa: QA Issue: /usr/lib/xen/bin/pygrub contained in package xen-tools-pygrub requires /s
      cratch/ayankuma/xilinx_yocto/build/tmp/work/cortexa72-cortexa53-xilinx-linux/xen-tools/4.14+stable999-r0/recipe-sysroot-native/usr/bi
      n/python3-native/python3, but no providers found in RDEPENDS_xen-tools-pygrub? [file-rdeps]
      ERROR: xen-tools-4.14+stable999-r0 do_package_qa: QA run found fatal errors. Please consider fixing them.
      ERROR: Logfile of failure stored in: /scratch/ayankuma/xilinx_yocto/build/tmp/work/cortexa72-cortexa53-xilinx-linux/xen-tools/4.14+s$
      able999-r0/temp/log.do_package_qa.11224
      ERROR: Task (/scratch/ayankuma/xilinx_yocto/sources/core/../meta-virtualization/recipes-extended/xen/xen-tools_4.14.bb:do_package_qa)
      failed with exit code '1'


    2. The solution would be to remove pygrub. Under xen-tools.inc: do_install()

      Code Block
      languagebash
      rm -r ${D}${bindir}/pygrub
      rm -r ${D}${libdir}/xen/bin/pygrub


  6. During the boot, if you see issues for launch-xenstore as below

    1. Code Block
      languagebash
      Public key portion is:
      ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCGljmUgRv8Kri5kDW/iCBElkjPFNs8qUSqOPvxjxbKXR5ZFZg4TjFc3LbHUjZFuOIqCPqWv82qp77rdwnc1dGpKvD7F/FHt9oPJgsFukurhaXhs99P+Z5BLZInj7ygM8V0u6pkZ7u5QyaLDpff03T0yt1BwiLFFHAcWnspJCljzvOIXVGh7RdZ9ob39gA74VpxM+yAxfIafM2xBySYkK56BIL+Jzc810HTSKbP7h5jEYSDBkCa2Or3QEbXth+DAZpapV5no70FcgXWFarevCFIrLgw7T3z8eILg3YPyaSPuWSbLOzUYNGh/RYMMUgUjtAuS4+cm6u3N05O1xT2tJGf root@zcu102-zynqmp
      Fingerprint: sha1!! f3:af:73:ac:22:15:0b:b0:97:90:19:75:27:21:e0:ef:c5:cd:c8:99
      dropbear.
      Starting internet superserver: inetd.
      Starting syslogd/klogd: done
      Starting /usr/sbin/xenstored.../etc/xen/scripts/launch-xenstore: line 86: prlimit: command not found
      ..............................
      Could not start /usr/sbin/xenstored
      Starting domain watchdog daemon: xenwatchdogd startup


    2. The solution would be to add util-linux-prlimit and util-linux to the list of installed images.
      1. You can do either the below change

        Code Block
        languagebash
        diff --git a/recipes-extended/xen/xen-tools.inc b/recipes-extended/xen/xen-tools.inc
        index 722c503..791f6fb 100644
        --- a/recipes-extended/xen/xen-tools.inc
        +++ b/recipes-extended/xen/xen-tools.inc
        @@ -30,7 +30,7 @@ GLIBC32_class-native = ""
         DEPENDS += "${GLIBC32}"
        
         RDEPENDS_${PN} = "\
        -    bash perl xz \
        +    bash perl xz util-linux-prlimit util-linux \
             ${PN}-console \
             ${PN}-libxenguest \
             ${PN}-libxenlight \


      2. Or a proper solution would be to do the following


        Code Block
        languagebash
        diff --git a/recipes-extended/images/xen-image-minimal.bb b/recipes-extended/images/xen-image-minimal.bb
        index 6733801..ee5bf02 100644
        --- a/recipes-extended/images/xen-image-minimal.bb
        +++ b/recipes-extended/images/xen-image-minimal.bb
        @@ -9,6 +9,8 @@ XEN_KERNEL_MODULES ?= "kernel-module-xen-blkback kernel-module-xen-gntalloc \
                               "
        
         IMAGE_INSTALL += " \
        +    util-linux \
        +    util-linux-prlimit \
             packagegroup-core-boot \
             packagegroup-core-ssh-openssh \
             ${XEN_KERNEL_MODULES} \


  7. During boot, if you see this error

    1. Code Block
      languagebash
      Setting domain 0 name, domid and JSON config...
      FATAL: Failed to open connection to gnttab: No such file or directory
      cannot set domid for Dom0


    2. Solution :- The kernel image is missing some of the XEN configs. You may use the kernel configuration attached here 1697873938
  8. If you see errors like the the following

    1. Code Block
      languagebash
      | ERROR: pkg-config binary 'aarch64-xilinx-linux-pkg-config' not found                                                    
      |                                                                                                                         
      | make: Entering directory '/scratch/ayankuma/upstream_xen/xen/tools/qemu-xen-build'                                                 
      
      


      1. The remedy is as follows :-


        1. In xen/tools/Makefile, disable building subdir-install-qemu-xen-dir as follows

        2. Code Block
          languagebash
          -subdir-install-qemu-xen-dir: subdir-all-qemu-xen-dir
          -	cd emu-xen-build; \
          +#subdir-install-qemu-xen-dir: subdir-all-qemu-xen-dir
          +#	cd emu-xen-build; \
          	$(MAKE) install


    2. Or errors related to "unistd.h" not present or gcc related errors
      1. cd <to_dir>/xen
      2. git clean -fdx
      3. git clean -fdx
      4. git reset --hard HEAD
      5. cd <to_dir>/xen/tools/qemu-xen-dir-remote
      6. git clean -fdx
      7. And then in Yocto , do the following
        1. MACHINE=zcu102-zynqmp bitbake -c clean  xen-image-minimal
        2. MACHINE=zcu102-zynqmp bitbake  xen-image-minimal
  9. If you see errors like the following,

    1. Code Block
      ERROR: xen-tools-4.14+stable999-r0 do_compile: oe_runmake failed
      ERROR: xen-tools-4.14+stable999-r0 do_compile: Execution of '/scratch/ayankuma/xilinx_yocto/build/tmp/work/cortexa72-cortexa53-xilinx-linux/xen-tools/4.14+stable999-r0/temp/run.do_compile.23363' failed with exit code 1:
      make -C tools/include install                                                                                                                                 
      make[1]: Entering directory '/scratch/ayankuma/upstream_xen/xen/tools/include'                                                                                
      /scratch/ayankuma/upstream_xen/xen/tools/include/../../tools/Rules.mk:174: *** You have to run ./configure before building or installing the tools.  Stop.                                                                                                                                        


    2. rm -rf tmp


    3. Repeat the steps 5.b

  10. Also, if you see errors related to "version-going-backwards"  
    1. You need to remove "version-going-backwards" from ERROR_QA in core/meta/classes/insane.bbclass
    2. Thus  core/meta/classes/insane.bbclass will have the following for ERROR_QA

      Code Block
      languagebash
      ERROR_QA ?= "dev-so debug-deps dev-deps debug-files arch pkgconfig la \
                  perms dep-cmp pkgvarcheck perm-config perm-line perm-link \
                  split-strip packages-list pkgv-undefined var-undefined \
                  expanded-d invalid-chars \
                  license-checksum dev-elf file-rdeps configure-unsafe \
                  configure-gettext perllocalpod shebang-size \
                  already-stripped installed-vs-shipped ldflags compile-host-path \
                  install-host-path pn-overrides unknown-configure-option \
                  useless-rpaths rpaths staticdev \
                  "
      


  11. For release 2022.1 and 2022.2, there is an issue with task do_write_xen_qemuboot_dtb for xen-image-minimal build and this results in following error:

    Code Block
    languagebash
    | Multiarch not setup properly.
    | WARNING: /scratch/devops/yocto/yocto_test/build/tmp/work/vck190_versal-xilinx-linux/xen-image-minimal/1.0-r0/temp/run.do_write_xen_qemuboot_dtb.14323:165 exit 1 from 'qemu-system-aarch64-multiarch -device qemu-xhci -device usb-tablet -device usb-kbd -machine virt,gic-version=3 -machine virtualization=true ${QB_CPU} -m 512 -nographic -serial mon:stdio -machine "dumpdtb=/scratch/devops/yocto/yocto_test/build/tmp/work/vck190_versal-xilinx-linux/xen-image-minimal/1.0-r0/xen-image-minimal-1.0/qemu-dumped.dtb"'
    | WARNING: Backtrace (BB generated script):
    |       #1: generate_xen_qemuboot_dtb, /scratch/devops/yocto/yocto_test/build/tmp/work/vck190_versal-xilinx-linux/xen-image-minimal/1.0-r0/temp/run.do_write_xen_qemuboot_dtb.14323, line 165
    |       #2: do_write_xen_qemuboot_dtb, /scratch/devops/yocto/yocto_test/build/tmp/work/vck190_versal-xilinx-linux/xen-image-minimal/1.0-r0/temp/run.do_write_xen_qemuboot_dtb.14323, line 154
    |       #3: main, /scratch/devops/yocto/yocto_test/build/tmp/work/vck190_versal-xilinx-linux/xen-image-minimal/1.0-r0/temp/run.do_write_xen_qemuboot_dtb.14323, line 325
    ERROR: Task (/scratch/devops/yocto/yocto_test/sources/core/../meta-virtualization/recipes-extended/images/xen-image-minimal.bb:do_write_xen_qemuboot_dtb) failed with exit code '1'
    

    Please run below steps and apply the attached 0001-Fix-for-xen-image-minimal.patch :


    Code Block
    languagebash
    # If git tool is installed, user can apply patch using git:
    cd {Yocto_dir}/sources/meta-virtualization
    git am 0001-Fix-for-xen-image-minimal.patch
    
    # If git tools isn't installed, user can edit following file:
    vim {Yocto_dir}/sources/meta-virtualization/recipes-extended/images/xen-image-minimal.bb and add below line:
    deltask do_write_xen_qemuboot_dtb
    
    # Finally, re-run the bitbake
    MACHINE=vck190-versal bitbake xen-image-minimal


  12. If there are error about missing pmu-rom.elf like below: 

    Code Block
    languagebash
    runqemu - ERROR - Failed to run qemu:
    Error: Missing PMU ROM: /scratch/devops/yocto/yocto_xen/build/tmp/deploy/images/zcu102-zynqmp/pmu-rom.elf
    See "meta-xilinx/README.qemu.md" for more information on accquiring the PMU ROM.

    Please copy a pmu-rom.elf under pre-built/linux/images folder of a BSP(you can download from here) or can be build using yocto like below:

    Code Block
    languagebash
     MACHINE=zcu102-zynqmp bitbake petalinux-image-minimal


Creating boot scripts

To create boot scripts, we use Imagebuilder which generates a U-Boot script that loads all the necessary binaries and automatically adds the required entries to device tree at boot time. In order to use it, we will need to write a config file first.

First, cd to <current directory>/tmp/deploy/images/$(machine_name). Example: build/tmp/deploy/images/vck190-versal.

Next, create a file named "config" with the contents as illustrated below and save the file.


Code Block
themeMidnight
MEMORY_START="0x0"
MEMORY_END="0x80000000"

DEVICE_TREE="system.dtb"
XEN="xen"
DOM0_KERNEL="Image"
DOM0_RAMDISK="xen-image-minimal-vck190-versal.cpio.gz"

NUM_DOMUS=0

UBOOT_SOURCE="boot_xen.source"
UBOOT_SCRIPT="boot_xen.scr"


...