This article describes a flow to build a CentOS 8 system for Zynq UltraScale+ which includes building UEFI firmware, building the CentOS 8 kernel and installing these images on a prebuilt CentOS 8 disk OpenStack cloud image.
Table of Contents
Table of Contents | ||||
---|---|---|---|---|
|
Introduction
Embedded Linux systems are typically lightweight and cross compiled from build frameworks such as Yocto, Petalinux or Buildroot. However, as embedded systems become more powerful and look more like embedded servers, ARM64 machines are capable of running these higher level Linux distributions. Distros such as CentOS provide attractive features such as SELinux which are security enhancements including mandatory access controls (MAC), updated security patches, package managers such as Yum or DNF for installing software remotely from trusted repositories, a stable ABI for maintaining compatible software and modules and a well known compute environment. These distros maintain their own kernel repositories by pulling from mainline and applying patches. Xilinx is committed to upstreaming drivers, so drivers accepted by mainline will automatically get merged into the distro kernel source when they pull a corresponding release. Distros also provide a build system generally based on the package manager to authenticate source, sign binaries and automate the build (fetch, patch, configure, build, package).
...
Note |
---|
Building a CentOS 8 system for Zynq UltraScale+ is currently a proof-of-concept (POCPoC). You should not assume this builds a Xilinx supported, production-ready CentOS 8 system. Please do your due diligence before deploying CentOS 8 and regression test against your system requirements. |
Requirements
CentOS 8 VM or bare-metal host machine
Minimum hard disk space
25 GB without UEFI build
50 GB with UEFI build
Sudo privileges
Terminal emulator
Internet access
ZCU102 evaluation board
SD card (8 GB ( minimum)/16 GB ( recommended) SD card
Porting Philosophy and Strategy
The philosophy was to make the ZCU102 look as much like a standard Linux PC or server and rely on CentOS for most of the heavy lifting. Essentially we want to do as little work as possible. The strategy was to isolate the components where Xilinx tools build the low level architecture dependent components, UEFI firmware and DTB and the distro provides the kernel source, bootloader components and the root filesystem. The focus is on the kernel, which needs to be configured and built to support Xilinx platforms. Fortunately Xilinx has committed to upstreaming drivers, so many of the required drivers needed to boot the system are already present in the CentOS 8 kernel tree. That said, that the timing of getting these commits accepted upstream can be lengthy, so there may be significant lags until they are accepted. Therefore upstream kernel versions may not have the equivalent driver support of a similar kernel version in the Xilinx GitHub repository. If you need patches or drivers that are not upstreamed, then you will need to backport them yourself or build a kernel from the Xilinx GitHub repository.
CentOS 8 Host Machine
Install Host Package Dependencies
At this point I assume you have administered a base CentOS 8 machine including sudo privilege, terminal emulator and internet access. I recommend allocating 25 GB or more free space. If you are building the UEFI firmware, then you should allocate a minimum of 50 GB. First lets install some host dependencies.
...
CentOS 8 supports GCC 8.2.1, so I recommend sticking with the same version for to guarantee ABI compatibility. You may install GCC from any provider such as ARM, Linaro , or Xilinx ( Vitis), etc. The link below will pull the aarch64 cross compiler 8.2.1 binaries from ARM.
...
Once the cross compiler is downloaded, extract it to a location of your choice and add the bin directory to your PATH. For example, I installed it in $HOME/bin and prepended PATH in my “.bashrc” file.
Code Block | ||
---|---|---|
| ||
PATH="$HOME/bin:$HOME/bin/gcc-arm-8.2-2019.01-x86_64-aarch64-linux-gnu/bin:$PATH" |
Prepare Workspace
Here we are going to simply create a basic top level directory structure from which we will build our system.
Code Block | ||
---|---|---|
| ||
$ export COS_BUILD=$HOME/centos8-dev
$ mkdir -p $COS_BUILD/rpmbuild $COS_BUILD/images/{efi,boot,rootfs} $COS_BUILD/deploy/{efi/dtb/xilinx,boot} |
Build the UEFI Firmware
In order to boot a Zynq UltraScale+ device, the boot ROM expects a “boot.bin” binary image. The “boot.bin” is composed of partitions that hold images for various portions of the boot process and run-time. In a Linux system this image generally holds the FSBL, PMU firmware, PL bitstream and u-boot. For the purposes of this article, I am going to ignore the PL bitstream. These images can be built and packaged in several ways, but I’ll show how to build it it through Xilinx Yocto.
...
Copy the generated “boot.bin” into the deploy directory we created earlier.
Code Block | ||
---|---|---|
| ||
$ cp tmp/deploy/images/zcu102-zynqmp/BOOT-zcu102-zynqmp.bin $COS_BUILD/deploy/efi/boot.bin |
Build the Kernel
This build section walks through two builds of the kernel. The first is a bootstrap kernel, which is a straight forward make build. The second uses the official bootstrap kernel does not rely on any kernel modules to mount the final root filesystem. The second kernel is built with the CentOS RPM build system for CentOS. The This build system pulls directly from the CentOS kernel repository which also includes patches, kernel configurations and utility build scripts.
Clone the CentOS 8 Kernel Source
Here we are going to clone “centos-git-common” which includes utility build scripts for CentOS and the CentOS “kernel” repository which includes source and spec files for the RPM build. The “get_sources.sh” script will fetch the kernel source tarball.
Code Block | ||
---|---|---|
| ||
$ cd $COS_BUILD/rpmbuild $ git clone https://git.centos.org/centos-git-common.git $ git clone https://git.centos.org/rpms/kernel.git $ cd kernel $ git checkout -b kernel-4.18.0-147.3.1.el8_1.xlnx imports/c8/kernel-4.18.0-147.3.1.el8_1 $ ../centos-git-common/get_sources.sh |
As a good measure, lets install any additional package dependencies required by the SPEC file that we may have missed.
Code Block |
---|
sudo dnf builddep SPECS/kernel.spec |
Note |
---|
This may may cause your host to hang on a reboot. Try building without this first and install any missing packages manually. |
Build a Bootstrap Kernel
Build a Bootstrap Kernel
Because the installed CentOS kernel image may not include some critical configurations required for Zynq UltraScale+, you may need to build a bootstrap kernel. Once we boot the bootstrap kernel, we can formally install the kernel RPMS which will installs the newly configured kernel and updates the initrd and GRUB configuration.
...
Tip |
---|
If all of these are options are enabled, then the installed kernel should boot on the ZCU102 and you can skip to “Build the Kernel RPM Packages.” |
Warning |
---|
If the kernel hangs, i.e. no kernel console messages, then revisit this section and try building a bootstrap kernel. Note you should still see the FSBL banner, PMUFW, ATF and u-boot messages. If you don’t, then something is wrong with your boot.bin on the FAT partition. |
First we need to run the “%prep” stage First we need to run the “%prep” stage in our SPEC file which will unpack and patch the kernel source.
...
Code Block |
---|
$ make distclean |
Build the Kernel RPM Packages
Create a Kernel Configuration Fragment
Create the configuration fragment file “kernel-aarch64-zynqmp.cfg” in the SOURCES directory with the kernel configuration as shown below.
...
Code Block | ||
---|---|---|
| ||
$ cd $COS_BUILD/rpmbuild/kernel/SOURCES $ cat << EOF > kernel-aarch64-zynqmp.cfg CONFIG_ARCH_ZYNQMP=y CONFIG_SERIAL_XILINX_PS_UART=y CONFIG_SERIAL_XILINX_PS_UART_CONSOLE=y CONFIG_MMC_SDHCI_OF_ARASAN=m CONFIG_I2C_CADENCE=m CONFIG_GPIO_ZYNQ=m CONFIG_NET_CADENCE=y CONFIG_MACB=m CONFIG_RTC_DRV_ZYNQMP=m CONFIG_CADENCE_WATCHDOG=m CONFIG_USB_DWC3=m CONFIG_XILINX_ZYNQMP_DMA=m CONFIG_SPI_ZYNQMP_GQSPI=m EOF |
Create a Custom Kernel SPEC File
The SPEC file is the script which controls the build of an RPM package. We will make a few edits to a copy of the default aarch64 SPEC file to support Zynq UltraScale+.
...
Code Block |
---|
Source1000: kernel-aarch64-zynqmp.cfg |
Inside the function “BuildKernel()”“BuildKernel”, add line #2 as shown below. This will merge our configuration fragment with the default kernel configuration.
Code Block | ||
---|---|---|
| ||
cp configs/$Config .config scripts/kconfig/merge_config.sh -m -r .config %{SOURCE1000} |
...
Code Block | ||
---|---|---|
| ||
$ git add SPECS/kernel-xlnx.spec SOURCES/kernel-aarch64-zynqmp.cfg $ git commit -m'added zynqmp machine' -a |
Build the Kernel RPM Packages
Code Block | ||
---|---|---|
| ||
$ cd $COS_BUILD/rpmbuild/kernel $ rpmbuild --define "%_topdir `pwd`" -bb SPECS/kernel-xlnx.spec --with cross --target aarch64 --with baseonly --without debuginfo \ --without tools --without perf --without bpftool --without selftests --nodeps --noclean |
...
Code Block |
---|
$ cp BUILDROOT/kernel-4.18.0-147.3.1.el8.xlnx.aarch64/boot/dtb-4.18.0-147.3.1.el8.xlnx.aarch64/xilinx/zynqmp-zcu102-rev1.0.dtb $COS_BUILD/deploy/efi/dtb/xilinx/ |
Code Block | ||
---|---|---|
| ||
cp RPMS/aarch64/kernel-4.18.0-147.3.1.el8.xlnx.aarch64.rpm RPMS/aarch64/kernel-core-4.18.0-147.3.1.el8.xlnx.aarch64.rpm \ RPMS/aarch64/kernel-modules-4.18.0-147.3.1.el8.xlnx.aarch64.rpm $COS_BUILD/deploy/boot/ |
Install UEFI and Kernel on a Preinstalled CentOS Disk Image
We will start with a raw disk image of a CentOS server as the basis for our system.
...
*.rpm $COS_BUILD/deploy/boot/ |
Install UEFI and Kernel on a Preinstalled CentOS Disk Image
We will install the bootloader and kernel on a prebuilt CentOS 8 Stream OpenStack cloud disk image.
Installing Centos 8 from an ISO is beyond the scope of this article. Please see Virtually Install CentOS and Fedora on Zynq UltraScale+ to install a CentOS 8 image before proceedingfrom an ISO.
Code Block | ||
---|---|---|
| ||
$ cd $COS_BUILD/images
$ wget https://cloud.centos.org/centos/8-stream/aarch64/images/CentOS-Stream-GenericCloud-8-20200113.0.aarch64.qcow2 |
Change the root password with “virt-customize” to “zynqmp” or a password of your choice.
Code Block | ||
---|---|---|
| ||
$ cd $COS_BUILD/images
# copy the qcow2 image here |
You may optionally change the password with “virt-customize”.
Code Block | ||
---|---|---|
| ||
$ virt-customize -a CentOS-8-aarch64-1905-dvd1.qcow2 --root-password password:zynqmp virt-customize -a CentOS-Stream-GenericCloud-8-20200113.0.aarch64.qcow2 --root-password password:zynqmp [ 0.0] Examining the guest ... [ 33.4] Setting a random seed [ 33.7] Setting the machine ID in /etc/machine-id [ 33.8] Setting passwords [ 51.3] Finishing off |
The installed file is a QCOW2 (QEMU copy-on-write) virtual disk, so we need to convert it to a raw disk image.
Code Block | ||
---|---|---|
| ||
$ qemu-img convert CentOS-Stream-GenericCloud-8-aarch64-1905-dvd120200113.0.aarch64.qcow2 CentOS-Stream-GenericCloud-8-aarch64-1905-dvd120200113.0.aarch64.raw |
Now lets examine the partitions in this disk image. You will see there are 3 partitions. The first is the EFI System Partition (ESP) which is a FAT formatted partition. The second partition is an EXT4 a Linux formatted partition. The third partition is an LVM root filesystem.
Code Block | ||
---|---|---|
| ||
$ fdisk -l CentOS-Stream-GenericCloud-8-aarch64-1905-dvd120200113.0.aarch64.raw Disk CentOS-Stream-GenericCloud-8-aarch64-1905-dvd120200113.0.aarch64.raw: 710 GiB, 751619276810737418240 bytes, 1468006420971520 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: gpt Disk identifier: E51C2C48079AB561-87F2A1E1-419D4340-BA13A744-687E3930A160C041B93E2E67 Device Start End Sectors Size Type CentOS-Stream-GenericCloud-8-aarch64-1905-dvd120200113.0.aarch64.raw1 2048 1230847 1228800 600M EFI System CentOS-8Stream-aarch64GenericCloud-19058-dvd1.raw2 1230848 3327999 2097152 1G Linux filesystem CentOS-8-aarch64-1905-dvd1.raw3 3328000 14678015 11350016 5.4G Linux LVM20200113.0.aarch64.raw2 1230848 17614847 16384000 7.8G Linu |
The first thing we note is that “Disklabel type” is set to “gpt”. However, Zynq UltraScale+ only support MBR partitioned disks, so we need to convert from GPT to MBR with “sgdisk”.
Code Block | ||
---|---|---|
| ||
$ sgdisk -m 1:2:3 CentOS-Stream-GenericCloud-8-aarch64-1905-dvd120200113.0.aarch64.raw Warning: The kernel is still using the old partition table. The new table will be used at the next reboot or after you run partprobe(8) or kpartx(8) GPT data structures destroyed! You may now partition the disk using fdisk or other utilities. |
We also need to mark the 1st sector as the boot partition with “sfdisk”.
Code Block |
---|
$ sfdisk -A CentOS-Stream-GenericCloud-8-aarch64-1905-dvd120200113.0.aarch64.raw 1 The bootable flag on partition 1 is enabled now. The partition table has been altered. Syncing disks. |
...
Code Block |
---|
$ fdisk -l CentOS-Stream-GenericCloud-8-aarch64-1905-dvd120200113.0.aarch64.raw Disk CentOS-Stream-GenericCloud-8-aarch64-1905-dvd120200113.0.aarch64.raw: 710 GiB, 751619276810737418240 bytes, 1468006420971520 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0x00000000 Device Boot Start End Sectors Size Id Type CentOS-8-aarch64-1905-dvd1.raw1 * Boot Start 2048 1230847 End 1228800 Sectors 600M efSize EFI (FAT-12/16/32)Id Type CentOS-Stream-GenericCloud-8-aarch64-1905-dvd1.raw220200113.0.aarch64.raw1 * 1230848 2048 3327999 1230847 2097152 1228800 600M 1Gef 83EFI Linux CentOS-Stream-GenericCloud-8-aarch64-1905-dvd1.raw320200113.0.aarch64.raw2 33280001230848 1467801517614847 1135001616384000 57.4G8G 8e83 Linux LVMLinu |
We need to setup the map devices for the raw disk image we prepared earlier. Mount the first two partitions on “efi” and “boot” respectivelythe map devices for the raw disk image we prepared earlier. Mount the two partitions on “efi” and “rootfs” respectively.
If you are installing on a raw disk image created from an ISO, then there may be three partitions, EFI, boot and rootfs. In this case you will want to mount the second partition on “boot” and copy the kernel RPM packages here.
Code Block |
---|
$ cd $COS_BUILD/images $ sudo kpartx -va CentOS-Stream-GenericCloud-8-aarch64-1905-dvd120200113.0.aarch64.raw add map loop0p1 (253:12): 0 1228800 linear 7:4 2048 add map loop0p2 (253:133): 0 20971521228800 linear 7:40 12308482048 add map loop0p3loop0p2 (253:144): 0 1135001616384000 linear 7:40 33280001230848 $ sudo mount /dev/mapper/loop0p1 efi $ sudo mount /dev/mapper/loop0p2 bootrootfs |
The Zynq UltraScale+ boot ROM will search for “boot.bin” on the first partition. The boot ROM will load FSBL and PMU firmware. FSBL will in turn load u-boot. U-boot acting as the UEFI firmware will search the current partition for the corresponding board DTB file and pass it to the EFI bootloader through the EFI configuration table. If it doesn’t find one, it will pass its own DTB to the EFI bootloader. However, this DTB may not be compatible with your kernel version, so I recommend using the DTB built with the kernel. So let's install the UEFI firmware (boot.bin) and ZCU102 DTB as shown below from the deploy directory.
...
Since we are not using UEFI secure boot, we want to will boot GRUB directly. So we need to replace shim (bootaa64.efi) with GRUB (grubaa64.efi).
Code Block | ||
---|---|---|
| ||
$ sudo cp efi/EFI/centos/grubaa64.efi efi/EFI/BOOT/bootaa64.efi |
...
Next lets install the bootstrap kernel and kernel RPM packages into the BOOT directory.
Code Block | |||
---|---|---|---|
| |||
$ sudo cp ../deploy/boot/* rootfs/boot/ |
Since we are installing on a cloud image, we need to disable the cloud-init to make it look more like a standard distro.
Code Block |
---|
$ sudo cptouch ..rootfs/deployetc/boot/* boot/cloud/cloud-init.disabled |
Skip this step if you are installing on a disk image generated from an ISO or want to keep the OpenStack initialization.
Finally, let’s unmount the partitions and cleanup the device mappings.
Code Block | ||
---|---|---|
| ||
$ sudo umount efi $ sudo umount bootrootfs $ sudo kpartx -d CentOS-Stream-GenericCloud-8-aarch64-1905-dvd120200113.0.aarch64.raw loop deleted : /dev/loop0 |
Now the raw disk image is ready to write to the SD card. With the CD SD card plugged into your CentOS machine, note the device node of your card (/dev/sdX) and install the image as shown below.
Code Block | ||
---|---|---|
| ||
$ sudo dd if=CentOS-Stream-GenericCloud-8-aarch64-1905-dvd120200113.0.aarch64.raw of=/dev/sdX bs=4M iflag=fullblock oflag=direct status=progress; sync |
Info |
---|
You may use “dmesg” to find the device node corresponding to your SD card. |
Booting ZCU102
Open a terminal emulator of your choice on your host machine and connect to the serial port on the ZCU102. Insert the SD card on the ZCU102, make sure the dip switches are set to SD boot and power on the board. If everything goes well, you should see an FSBL banner, PMU firmware version and U-boot messages.
...
Before we install the kernel, we need to ensure that the SD/MMC modules are included in the initramfs by adding it to a dracut conf configuration file as shown below. If you need any additional kernel modules for boot that are not included by default, you may add them here as well.
...
Code Block | ||
---|---|---|
| ||
# vi /etc/default/grub ## add "noefi" and any additional kernel options to GRUB_CMDLINE_LINUX andthen save # grub2-mkconfig -o /boot/efi/EFI/centos/grub.cfg |
...
Once you get to the GRUB menu, you should see that the first option is your updated kernel.
...
References
...