The ZynqUltraScale+ is a powerful SoC platform combining multi-core ARM64 CPUs and FPGA technology. There are plenty of projects that can take advantage of running a Linux kernel on the ARM64 processor and sometimes they require some degree of kernel hacking. In this post we will setup a build environment to modify the Linux kernel, compile it, and run it rapidly on a ZCU106 development board.
For this we want the following :
- Be able to modify and compile the Linux kernel easily
- Boot the kernel on the ZCU106 board rapidly
- Have a fast “code-build-test” development cycle
And to avoid development hell we want to avoid things such as :
- Having to prepare patches and apply them manually to yocto recipes
- Rebuild the whole kernel for small changes (we want incremental build)
- Programming the SoC through JTAG
- Or writing SD cards
The Linux build environment
We want to run a Linux kernel on AMD/Xilinx ZynqUS+ platforms, a kernel that we can modify. For this AMD/Xilinx provides a build tool named Petalinux, this allows to rapidly generate everything required to run Linux on AMD/Xilinx hardware (Zynq, ZynqUS+, Microblaze, and Versal platforms), namely the different boot loaders (first stage boot loader (FSBL), U-Boot), a Linux kernel, and a root file system (rootfs) with applications.
The Petalinux tools are built upon Yocto a popular project that produces tools and processes that enable the creation of Linux distributions for embedded and IoT devices. This allows for a great deal of customisation but also comes with a lot of (hidden) complexity. Yocto itself would allow to build any kernel but the ZynqUS+ family of devices are not very well supported by the mainline Linux kernels, AMD/Xilinx provides it’s own fork of the Linux kernel that is used in Petalinux. The most recent stable supported version as of 2023 is Linux kernel 6.1. This fork provides support and drivers for AMD/Xilinx devices that are not yet in the mainline Linux kernels.
It is possible to build mainline Linux kernels for AMD/Xilinx devices and for the Zynq 7000 series there is good support, AMD/Xilinx have some documentation on this here. But for the ZynqUS+ family of SoCs the support seems limited as for now and the effort to build a mainline Linux kernel for them seems quite substantial, and this would leave out the drivers and support provided by linux-xlnx. So here we will base ourselves on Petalinux and linux-xlnx kernels.
Creating the Petalinux Project
For this step we assume that :
- A hardware project (in Vivado) and platform (XSA) for the ZCU106 have been created and exported (if not see here is an example, for a ZCU104 board, but the steps are similar)
- Petalinux 2023.1 is installed (if not see here) for documentation refer to the Petalinux Tools Documentation (UG1144)
We’ll start by creating a new project for our board :
# Source the settings for Petalinux
source <path-to-petalinux-install>/xilinx/PetaLinux/2023.1/tool/settings.sh
# Create a new project based on ZynqMP template
petalinux-create -t project -n <project-name> --template zynqMP
# Go in the project directory
cd <project-name>
# Configure the project based on the hardware description file (XSA)
petalinux-config --get-hw-description=<path-to-xsa>/<zcu106-xsa-name>.xsa
This will open a configuration menu
In this menu we will for the moment only set the “Machine Name” for our ZCU106 board under “DTG Settings” so that the device tree generator can generate the main device tree for us. As documented here in UG1144.
Once set, save and exit the menu.
Getting the linux-xlnx source code
There are multiple ways of modifying the Linux kernel built by Petalinux, one way is by adding patches to the Yocto layer responsible to build the kernel as described in this blog post on fpgadeveloper.com. This is a rapid way to add existing patches, however, we would prefer to be able to directly modify the source code and try things out without having to make patches for each test.
Petalinux actually provides a workflow for this, it is described here in UG1144. This workflow will fetch the linux-xlnx sources and we can directly work with a “code-build-test” development flow.
# Launch the devtool command from the Petalinux project
petalinux-devtool modify linux-xlnx
This takes some time, once done it will have generated a directory in <project-name>/components/yocto/workspace/sources/linux-xlnx
Petalinux will now build the kernel from that directory, so if we modify it there, it will build with our changes. That directory is a git repository, this will allow us do source control and versioning which is very useful. We can check the log for example with git log.
Modify the linux-xlnx source code
Let’s open that directory and make a simple change to check if we can actually do some kernel hacking, build, and test our changes.
Let’s begin with the iconic and simple “Hello World” example. We chose to add it in smp_setup_processor_id() in arch/arm64/kernel/setup.c since this is the very first message printed by our kernel on this architecture. We can now build and test.
Building everything
# Launch the build
petalinux-build
Since this is our very first build, it will take some time (because Petalinux has to build the other components as well, the FSBL, U-Boot, the RootFS, etc. and the kernel itself).
First test for our kernel
For this first test we will boot everything through JTAG, with the JTAG probe connected, the board in JTAG mode (boot switches – SW6), and Serial port connected through USB. For more information see the documentation for the ZCU106 board.
The Petalinux command to boot through JTAG is the following (documentation) :
# The hw_server is launched on the same machine so local host
# IP 127.0.0.1 is used with the default port 3121
petalinux-boot --jtag --kernel --hw_server-url 127.0.0.1:3121
We launched the Petalinux command to boot through JTAG, this will take some time as the default bit rate for JTAG isn’t that high and we load everything. We could also boot through a SD card, more info on this in the documentation. Anyways we will setup a network boot below, to speed things up, so this doesn’t really matter.
We will also open the serial port to communicate with our board (here we use picocom) in another terminal.
After waiting some time we can finally see our kernel booting and our “Hello World” message displayed !
Preparing the ZCU106 board for network boot
For our first test we booted through JTAG which took a very long time. This is not perfect for a rapid development cycle. Another option would be to write images to a SD card, but this is cumbersome as well. So for a more rapid development cycle we will boot U-Boot from the QSPI (the SD card would be another possibility) and from U-Boot use the network to load the kernel to the board and boot it. The different boot methods are documented here in UG1144.
First, we’ll change the autoboot setting in U-Boot so that we remain in U-Boot and don’t automatically boot Linux. Open the U-Boot configuration menu with the following command :
Petalinux-config -c u-boot
Under “Boot options -> Autoboot options” uncheck “Autoboot”, save and exit.
Rebuilt U-Boot by running one of the following commands
petalinux-build # Build everything
petalinux-build -c u-boot # Build U-Boot only
Once rebuilt, we will then use the package option to generate images that we will be able to flash to the QSPI. For more information see documentation. Petalinux will copy these to the /tftpboot folder that we will use to transfer them to the board over TFTP. For more information on TFTP setup see documentation. A good explanation on how to setup TFTP on Ubuntu is available here.
# Generate a boot image, with First Stage Boot Loader
# FPGA bitstream, PMU Firmware, and U-Boot
# --force allows to overwrite files on disk if present
petalinux-package --boot --fsbl --fpga --pmufw --u-boot --force
One last time we will boot again from JTAG, but this time we will boot U-Boot only and not load the kernel and RootFS, this will save some time.
On the serial console we can see that U-Boot has launched and stopped at the prompt, because we unchecked “Autoboot”.
Flashing U-Boot to the QSPI
Now that we have U-Boot running we can use it to load the images from the host computer over TFTP (note that the board requires to be connected to the network with an ethernet cable). The procedure to flash the QSPI is described in the documentation, we will however flash only U-Boot here to the QSPI since we will launch the Linux kernel over ethernet, because we want to test it rapidly, without having to flash anything.
So first, we need to detect the flash and erase it, this can be done in U-Boot with the following commands :
# Detect the flash
sf probe 0 0 0
# Erase the flash
sf erase 0 <Flash size in hex>
Then, we will transfer the boot image (with FSBL, Bitstream, PMU Firmware and U-Boot) over TFTP :
# Set the IP over the host machine (can be found with "ifconfig" on host)
setenv serverip <IP of host computer>
# Set an IP for the board (here chosen arbitrarily)
setenv ipaddr <IP for board>
# Transfer the packaged BOOT.BIN file
tftpboot 0x80000 BOOT.BIN
# Write the file to flash
sf write 0x80000 0x0 <size in hex>
Everything we need to launch U-Boot is now flashed to the QSPI ! From here we will not need to boot over JTAG anymore, we will be able to boot from the QSPI and transfer the kernel over ethernet.
Booting from QSPI and launching the Linux kernel over ethernet with PXE
We can now turn off the ZCU106 development board, and set the boot switches to QSPI mode.
When we turn the power back on, we can see that U-Boot directly loads from the QSPI, this is very fast and no operation is needed on the host computer.
We will now load the Linux kernel through PXE boot. For further information refer to the documentation. In U-Boot we will launch the following commands :
# Set the IP over the host machine (can be found with "ifconfig" on host)
setenv serverip <IP of host computer>
# Set an IP for the board (here chosen arbitrarily)
setenv ipaddr <IP for board>
# Get files through PXE
pxe get
# Boot
pxe boot
Or as a single command (that we can just prepare once and copy paste in U-Boot each time needed) :
setenv serverip <IP of host computer> && setenv ipaddr <IP for board> && pxe get && pxe boot
We could even set it up as a U-Boot environment variable if needed and prepare and autoboot command, I did not do this because we don’t have fixed IP addresses and so it is simpler for us this way, than to create U-Boot environment variables and save them to flash, but the possibility is there (example here).
PXE Boot will look for several file names on the host computer until it finds one that fits, then it will automatically download the Linux kernel image, RootFS, and device tree blob and start the kernel as described in the PXE boot file (pxelinux.cfg generated by Petalinux and put in /tftpboot).
A short code-build-test cycle
We now have a faster “code-build-test” cycle to do some further kernel hacking. We can modify the source directly, build incrementally with Petalinux (petalinux-build or petalinux-build -c kernel), and test rapidly by loading the kernel over Ethernet as shown above.
Save changes to a patch and close the development cycle
Now that we are satisfied with our changes and are done with hacking the kernel for the moment we can save them to a patch, add it to our kernel recipe, and close the development cycle. More info in documentation.
Go in the linux-xlnx directory, add the changes and commit them.
# Go to the source directory
cd components/yocto/workspace/sources/linux-xlnx/
# Set the name and email for the git config (only needed once)
git config user.name "Your name"
git config user.email "Your email"
# Add the modified files
git add arch/arm64/kernel/setup.c
# Commit the changes
git commit -m "Added \"Hello World\" message to kernel"
# Go back
cd ../../../../..
Finally use the petalinux-devtool finish command (for some reason it does not accept a relative path) :
petalinux-devtool finish linux-xlnx <absolute path>/project-spec/meta-user
The linux-xlnx directory has now been deleted from components/yocto/sources and a patch has been generated in the Yocto layer recipe for the kernel.
This patch can now be easily shared and applied to other Petalinux projects.
Conclusion
In this article we :
- Learned how to modify the linux-xlnx kernel built with Petalinux
- Flashed U-Boot in QSPI to boot it rapidly
- Loaded a Linux kernel through ethernet with PXE
- Setup a rapid “code-build-test” development cycle
- Transformed our changes to a patch that could be easily applied and shared
This is was all very useful for me so I shared this here for future reference and for others to follow. Happy Hacking !
References
- Petalinux : https://www.xilinx.com/products/design-tools/embedded-software/petalinux-sdk.html#tools
- Petalinux Documentation : https://docs.xilinx.com/r/en-US/ug1144-petalinux-tools-reference-guide
- Xilinx Linux fork : https://github.com/Xilinx/linux-xlnx
- Yocto : https://www.yoctoproject.org/
- ZCU106 Documentation : https://www.xilinx.com/products/boards-and-kits/zcu106.html#documentation
- TFP on Ubuntu : https://help.ubuntu.com/community/TFTP
- Patching Petalinux : https://www.fpgadeveloper.com/how-to-patch-petalinux/
- Edit U-Boot Environment Variables : https://www.fpgadeveloper.com/how-to-modify-u-boot-environment-variables-in-petalinux/
Note : This article is based on Petalinux tools v2023.1