Karol Krizka requested to merge kk_hog into main Jul 21, 2021

Xilinx 2021.1

The firmware now depends on Vivado/Petalinux 2021.1.

Advantages:

Supports CentOS8, which provides a newer version of Git required by Hog.
Fix of disappearing UART when booting Petalinux on ZTurn.

Disadvantages:

Network cable needs to be physically re-plugged on boot before an IP address is obtained.

Hog

Support for multiple SoC boards is done by moving the build management to Hog. The different boards are different Hog projects. The following projects are available:

zturn_7020: For use with ZTurn 7z020 and multiplexed communication (Active Board 20210504)
microzed_7020.1: For use with MicroZed 7z020 and direct communication (Active Board 20190718)
microzed_7020.2: For use with MicroZed 7z020 and multiplexed communication (Active Board 20210504)

Block Diagram Generation

The SoC Xilinx workflow requires the generation of a BD (Block Design) file to act at the top-level. The BF format, while test-based, is computer generated and thus hard to track inside a file itself is computer generated, which can be hard to version control. The approach here is to generate the BD file during project creation using Tcl.

The function to populate an active board BD, create_root_design {parentCell numDirect numDaisyChain} is defined in bd/ActiveBoard.tcl. It is called inside post-creation.tcl for all projects, along with the creation of the VHDL wrapper entity.

The downside is that the BD file is not officially registered with Hog. This means that Hog will complain as follows:

/builds/berkeleylab/pbv3_test_adapter/firmware/Projects/microzed_7020.1/microzed_7020.1.srcs/sources_1/bd/ActiveBoard/ActiveBoard.bd is used in the project but is not in the list files.

Custom IP Generation (Endeavour)

All blocks inside a BD file must be packaged as IP. This includes the custom entity used for the AMAC communication using the endeavour protocol. The IP containing the endeavour block is the only custom IP in this project.

Custom IP's are implemented as invidual projects (ie: Top/endeavour_ip/) targeting a xc7z020clg400-1 FPGA on a "random" board. The IP project includes a post-creation.tcl script that packages the IP. The actual code doing the packaging is located at tcl/build_ip_endeavour.tcl. The package IP is stored under $globalSettings::user_ip_repo/endeavour.

The IP project has to be created using Hog/CreateProject.sh endeavour_ip before any firmware is build. It also has to be recreated anytime the IP source code changes. The Hog/LaunchWorkflow.sh does not do anything useful. In CI, the IP is packaged at the user_pre stage.

Hardware Description

Petalinux requires the hardware description of the firmware. This is useful for autoconfiguring drivers for official IP, like I2C/SPI blocks. The post-bitstream.tcl automatically exports an XSA file at the end of every build. It is then added to other firmware artifacts under bin with the name project-version.xsa.

Petalinux Organization

Configuration

The configuration of the Petalinux project is stored inside the petalinux directory. The default setup targets reception system (SD card boot) with multiplexed Powerboard communication. Manual changes have to be done to target NFS boot and direct Powerboard communication (Active Board 20190718). See gitlab CI file to see what they are.

The multiplexed vs direct firmware device trees are stored under petalinux/project-spec/meta-user/recipes-bsp/device-tree/files as system-workers.dtsi and system-direct.dtsi. The system-user.dtsi should be a link to the one corresponding to the firmware being compiled.

Variables in petalinux/project-spec/configs/config can be updated using the Pog/SetPetalinuxConfig.sh script. Examples are as follows. The first line unsets CONFIG_SUBSYSTEM_ROOTFS_INITRD and the second one changes CONFIG_SUBSYSTEM_ROOTFS_NFS to y.

./Pog/SetPetalinuxConfig.sh CONFIG_SUBSYSTEM_ROOTFS_INITRD
./Pog/SetPetalinuxConfig.sh CONFIG_SUBSYSTEM_ROOTFS_NFS y

Building

Run ./Pog/BuildPetalinux.sh after compiling firmware (and setting up the petalinux tools). This loops over all firmwares stored inside the bin/ directory and:

Reconfigures the Petalinux project using the firmware's hardware description.
Builds the Petalinux project.
Packages the Petalinux project (including firmware).
Copies results (BOOT.bin, image.ub, boot.scr and rootfs.tar.gz) into the firmware result directory under bin/.

Continuous Integration

The CI uses the Hog-defined .gitlab-ci.yml. The departures from the standard flow are:

user_pre stage to package endeavour_ip. The resulting IP_repository is stored as an artifact.
user_proj stage to build Petalinux

Petalinux CI Details

The Petalinux build uses the artifacts from the generation_and_simulation stage. Multiple firmware projects, sharing the same addendum to the device tree, are fed to a single Petalinux build job. This results in two build jobs: multiplexed (zturn_7020, microzed_7020.2) and direct (microzed_7020.1).

Each build job handles two Petalinux configurations: SD card and NFS boot. The artifacts of each are stored under correspondingly named directories in each firmware's artifacts directory.

The build jobs are not parallelized further (separate jobs per firmware X boot configuration) to reduce the resource usage. Most of the Petalinux build is taken up compiling standard Linux software (ie: bash, ssh) that does not depend on the actual firmware or device tree. Bundling everything into one job reduces the amount of times the common software is compiled. However the Petalinux build jobs have to be split if a specific Petalinux configuration depends on the firmware (direct and multiplexed variants). There is currently no mechanism that configures Petalinux based on the Vivado project.

The whole process is defined inside a .base:petalinux template in the top-level .gitlab-ci.yml. An example Petalinux job when looks as:

petalinux:direct:
  extends: .base:petalinux
  needs: ["generate_project:microzed_7020.1"]
  before_script:
    - pushd petalinux/project-spec/meta-user/recipes-bsp/device-tree/files
    - rm system-user.dtsi
    - ln -s system-direct.dtsi system-user.dtsi
    - popd

Docker

Everything is build inside Docker containers. There are two separate images; containing Vivado and Petalinux. The former builds the firmware and latter the Petalinux projects.

The Vivado image can only run on dedicated runners tagged with hog. This is because it is quite large and CI jobs will timeout while downloading it. The dedicated runners (machines at LBL) have the image already downloaded. Other than that, there is no reason why general runners with Docker support cannot be used.

The Petalinux image is much lightweight and is easily downloaded on general runners. This repository uses CERN's GitLab runners to build this stage.

Future Improvements

There are a few improvements that could be done to the flow.

Pass firmware version to Petalinux project configuration.
Generalization of making variations on a Petalinux project.
Cache Petalinux builds between jobs to reduce the number of times bash has to be rebuild.

Edited Aug 03, 2021 by Karol Krizka

Add support for ZTurn board