We provide a rebuild of the Red Hat "Extended Lifecycle Support" (ELS) product, which extends the life of EL7 to 30.06.2028
At CERN we have agreed with the ATS sector that we will provide the rebuilt RPMs for consumption until the end of Run3 (30.06.2026). More details about this agreement can be found [here](https://edms.cern.ch/ui/file/2749524/1.1/LHC-CF-EN-0001_1_1.pdf)
A recent [presentation](https://indico.cern.ch/event/1477299/contributions/6363918/) from HEPiX Spring 2025 can be consulted for even more detail
## ELS7 cronjob
The entire process is automated via a bash [script](https://gitlab.cern.ch/linuxsupport/cronjobs/els7_release) running via nomad on a daily basis
Manual intervention is not required, unless a fault is detected in which case administrators are emailed
The script will:
* Compare source package lists between RHEL7 content and ELS content (thus determining which source packages are from ELS7)
* Checks for packages already present in the `els7-stable` tag to avoid sending duplicate builds to koji
* Builds missing packages via koji
* Tags successfully built packages to the `els7-stable` tag
* Sends informational emails to administrators about build results (both successes and fails)
* Updates the [linux.cern.ch](https://linux.web.cern.ch/updates/els7/latest_updates/) website with new ELS7 package (including errata and CVE information)
## Additional features
Apart from building ELS7 source RPMs in an automated fashion, the script also supports the following items which may not be obvious:
### Building `firefox` for EL7
*`firefox` requires a recent version of [`rust`](https://gitlab.cern.ch/linuxsupport/rpms/rust) which is added as a koji external repo
* Newer versions of `clang`, `llvm` are also required.
To support these requirements the els7 script modifies the `firefox.spec` to utilise the "llvm-toolkit-14" Software Collection, then creates a new source rpm and utilises this source rpm for the koji build.
### Building multilib packages
The script supports rebuilding 32-bit multilib packages. The logic to support this is as follows:
1. The spec file is first inspected with `rpmspec` to generate a list of sub packages defined
2. The above list is looped and each item is checked against the existing RHEL7 package for an `i686` architecture. If there is a hit, this package was provided as multilib in RHEL7 and should also be provided as a multilib in ELS7
3. The source rpm is sent to koji as a `--scratch` build on the `els7_9_i386` tag. We send as `--scratch` as there will already be a NVR from the `x86_64` build, and we can't just add `i686` to the architecture list of `els7_9` as Red Hat does not include ALL `i686` sub packages
4. The binary packages from the `--scratch` build are downloaded
5. The `i686` packages that should be included (as determined in point #2) are then imported (`koji import`) into the `x86_64` NVR
6. To support dist-repo multilib we are applying this [patch](https://pagure.io/koji/pull-request/4293)(not yet merged)
## Operations
Usually the script does not need any maintenance, unless a package fails to build in koji. In this case an email is sent to administrators including a link to the koji task which will need to be consulted to determine the reason for the failure. In the past this has mostly been related to missing dependencies, though now the addition of new external repos to the koji tag should not be necessary anymore.
In the event of a package build failure that cannot be fixed, a [skip list](https://gitlab.cern.ch/linuxsupport/cronjobs/els7_release/-/blob/master/prod.variables.sh?ref_type=heads#L7) is supported to allow a package to be excluded from the build candidate logic
The [presentation](https://indico.cern.ch/event/1477299/contributions/6363918/) from HEPiX Spring 2025 provides some history of issues encountered in the past and how they were fixed
## Dependencies
This script assumes that the following items are functional:
* reposync for [ELS7](https://gitlab.cern.ch/linuxsupport/cronjobs/reposync/-/blob/master/prod.repos.d/redhat-7-els-x86_64.repo?ref_type=heads)
* advisories for [ELS7](https://gitlab.cern.ch/linuxsupport/cronjobs/advisories/-/blob/master/prod.advisories.yaml?ref_type=heads#L60-65)
## ELS7 tag configuration
The tag configuration for koji can be found [here](https://gitlab.cern.ch/linuxsupport/lxdist-build/-/blob/master/bin/tags/els7.sh?ref_type=heads)
In total 4 tags are defined in koji for the purposes of building ELS7 packages
The disttag is determined from the source rpm. Whilst most packages will be `.el7_9`, there may be some that are only `.el7`
| tag | disttag | buildsys_package | external repos (in order of priority) |
Some of the above external repos may not be completely intuitive, thus the outliers are documented here:
*[centos7-dependency-fixes-i386](https://linuxsoft.cern.ch/enterprise/centos7-i386/): Contains recent versions of `tzdata` to avoid missing package dependencies with base/updates within the `centos7-base-i386` / `centos7-updates-i386` external repos. These mismatches were seen most recently when attempting to build `java-1.8.0-openjdk` which required `tzdata >= 2024` and was not "seen" by koji as the version of `tzdata` from base was taken as precedence
*[centos7-base-i386](https://linuxsoft.cern.ch/centos-vault/altarch/7/os/i386/): Base from "CentOS 7 alternative architecture (i686)" (required for multilib package builds on _i386 tags)
*[centos7-updates-i386](https://linuxsoft.cern.ch/centos-vault/altarch/7/updates/i386/): Updates from "CentOS 7 alternative architecture (i686)" (required for multilib package builds on _i386 tags)
*[linuxsupport7-stable](https://linuxsoft.cern.ch/internal/repos/linuxsupport7-stable): includes a EL7 `rust`[build](https://gitlab.cern.ch/linuxsupport/rpms/rust)(required to build firefox on EL7)
*[buildsys7](http://linuxsoft.cern.ch/internal/buildsys/7/): includes the `buildsys-macros-el7` and `buildsys-macros-el7_9` disttag macro RPMs
For ELS7 builds we disable the %check stage during the build. We do this for several reasons:
* Some upstream packages require custom rpm macros to be set in order for the tests to pass
* As we are only rebuilding packages, we don't expect that the rpm tests will fail
* We can speed up the koji build system by disabling the checks
As there is not currently a nice way to instruct rpmbuild to disable checks, we can simply exit at the [%check stage](https://gitlab.cern.ch/linuxsupport/lxdist-build/-/blob/master/bin/tags/els7.sh?ref_type=heads#L62)
## Miscellaneous details
* As multilib packages are built as scratch builds and then the binary RPM is imported into koji, the [koji-hub-plugin-cern](https://gitlab.cern.ch/linuxsupport/rpms/koji-hub-plugins-cern) plugin was updated to perform package signing on imports
* A [cern-els7-release](https://gitlab.cern.ch/linuxsupport/rpms/releases/cern-els7-release) is provided in the [CC7/cern](https://linuxsoft.cern.ch/cern/centos/7/cern/x86_64) repository to enable users to easily configure the CERN ELS7 repos on a CC7 system (`yum install cern-els7-release`)
