Skip to content
Snippets Groups Projects

Revisiting the "release test boot media + release to prod" with the 7.9 release

Merged Revisiting the "release test boot media + release to prod" with the 7.9 release
release_test_boot_media into master
All threads resolved!
Merged Ghost User requested to merge release_test_boot_media into master
All threads resolved!
Compare
and
1 file
+ 130
− 49
Compare changes
  • Side-by-side
  • Inline
docs/distributions/cc7.md
+ 130
− 49
@@ -323,44 +323,105 @@ Ben Morrice for Linux.Support@cern.ch
This stage involves working on a new tree from upstream (QA). We add CERN specific packages to an anacondacern repository which is referred to via custom anaconda additions on a boot.iso which is also authored by us.
* `mkdir -p /mnt/data2/dist/cern/centos/7.$release/os/x86_64/`
* `cp -a /mnt/data2/dist/cern/centos/7.$previousrelease/os/x86_64/CERN /mnt/data2/dist/cern/centos/7.$release/os/x86_64/`
* `cp /mnt/data2/dist/cern/centos/7.$previousrelease/os/x86_64/RPM-GPG-KEY-cern /mnt/data2/dist/cern/centos/7.$release/os/x86_64/`
* Note: the scripts contained in the CERN directory can also be found in git here: [https://gitlab.cern.ch/linuxsupport/lxdist-build/tree/master/newrelease](https://gitlab.cern.ch/linuxsupport/lxdist-build/tree/master/newrelease)
* Edit `/mnt/data2/dist/cern/centos/7.$release/os/x86_64/CERN/build/syncfromcentosqa.sh` to the correct $release and execute the script to download a local tree to data2
* Build centos-release for newer version
* Install the previous src rpm (/mnt/data1/dist/cern/centos/7/cern/Source/SPackages/)
* Extract the upstream rpm and compare it's contents with the tarball from the cern src rpm (usr/lib/systemd may change slightly)
* Rebuild the source tar if anything needs to be added
* Edit `~/rpmbuild/SPECS/centos-release-cern.spec` and change release numbers, changelog
* Build the new package with `rpmbuild -ba centos-release-cern.spec`
You can check whether the ostree has been already made public:
```
PREVIOUS_RELEASE="8"
RELEASE="9"
YEARMONTH="2009"
rsync -4 --list-only rsync://qa.centos.org/c5beta-qa/CentOS/7.$RELEASE.$YEARMONTH/
```
Once it is there (showing some contents):
* `mkdir -p /mnt/data2/dist/cern/centos/7.$RELEASE/os/x86_64/`
* `cp -a /mnt/data2/dist/cern/centos/7.$PREVIOUS_RELEASE/os/x86_64/CERN /mnt/data2/dist/cern/centos/7.$RELEASE/os/x86_64/`
* `cp /mnt/data2/dist/cern/centos/7.$PREVIOUS_RELEASE/os/x86_64/RPM-GPG-KEY-cern /mnt/data2/dist/cern/centos/7.$RELEASE/os/x86_64/`
* Note: the scripts contained in the CERN directory can also be found in git here: <https://gitlab.cern.ch/linuxsupport/lxdist-build/tree/master/newrelease>
* Edit `/mnt/data2/dist/cern/centos/7.$RELEASE/os/x86_64/CERN/build/syncfromcentosqa.sh` to the correct $RELEASE and execute the script to download a local tree to data2
* Build a new `centos-release` package for the new released version:
* Install the previous src rpm (You can find it `/mnt/data1/dist/cern/centos/7/cern/Source/SPackages/`)
```
rpm -i /mnt/data1/dist/cern/centos/7/cern/Source/SPackages/centos-release-7-$PREVIOUS_RELEASE.$YEARMONTH.0.el7.cern.src.rpm
```
* Extract the upstream rpm and compare it's contents with the tarball from the cern src rpm (`usr/lib/systemd` may change slightly, e.g. `90-default.preset` or `85-display-manager.preset`)
```
rpm2cpio centos-release-7-$RELEASE.$YEARMONTH.0.el7.centos.x86_64.rpm | cpio -idmv
```
* Rebuild the source tar if anything changed, and thus needs to be added
* Edit `~/rpmbuild/SPECS/centos-release-cern.spec` and update the release numbers, the changelog:
e.g.
* Build the new package:
```
rpmbuild -ba centos-release-cern.spec
```
* Upload it to koji as well:
```
koji build cc7-cern centos79release/centos-release-7-$RELEASE.$YEARMONTH.0.el7.cern.src.rpm
```
* Sign centos-release with `rpm --define '_gpg_name 1D1E034B' --define '_signature gpg' --addsign centos-release7-$release.0.el7.cern.x86_64.rpm`
* Move centos-release to /mnt/data2/dist/cern/centos/7.$release/os/x86_64/CERN/Packages
```
# Get the signing passphrase from:
tbag show distribution_signing_key_passphrase --hg lxsoft/adm
```
* Move centos-release to `/mnt/data2/dist/cern/centos/7.$RELEASE/os/x86_64/CERN/Packages`
* Update CERN/Packages to newer versions if needed
* You can use the `/mnt/data2/dist/cern/centos/7.$release/os/x86_64/CERN/build/generate_list_CERN_rpms` to display the packages that should be updated
* You can use the `/mnt/data2/dist/cern/centos/7.$RELEASE/os/x86_64/CERN/build/generate_list_CERN_rpms` to display the packages that should be updated
* Note: We only keep one version in the cernanaconda repository, delete all older versions
* Ensure comps.xml is current (cp /mnt/data1/dist/cern/centos/7/cern/x86_64/comps.xml to /mnt/data2/dist/cern/centos/7.$release/os/x86_64/CERN/build/CERN-comps.xml)
* Generate repodata for cernanaconda by editing and executing the `/mnt/data2/dist/cern/centos/7.$release/os/x86_64/CERN/build/makerepo.sh`. Note - if a 'repodata' directory existed from a previous copy, remove it first to start clean.
* Check that generated content is correct - edit and then run `/mnt/data2/dist/cern/centos/7.$release/os/x86_64/CERN/build/check.sh`
* rsync /mnt/data2/dist/cern/centos/7.$release/os/x86_64/ to an accessible location on /mnt/data1/dist/tmp/ (this correlates to [http://linuxsoft.cern.ch/tmp/](http://linuxsoft.cern.ch/tmp/))
* Generate CERN version of boot.iso
* Download upstream boot.iso, mount to a temporary location and copy the contents to /mnt/data2/dist/cern/centos/7.$release/os/x86_64/CERN/build/boot.iso.org
* `unmount` temporary boot.iso mount
* `unsquash` /mnt/data2/dist/cern/centos/7.$release/os/x86_64/CERN/build/boot.iso.org/LiveOS/squashfs.img and then mount rootfs.img to a temporary location
* If required, rebase the files in /mnt/data2/dist/cern/centos/7.$release/os/x86_64/CERN/build/updates.img/run/install/ off the versions contained in rootfs.img
* Ensure comps.xml is up to date (`cp /mnt/data1/dist/cern/centos/7/cern/x86_64/comps.xml /mnt/data2/dist/cern/centos/7.$RELEASE/os/x86_64/CERN/build/CERN-comps.xml`)
* Generate repodata for cernanaconda by editing and executing the `/mnt/data2/dist/cern/centos/7.$RELEASE/os/x86_64/CERN/build/makerepo.sh`. Note - if a 'repodata' directory existed from a previous copy, remove it first to start clean.
* Check that generated content is correct - edit and then run `/mnt/data2/dist/cern/centos/7.$RELEASE/os/x86_64/CERN/build/checks.sh`
* rsync `/mnt/data2/dist/cern/centos/7.$RELEASE/os/x86_64/` to an accessible location on `/mnt/data1/dist/tmp/` (this correlates to [http://linuxsoft.cern.ch/tmp/](http://linuxsoft.cern.ch/tmp/)):
* Generate the CERN version of boot.iso:
* Download the upstream boot.iso, mount to a temporary location and copy the contents to `/mnt/data2/dist/cern/centos/7.$RELEASE/os/x86_64/CERN/build/boot.iso.org`:
```
mkdir /mnt/iso79
mount -o loop /mnt/data2/dist/cern/centos/7.$RELEASE.$YEARMONTH/os/x86_64/images/boot.iso /mnt/iso79/
cp /mnt/iso79 /mnt/data2/dist/cern/centos/7.$RELEASE.$YEARMONTH/os/x86_64/CERN/build/boot.iso.org
```
* `unmount` temporary boot.iso mount:
```
umount /mnt/iso79
```
* `unsquash` /mnt/data2/dist/cern/centos/7.$RELEASE/os/x86_64/CERN/build/boot.iso.org/LiveOS/squashfs.img and then mount rootfs.img to a temporary location
```
unsquashfs /mnt/data2/dist/cern/centos/7.$RELEASE.$YEARMONTH/os/x86_64/CERN/build/boot.iso.org/LiveOS/squashfs.img
mkdir /tmp/iso97rootfs
mount /mnt/data2/dist/cern/centos/7.$RELEASE.$YEARMONTH/os/x86_64/CERN/build/boot.iso.org/LiveOS/squashfs-root/LiveOS/rootfs.img /tmp/iso97rootfs
```
* If required, rebase the files in /mnt/data2/dist/cern/centos/7.$RELEASE/os/x86_64/CERN/build/updates.img/run/install/ off the versions contained in rootfs.img:
```
# For an example of what was done from 7.8 to 7.9:
diff -u /mnt/data2/dist/cern/centos/7.9/os/x86_64/CERN/build/updates.img/run/install/updates/pyanaconda/packaging/yumpayload.py.{78,79}
```
* `unmount` temporary roofs.img mount
* Edit and execute `/mnt/data2/dist/cern/centos/7.$release/os/x86_64/CERN/build/makeimages.sh`
```
umount /mnt/iso79rootfs
```
* Edit and execute `/mnt/data2/dist/cern/centos/7.$RELEASE/os/x86_64/CERN/build/makeimages.sh`
* Test boot.iso
* `dd` the boot.iso file to a usb stick and test that it can be booted
* `dd` the boot.iso file to a USB stick and test that it can be booted (physically, on a desktop, for instance). You could also test the ISO on any virtualization software such as VirtualBox.
* Boot a VM with the iso and go through the complete installation process using the path you populated before as the install media location ([http://linuxsoft.cern.ch/tmp/](http://linuxsoft.cern.ch/tmp/))
```
# You'll need to upload it to OpenStack:
openstack image create --file boot.iso --disk-format raw C7$RELEASE$YEARMONTH
```
* Confirm that there are no errors during installation and that the CERN addon is loaded on system boot
* Remove the temporary location you created at /mnt/data1/dist/tmp
* Edit and run `/mnt/data2/dist/cern/centos/7.$release/os/x86_64/CERN/build/sync.sh` to populate /mnt/data1/dist/cern/centos/7.$release/os
* Edit and run `/mnt/data2/dist/cern/centos/7.$RELEASE/os/x86_64/CERN/build/sync.sh` to populate /mnt/data1/dist/cern/centos/7.$RELEASE/os
NOTE: Make sure the sync command contains `x86_64`, as opposed to `SOURCES`, which is done at a later step.
* Create a TEST glance image through [koji-image-build](https://gitlab.cern.ch/linuxsupport/koji-image-build)
* You will need to update the cc7-base.ks and buildimage.sh with reference to the new $release
* You will need to update the cc7-base.ks and buildimage.sh with reference to the new $RELEASE
* Run the [monthly scheduled pipeline](https://gitlab.cern.ch/linuxsupport/koji-image-build/-/pipeline_schedules) of koji-image-build, which will build and test (using [image-ci](https://gitlab.cern.ch/linuxsupport/testing/image-ci/)) new images for CC7 and C8 (which you can ignore). If the tests go well, the new images will be uploaded as TEST.
* Add .htaccess file in /mnt/data1/dist/cern/centos/7.$release to prohibit consuming until CentOS QA content is released publically
* Edit and execute `/mnt/data2/dist/cern/centos/7.$release/os/x86_64/CERN/build/aims2.sh` to configure a target on aimstest and a TEST target on aims
* Add an `.htaccess` file in /mnt/data1/dist/cern/centos/7.$RELEASE to forbid consuming until CentOS QA content is released publicly:
```
cat /mnt/data1/dist/cern/centos/7.$RELEASE/.htaccess
Deny from all
```
* Edit and execute `/mnt/data2/dist/cern/centos/7.$RELEASE/os/x86_64/CERN/build/aims2.sh` to configure a target on aimstest and a TEST target on aims
* Test the PXE install process on both `aimstest` and `aims`
* Upload the image to AIMS with `aims2client`, accessible on `aiadm`. i.e. `aims2client addimg ...` and `aims2client --testserver addimg ...`
* On the `"IT Linux Support - Test VMs"` create a VM with `landb-os=LINUX` as metadata and https://gitlab.cern.ch/linuxsupport/aims2-ipxe as image. You will already find uploaded images though so reuse one of them.
@@ -369,35 +430,55 @@ This stage involves working on a new tree from upstream (QA). We add CERN specif
* Open a VNC console on the web browser by going to the "Console" tab on the given test VM
* Go through the manual installation configuration. You will need to select "Reclaim space" on the partitioning configuration.
* If at any point you need to re-run the installation, rebuild the same VM with the same `iPXE-YYYYMMDD-aimsprod` image as before, then reboot and start again, it only takes a minute.
* Update aims2-loaders to reference the new $release (see [https://gitlab.cern.ch/linuxsupport/aims2-loaders/commit/30de29e5cbd24b12379aaab098ed81d16340dded](https://gitlab.cern.ch/linuxsupport/aims2-loaders/commit/30de29e5cbd24b12379aaab098ed81d16340dded) for an example) and install on both aims and aimstest hosts
* Update aims2-loaders to reference the new $RELEASE (see <https://gitlab.cern.ch/linuxsupport/rpms/aims2-loaders/-/commit/bafa20aa91f6606d41bb9e98d7bda9bd913f573e> for an example) and install on both aims.cern.ch and aimstest.cern.ch.
* Update the webpage [https://gitlab.cern.ch/linuxsupport/websites/linux.cern.ch](https://gitlab.cern.ch/linuxsupport/websites/linux.cern.ch)
* Send email template to users informing that CentOS7.$release is available for testing. The recipient should be 'linux-users@cern.ch' and the following recipients should be included in BCC 'linux-announce@cern.ch', 'linux-announce-test@cern.ch', 'ai-admins@cern.ch'
* You may use `/mnt/data2/dist/cern/centos/7.$release/os/x86_64/CERN/build/release-email-test.sh` to generate the text
* Note: this script does not SEND the email
* Send email template to users informing that CentOS7.$RELEASE is available for testing. The recipient should be 'linux-users@cern.ch' and the following recipients should be included in BCC 'linux-announce@cern.ch', 'linux-announce-test@cern.ch', 'ai-admins@cern.ch'
* You may use `/mnt/data2/dist/cern/centos/7.$RELEASE/os/x86_64/CERN/build/release-email-test.sh` to generate the text
* Note: this script does NOT send the email
### Release to production
* Remove /mnt/data1/dist/cern/centos/7.$release/.htaccess
* Change the symlink /mnt/data2/dist/cern/centos/7 to point to 7.$release
* Update the CENTOS_VERSION string in `~/bin/config.sh`
* Remove `/mnt/data1/dist/cern/centos/7.$RELEASE/.htaccess`
* Change the symlink `/mnt/data2/dist/cern/centos/7` to point to `7.$RELEASE`
```
ln -sfn /mnt/data2/dist/cern/centos/7.$RELEASE.$YEARMONTH/ /mnt/data2/dist/cern/centos/7
```
* Update the `CENTOS_VERSION` string in `~/bin/config.sh`
* Add centos-release, afs (both rpm and src rpm) to cern repo (bypassing cern-testing)
* Perform any clean up in 'extras' or 'updates' (any deprecated package removal from upstream)
* rsync sources from vault `mkdir -p /mnt/data2/dist/cern/centos/7.$release/os/Source/SPackages; cd /mnt/data2/dist/cern/centos/7.$release/os; ln -s Source Sources; cd Source ; rsync -avP rsync://mirror.nsc.liu.se/centos-store/7.$release/os/Source/ /mnt/data2/dist/cern/centos/7.$release/os/Source/`
* Edit and run `/mnt/data2/dist/cern/centos/7.$release/os/x86_64/CERN/build/sync.sh` to populate /mnt/data1/dist/cern/centos/7.$release/os/Source
* rsync /mnt/data2/dist/cern/centos/7.$release/os/x86_64/Packages to /mnt/data2/dist/cern/centos/7/updates/x86_64/Packages/ (this is needed to koji dependencies)
* rsync sources from vault:
```
mkdir -p /mnt/data2/dist/cern/centos/7.$RELEASE/os/Source/SPackages
cd /mnt/data2/dist/cern/centos/7.$RELEASE/os
ln -s Source Sources
cd Source
rsync -avP rsync://mirror.nsc.liu.se/centos-store/7.$RELEASE/os/Source/ /mnt/data2/dist/cern/centos/7.$RELEASE/os/Source/
```
* Edit and run `/mnt/data2/dist/cern/centos/7.$RELEASE/os/x86_64/CERN/build/sync.sh` to populate `/mnt/data1/dist/cern/centos/7.$RELEASE/os/Source`
* rsync `/mnt/data2/dist/cern/centos/7.$release/os/x86_64/Packages` to `/mnt/data2/dist/cern/centos/7/updates/x86_64/Packages/` (this is needed for koji dependencies)
* regen repo for cern, cr, cr-testing, updates (and extras if touched)
```
/mnt/data2/home/build/bin/bsregenrepos -d cc7 -r "cern cr cr-testing updates"
```
* sync to data1 `/mnt/data2/home/build/bin/bssync -t "data1" -d cc7 -r "cern cr cr-testing updates"`
* Promote TEST VM to Production using the [koji-image-build monthly scheduled pipeline](https://gitlab.cern.ch/linuxsupport/koji-image-build/-/pipeline_schedules) you created previously
* Change the symlink /mnt/data1/dist/cern/centos/7 to point to 7.$release
* Remove $release TEST target on aims, add $release target on aims
* Update aims2-loaders, removing reference to TEST and promoting $release as default (you can use `/mnt/data2/dist/cern/centos/7.$release/os/x86_64/CERN/build/aims2.sh`)
* Upgrade aims2-loaders on aimstest/aims
* Test pxe installation works by using default CC7 entry and seeing $release on the installed host
* Create Docker image and upload to gitlab registry and dockerhub
* Promote the TEST VM to Production using the [koji-image-build monthly scheduled pipeline](https://gitlab.cern.ch/linuxsupport/koji-image-build/-/pipeline_schedules) you created previously
* Change the symlink `/mnt/data1/dist/cern/centos/7` to point to `7.$release`:
```
ln -sfn /mnt/data1/dist/cern/centos/7.$RELEASE.$YEARMONTH/ /mnt/data1/dist/cern/centos/7
```
* Remove the `$RELEASE TEST` target on aims, add $RELEASE target on aims
* Update aims2-loaders, removing reference to TEST and promoting $release as default (you can use `/mnt/data2/dist/cern/centos/7.$release/os/x86_64/CERN/build/aims2.sh`). You might want to update the `BE/CO` dedicated menu as well.
* Upgrade `aims2-loaders` on all `aimstest/aims` machines
* Test the PXE installation works by using the default CC7 entry and seeing $RELEASE on the installed host
* Create Docker image and upload it to gitlab registry and dockerhub, which is done in: `https://gitlab.cern.ch/linuxsupport/cc7-base/`
* Update the webpage [https://gitlab.cern.ch/linuxsupport/websites/linux.cern.ch](https://gitlab.cern.ch/linuxsupport/websites/linux.cern.ch)
* Add an entry to the ITSSB (the following can be used an example [https://cern.service-now.com/service-portal/view-outage.do?n=OTG0052544](https://cern.service-now.com/service-portal/view-outage.do?n=OTG0052544)
* Send email template to users informing that CentOS7.$release is available for testing. The recipient should be 'linux-users@cern.ch' and the following recipients should be included in BCC 'linux-announce@cern.ch', 'linux-announce-test@cern.ch', 'ai-admins@cern.ch'
* Send email template to users informing that CentOS7.$release is available in production. The recipient should be 'linux-users@cern.ch' and the following recipients should be included in BCC 'linux-announce@cern.ch', 'linux-announce-test@cern.ch', 'ai-admins@cern.ch'
* You may use `/mnt/data2/dist/cern/centos/7.$release/os/x86_64/CERN/build/release-email-prod.sh` to generate the text
* Note: this script does not SEND the email
* create 7.$release in /eos/project/l/linux/cern/centos/
* sync to eos `/mnt/data2/home/build/bin/bssync -t "eos" -d cc7 -r "all"
* Note: this script does NOT send the email
* Create `7.$RELEASE` under `/eos/project/l/linux/cern/centos/`
* Sync to EOS:
```
/mnt/data2/home/build/bin/bssync -t "eos" -d cc7 -r "all"
```
This takes a long time. There might be quota issues, so several runs might be needed.
\ No newline at end of file