Skip to content
Snippets Groups Projects

Update koji docs

Merged Update koji docs
upgrade into master
1 unresolved thread
Merged Marta Vila Fernandes requested to merge upgrade into master
1 unresolved thread
Compare
and
1 file
+ 71
− 26
Compare changes
  • Side-by-side
  • Inline
docs/koji/upgrading.md
+ 71
− 26
@@ -3,22 +3,63 @@
## Overview
The standard process to upgrade koji is:
1. Build the new RPMs
2. Stopping all Koji services
3. Running the DB migration script (or anything else listed at [https://docs.pagure.org/koji/migrations/migrations/](https://docs.pagure.org/koji/migrations/migrations/))
4. Upgrade the Koji RPMs
5. (Optional) Upgrade the ` edk2.git-ovmf-x64-0` RPM
6. Restarting Koji services
**Note**: The below instructions are tailored for kojitest - adaption will be required to execute against koji production
**Note**: Upgrading koji production will require the usual ITSSB notifications. Take [OTG0069160](https://cern.service-now.com/service-portal?id=outage&n=OTG0069160)
as an example. Try to schedule them with at least one week in advance.
### Build new rpm and tag appropriately
See [https://gitlab.cern.ch/linuxsupport/rpms/koji](https://gitlab.cern.ch/linuxsupport/rpms/koji)
1. Build the new RPM
2. Create a tag and tag it to qa
3. Upgrading Koji in the test environment:
- Stopping all Koji services
- Running the DB migration script (or anything else listed at [https://docs.pagure.org/koji/migrations/migrations/](https://docs.pagure.org/koji/migrations/migrations/))
- Upgrade the Koji RPMs
- (Optional) Upgrade the ` edk2.git-ovmf-x64-0` RPM
- Restarting Koji services
- Re-enable all Koji services
- Test it
4. Scheduling an OTG
5. On the intervetion day:
- Tag it to production
- Perform the same steps described in 3. for the production environment
6. Bravo!
Detailed information in the next items.
## Build new rpm and tag it appropriately
Our Koji rpm: [https://gitlab.cern.ch/linuxsupport/rpms/koji](https://gitlab.cern.ch/linuxsupport/rpms/koji)
1) Get the new Koji upstream version
1.1) go to https://koji.fedoraproject.org/koji/packageinfo?packageID=1181 *OR* https://pagure.io/koji/releases get the last release;
1.2) in aiadm download the src.rpm `wget https://kojipkgs.fedoraproject.org//packages/koji/1.33.0/1.el8/src/koji-1.33.0-1.el8.src.rpm` *OR* `https://pagure.io/koji/archive/koji-1.33.1/koji-koji-1.33.1.tar.gz`
1.3) install the src.rpm `rpm -ivh koji-1.33.0-1.el8.src.rpm` *OR*
```
gunzip koji-koji-1.33.1.tar.gz
tar -xf koji-koji-1.33.1.tar
mv koji-koji-1.33.1 koji-1.33.1
```
1.4) go to `cd ~/rpmbuild/SOURCES` *OR* create the `tar -cvjSf koji-1.33.1.tar.bz2 koji-1.33.1`
1.5) copy the `koji-<>.tar.bz2` to `/koji/src` of [https://gitlab.cern.ch/linuxsupport/rpms/koji](https://gitlab.cern.ch/linuxsupport/rpms/koji) and change the spec file.
2) Create a MR and check if the pipeline run without errors.
3) If there is a error on the pipeline related with the `prometheus-metrics-index.patch`, it means that it needs to be recreated.
3.1) Create a copy of the last release of koji `koji-<last_version>_orig`
3.2) Change the index.py in /usr/share/koji-web/scripts to add a metrics function that will call kojiexporter script
```
+from kojiweb.kojiexporter import prometheus_metrics
+
+
+def metrics(environ):
+ _getServer(environ)
+ _initValues(environ, 'PrometheusMetrics', 'metrics')
+ environ['koji.headers'].append(['Content-Type', 'text/plain'])
+ metrics = prometheus_metrics(environ)
+ return metrics
+
```
3.3) Create the patch: `diff -urN koji-1.33.1_orig/www koji-1.33.1/www
3.4) Add it to `/koji/src`
4) When the pipeline succeed, and it is reviewed by our colleagues, it is ready to merge to master!
5) Create a new tag
**Before you tag the rpm to `qa` make sure all koji.cern.ch nodes are on the production environment and all kojitest.cern.ch nodes are on the qa environment.**
@@ -26,17 +67,10 @@ If you do not do so, you may have prod nodes with qa repos which will make versi
**Note**: Don't forget to tag the build as needed **before** shutting down Koji!
### Test it
The following process applies for both test and prod nodes, so please first do it on <kojitest.cern.ch> involved nodes, adapting the commands when necessary.
Once you have tested a few builds, tags, image-builds or else, you will have validated that the built rpm works as it should and you can announce the OTG.
You may want to install the test rpm from <linuxsoft.cern.ch/internal/repos/linuxsupport8s-qa/> (or the corresponding one for other distros) locally to do this. Please be aware of your `/etc/koji.conf` content, take it from `aiadm.cern.ch` if you need to adjust it.
In order to ease testing, you can run test pipelines from <https://gitlab.cern.ch/linuxsupport/testing/koji-tester>.
## Upgrading Steps:
The following process applies for both test and prod nodes, so please first do it on kojitest.cern.ch involved nodes, adapting the commands when necessary.
**Note**: Don't worry if mco command doesn't work in the next steps, use wassh commands instead.
### 1) Disable alerts
@@ -241,4 +275,15 @@ wassh -l root -c lsb/adm 'puppet agent --enable && sudo roger update --all_alar
### 8) Test
Trigger the pipeline: https://gitlab.cern.ch/linuxsupport/testing/koji-tester - Change the koji variable if you are doing it to prod.
Once you have tested a few builds, tags, image-builds or else, you will have validated that the built rpm works as it should and you can announce the OTG.
You may want to install the test rpm from linuxsoft.cern.ch/internal/repos/linuxsupport8al-qa/ (or the corresponding one for other distros) locally to do this. Please be aware of your `/etc/koji.conf` content, take it from `aiadm.cern.ch` if you need to adjust it.
Trigger the pipeline: <https://gitlab.cern.ch/linuxsupport/testing/koji-tester> - Change the koji variable if you are doing it to prod.
## Scheduling an OTG
After the Koji upgrade in the test environment, it is time to schedule the OTG to upgrade Koji in production.
Take [OTG0078591](https://cern.service-now.com/service-portal?id=outage&n=OTG0078591) as an example.
Try to schedule them with at least one week in advance.
\ No newline at end of file