update updgrade koji

docs/koji/upgrading.md

 # Upgrading koji
 
+## Overview 
 The standard process to upgrade koji is:
 
 1. Build the new RPMs
@@ -10,14 +11,12 @@ The standard process to upgrade koji is:
 6. Restarting Koji services
 
 
-!!! Note ""
-    The below instructions are tailored for kojitest - adaption will be required to execute against koji production
+**Note**: The below instructions are tailored for kojitest - adaption will be required to execute against koji production
 
-!!! Note ""
-    Upgrading koji production will require the usual ITSSB notifications. Take [OTG0069160](https://cern.service-now.com/service-portal?id=outage&n=OTG0069160)
+**Note**: Upgrading koji production will require the usual ITSSB notifications. Take [OTG0069160](https://cern.service-now.com/service-portal?id=outage&n=OTG0069160)
     as an example. Try to schedule them with at least one week in advance.
 
-## Build new rpm and tag appropriately
+### Build new rpm and tag appropriately
 
 See [https://gitlab.cern.ch/linuxsupport/rpms/koji](https://gitlab.cern.ch/linuxsupport/rpms/koji)
 
@@ -25,10 +24,9 @@ See [https://gitlab.cern.ch/linuxsupport/rpms/koji](https://gitlab.cern.ch/linux
 
 If you do not do so, you may have prod nodes with qa repos which will make versions inconsistent across the LSB nodes.
 
-!!! Note ""
-    Don't forget to tag the build as needed **before** shutting down Koji!
+**Note**: Don't forget to tag the build as needed **before** shutting down Koji!
 
-# Test it
+### Test it
 
 The following process applies for both test and prod nodes, so please first do it on <kojitest.cern.ch> involved nodes, adapting the commands when necessary.
 Once you have tested a few builds, tags, image-builds or else, you will have validated that the built rpm works as it should and you can announce the OTG.
@@ -37,57 +35,73 @@ You may want to install the test rpm from <linuxsoft.cern.ch/internal/repos/linu
 
 In order to ease testing, you can run test pipelines from <https://gitlab.cern.ch/linuxsupport/testing/koji-tester>.
 
-## Disable alerts
+## Upgrading Steps:
 
-!!! Note ""
-    These wassh commands may not work for the case of 2FA protected nodes, so you may need to run it yourself from within those.
+**Note**: Don't worry if mco command doesn't work in the next steps, use wassh commands instead.
+
+### 1) Disable alerts
+
+**Note**: These wassh commands may not work for the case of 2FA protected nodes, so you may need to run it yourself from within those.
 
 Make sure Roger knows something is going on:
 
 ```
-# roger status changes must be ran with sudo even when ssh'ing as root
+ The roger status changes must be ran with sudo even when ssh'ing as root
+
 # test
 wassh -l root -c lsb/test2 'sudo roger update --all_alarms false --message OTGXXXXXX --duration 2h'
+
 # prod (limit it to prod nodes, not the whole hostgroup)
 wassh -l root -c lsb/hub 'sudo roger update --all_alarms false --message OTGXXXXXX --duration 2h'
 wassh -l root -c lsb/web 'sudo roger update --all_alarms false --message OTGXXXXXX --duration 2h'
 wassh -l root -c lsb/builder 'sudo roger update --all_alarms false --message OTGXXXXXX --duration 2h'
+wassh -l root -c lsb/adm 'sudo roger update --all_alarms false --message OTGXXXXXX --duration 2h'
 ```
 
 (If you change the appstate, the machines would be removed from the LB alias and the intervention will take longer)
 
-## Shutdown Koji
+### 2) Shutdown Koji
 
 *puppet (disable)*
 
 ```
 # test
 mco puppet disable "koji upgrade OTGXXXXXX" --dm puppetdb -T lsb -F 'hostgroup_1=test2'
+
 # prod (limit it to prod nodes, not the whole hostgroup)
 mco puppet disable "koji upgrade OTGXXXXXX" --dm puppetdb -T lsb -F 'hostgroup_1=hub'
 mco puppet disable "koji upgrade OTGXXXXXX" --dm puppetdb -T lsb -F 'hostgroup_1=web'
 mco puppet disable "koji upgrade OTGXXXXXX" --dm puppetdb -T lsb -F 'hostgroup_1=builder'
 
+
+
 # If mco does not work for you:
 # test
 wassh -l root -c lsb/test2 'puppet agent --disable'
+
 # prod
 wassh -l root -c lsb/hub 'puppet agent --disable'
 wassh -l root -c lsb/web 'puppet agent --disable'
 wassh -l root -c lsb/builder 'puppet agent --disable'
+wassh -l root -c lsb/adm 'puppet agent --disable'
 ```
 
+Run the following command just after the tag the build to qa (test) /master (prod):
+
 *builders*
 
 ```
 # test
 mco service stop kojid --dm puppetdb -T lsb -F 'hostgroup_1=test2' -F 'hostgroup_2=builder'
+
 # prod
 mco service stop kojid --dm puppetdb -T lsb -F 'hostgroup_1=builder'
 
+
 # If mco does not work for you:
 # test
 wassh -l root -c lsb/test2/builder 'service kojid stop'
+
 # prod
 wassh -l root -c lsb/builder 'service kojid stop'
 ```
@@ -98,14 +112,17 @@ wassh -l root -c lsb/builder 'service kojid stop'
 # test
 mco service stop kojira --dm puppetdb -T lsb -F 'hostgroup_1=test2' -F 'hostgroup_2=hub'
 mco service stop httpd --dm puppetdb -T lsb -F 'hostgroup_1=test2' -F 'hostgroup_2=hub'
+
 # prod
 mco service stop kojira --dm puppetdb -T lsb -F 'hostgroup_1=hub'
 mco service stop httpd --dm puppetdb -T lsb -F 'hostgroup_1=hub'
 
+
 # If mco does not work for you:
 # test
 wassh -l root -c lsb/test2/hub 'service kojira stop'
 wassh -l root -c lsb/test2/hub 'service httpd stop'
+
 # prod
 wassh -l root -c lsb/hub 'service kojira stop'
 wassh -l root -c lsb/hub 'service httpd stop'
@@ -116,36 +133,49 @@ wassh -l root -c lsb/hub 'service httpd stop'
 ```
 # test
 mco service stop httpd --dm puppetdb -T lsb -F 'hostgroup_1=test2' -F 'hostgroup_2=web'
+
 # prod
 mco service stop httpd --dm puppetdb -T lsb -F 'hostgroup_1=web'
 
+
 # If mco does not work for you:
 # test
 wassh -l root -c lsb/test2/web 'service httpd stop'
+
 # prod
 wassh -l root -c lsb/web 'service httpd stop'
 ```
 
-## Backup and run migration script
+### 3) Backup and run migration script
 
 *Extract migration script **if there is one***
 
+On aiadm or lxplus:
+
+```
+$ wget https://pagure.io/koji/blob/master/f/docs/schema-upgrade-1.32-1.33.sql
+```
+
+Check if the file was downloaded correctly:
+
 ```
-yumdownloader koji
-rpm2cpio koji-1.18.1-1.el7.cern.noarch.rpm | cpio -idv ./usr/share/doc/koji-1.18.1/docs/schema-upgrade-1.17-1.18.sql
-scp usr/share/doc/koji-1.18.1/docs/schema-upgrade-1.17-1.18.sql aiadm:
+$ cat schema-upgrade-1.32-1.33.sql
 ```
 
 *dump existing db*
+
+This is too heavy, so you can just do a backup of the db in https://dbod.web.cern.ch/pages/instance/koji, a skip the next command.
+
 ```
-pg_dump -h $dbod.cern.ch -p $port -d $database -U $username > kojitest_1.17-`date +%Y%m%d%H%M`.sql
+pg_dump -h $dbod.cern.ch -p $port -d $database -U $username > kojitest_1.32-`date +%Y%m%d%H%M`.sql
 ```
+
 *run migration script*
 ```
-psql -h $dbod.cern.ch -p $port -d $database -U $username < schema-upgrade-1.17-1.18.sql
+psql -h $dbod.cern.ch -p $port -d $database -U $username < schema-upgrade-1.32-1.33.sql
 ```
 
-**Note**: You can retrieve the credentials from `/etc/koji-hub/hub.conf` or with `tbag`:
+**Note**: You can retrieve the credentials from `/etc/koji-hub/hub.conf` in a koji hub node or with `tbag`:
 
 ```
 # test
@@ -154,24 +184,29 @@ tbag show koji_db_password --hg lsb/test2
 tbag show koji_db_password --hg lsb
 ```
 
-## Upgrade Koji RPMs
+### 4) Upgrade Koji RPMs
 
 ```
 # test
 mco shell run '/usr/bin/yum clean all && /usr/local/sbin/distro_sync.sh' --dm puppetdb -T lsb -F 'hostgroup_1=test2'
+
 # prod
 mco shell run '/usr/bin/yum clean all && /usr/local/sbin/distro_sync.sh' --dm puppetdb -T lsb
 
+
+
 # If mco does not work for you:
 # test
 wassh -l root -c lsb/test2 '/usr/bin/yum clean all && /usr/local/sbin/distro_sync.sh'
+
 # prod
 wassh -l root -c lsb/hub '/usr/bin/yum clean all && /usr/local/sbin/distro_sync.sh'
 wassh -l root -c lsb/web '/usr/bin/yum clean all && /usr/local/sbin/distro_sync.sh'
 wassh -l root -c lsb/builder '/usr/bin/yum clean all && /usr/local/sbin/distro_sync.sh'
+wassh -l root -c lsb/adm '/usr/bin/yum clean all && /usr/local/sbin/distro_sync.sh'
 ```
 
-## (Optional) Upgrade ` edk2.git-ovmf-x64-0` RPM
+### 5) (Optional) Upgrade ` edk2.git-ovmf-x64-0` RPM
 
 This step is optional, as it is not strictly required to update the RPM. The reason behind the yumlock pinning is that since we are using nightly builds we have seen that certain updates make UEFI VM booting impossible, so we should better control the version we use at any given point to make sure it keeps on working.
 
@@ -179,7 +214,7 @@ Pinned at <https://gitlab.cern.ch/ai/it-puppet-module-koji/-/blob/qa/code/manife
 
 To test it you would need to follow the usual Puppet environments workflow as per <https://configdocs.web.cern.ch/changes/environment.html>.
 
-## Restart Koji
+### 6) Restart Koji
 
 ```
 mco puppet enable --dm puppetdb -T lsb -F 'hostgroup_1=test2'
@@ -188,7 +223,7 @@ mco puppet runonce --dm puppetdb -T lsb -F 'hostgroup_1=test2'
 
 (You can also reboot the nodes to get the latest kernel, etc.)
 
-## Re-enable puppet and all_alarms
+### 7) Re-enable puppet and all_alarms
 
 ```
 mco puppet enable --dm puppetdb -T lsb -F 'hostgroup_1=test2'
@@ -201,4 +236,9 @@ wassh -l root -c lsb/test2 'sudo roger update --all_alarms true'
 wassh -l root -c lsb/hub 'puppet agent --enable && sudo roger update --all_alarms true'
 wassh -l root -c lsb/web 'puppet agent --enable && sudo roger update --all_alarms true'
 wassh -l root -c lsb/builder 'puppet agent --enable && sudo roger update --all_alarms true'
+wassh -l root -c lsb/adm 'puppet agent --enable &&  sudo roger update --all_alarms true'
 ```
+
+### 8) Test
+
+Trigger the pipeline: https://gitlab.cern.ch/linuxsupport/testing/koji-tester - Change the koji variable if you are doing it to prod.