Koji upgrade review

I thought I had sent this on Friday

docs/koji/upgrading.md

@@ -13,20 +13,39 @@ The standard process to upgrade koji is:
     The below instructions are tailored for kojitest - adaption will be required to execute against koji production
 
 !!! Note ""
-    Upgrading koji production will require the usual ITSSB notifications
+    Upgrading koji production will require the usual ITSSB notifications. Take [OTG0069160](https://cern.service-now.com/service-portal?id=outage&n=OTG0069160)
+    as an example. Try to schedule them with at least one week in advance.
 
 ## Build new rpm and tag appropriately
-see [https://gitlab.cern.ch/linuxsupport/rpms/koji](https://gitlab.cern.ch/linuxsupport/rpms/koji)
+
+See [https://gitlab.cern.ch/linuxsupport/rpms/koji](https://gitlab.cern.ch/linuxsupport/rpms/koji)
+
+**Before you tag the rpm to `qa` make sure all koji.cern.ch nodes are on the production environment and all kojitest.cern.ch nodes are on the qa environment.**
+
+If you do not do so, you may have prod nodes with qa repos which will make versions inconsistent across the LSB nodes.
 
 !!! Note ""
     Don't forget to tag the build as needed **before** shutting down Koji!
 
+# Test it
+
+The following process applies for both test and prod nodes, so please first do it on <kojitest.cern.ch> involved nodes, adapting the commands when necessary.
+Once you have tested a few builds, tags, image-builds or else, you will have validated that the built rpm works as it should and you can announce the OTG.
+
+You may want to install the test rpm from <linuxsoft.cern.ch/internal/repos/linuxsupport8s-qa/> (or the corresponding one for other distros) locally to do this. Please be aware of your `/etc/koji.conf` content, take it from `aiadm.cern.ch` if you need to adjust it.
+
 ## Disable alerts
 
 Make sure Roger knows something is going on:
 
 ```
-roger update --all_alarms false --message OTG0062870 --duration 2h koji{10,11,20,21,22,30,31,32,33,34,35,40,41}
+# roger status changes must be ran with sudo even when ssh'ing as root
+# test
+wassh -l root -c lsb/test2 'sudo roger update --all_alarms false --message OTGXXXXXX --duration 2h'
+# prod (limit it to prod nodes, not the whole hostgroup)
+wassh -l root -c lsb/hub 'sudo roger update --all_alarms false --message OTGXXXXXX --duration 2h'
+wassh -l root -c lsb/web 'sudo roger update --all_alarms false --message OTGXXXXXX --duration 2h'
+wassh -l root -c lsb/builder 'sudo roger update --all_alarms false --message OTGXXXXXX --duration 2h'
 ```
 
 (If you change the appstate, the machines would be removed from the LB alias and the intervention will take longer)
@@ -36,30 +55,72 @@ roger update --all_alarms false --message OTG0062870 --duration 2h koji{10,11,20
 *puppet (disable)*
 
 ```
+# test
 mco puppet disable "koji upgrade OTGXXXXXX" --dm puppetdb -T lsb -F 'hostgroup_1=test2'
+# prod (limit it to prod nodes, not the whole hostgroup)
+mco puppet disable "koji upgrade OTGXXXXXX" --dm puppetdb -T lsb -F 'hostgroup_1=hub'
+mco puppet disable "koji upgrade OTGXXXXXX" --dm puppetdb -T lsb -F 'hostgroup_1=web'
+mco puppet disable "koji upgrade OTGXXXXXX" --dm puppetdb -T lsb -F 'hostgroup_1=builder'
+
+# If mco does not work for you:
+# test
+wassh -l root -c lsb/test2 'puppet agent --disable'
+# prod
+wassh -l root -c lsb 'puppet agent --disable'
 ```
 
 *builders*
 
 ```
+# test
 mco service stop kojid --dm puppetdb -T lsb -F 'hostgroup_1=test2' -F 'hostgroup_2=builder'
+# prod
+mco service stop kojid --dm puppetdb -T lsb -F 'hostgroup_1=builder'
+
+# If mco does not work for you:
+# test
+wassh -l root -c lsb/test2/builder 'service kojid stop'
+# prod
+wassh -l root -c lsb/builder 'service kojid stop'
 ```
 
 *hub*
 
 ```
+# test
 mco service stop kojira --dm puppetdb -T lsb -F 'hostgroup_1=test2' -F 'hostgroup_2=hub'
 mco service stop httpd --dm puppetdb -T lsb -F 'hostgroup_1=test2' -F 'hostgroup_2=hub'
+# prod
+mco service stop kojira --dm puppetdb -T lsb -F 'hostgroup_1=hub'
+mco service stop httpd --dm puppetdb -T lsb -F 'hostgroup_1=hub'
+
+# If mco does not work for you:
+# test
+wassh -l root -c lsb/test2/hub 'service kojira stop'
+wassh -l root -c lsb/test2/hub 'service httpd stop'
+# prod
+wassh -l root -c lsb/hub 'service kojira stop'
+wassh -l root -c lsb/hub 'service httpd stop'
 ```
 
 *web*
 
 ```
+# test
 mco service stop httpd --dm puppetdb -T lsb -F 'hostgroup_1=test2' -F 'hostgroup_2=web'
+# prod
+mco service stop httpd --dm puppetdb -T lsb -F 'hostgroup_1=web'
+
+# If mco does not work for you:
+# test
+wassh -l root -c lsb/test2/web 'service httpd stop'
+# prod
+wassh -l root -c lsb/web 'service httpd stop'
 ```
 
 ## Backup and run migration script
-*Extract migration script*
+
+*Extract migration script **if there is one***
 
 ```
 yumdownloader koji
@@ -67,6 +128,7 @@ rpm2cpio koji-1.18.1-1.el7.cern.noarch.rpm | cpio -idv ./usr/share/doc/koji-1.18
 scp usr/share/doc/koji-1.18.1/docs/schema-upgrade-1.17-1.18.sql aiadm:
 ```
 
+
 *Initialise postgresql 9.6 environment*
 ```
 ssh aiadm
@@ -81,13 +143,28 @@ pg_dump -h $dbod.cern.ch -p $port -d $database -U $username > kojitest_1.17-`dat
 psql -h $dbod.cern.ch -p $port -d $database -U $username < schema-upgrade-1.17-1.18.sql
 ```
 
-!!! Note ""
-    You can retrieve the credentials from /etc/koji-hub/hub.conf
+**Note**: You can retrieve the credentials from `/etc/koji-hub/hub.conf` or with `tbag`:
 
-##Upgrade Koji RPMs
+```
+# test
+tbag show koji_db_password --hg lsb/test2
+# prod
+tbag show koji_db_password --hg lsb
+```
+
+## Upgrade Koji RPMs
 
 ```
+# test
 mco shell run '/usr/bin/yum clean all && /usr/local/sbin/distro_sync.sh' --dm puppetdb -T lsb -F 'hostgroup_1=test2'
+# prod
+mco shell run '/usr/bin/yum clean all && /usr/local/sbin/distro_sync.sh' --dm puppetdb -T lsb
+
+# If mco does not work for you:
+# test
+wassh -l root -c lsb/test2 '/usr/bin/yum clean all && /usr/local/sbin/distro_sync.sh'
+# prod
+wassh -l root -c lsb '/usr/bin/yum clean all && /usr/local/sbin/distro_sync.sh'
 ```
 
 ## Restart Koji
@@ -103,5 +180,12 @@ mco puppet runonce --dm puppetdb -T lsb -F 'hostgroup_1=test2'
 
 ```
 mco puppet enable --dm puppetdb -T lsb -F 'hostgroup_1=test2'
-roger update --all_alarms true koji{10,11,20,21,22,30,31,32,33,34,35,40,41}
+
+# test
+wassh -l root -c lsb/test2 'puppet agent --enable'
+wassh -l root -c lsb/test2 'sudo roger update --all_alarms true'
+
+# prod
+wassh -l root -c lsb 'puppet agent --enable'
+wassh -l root -c lsb 'sudo roger update --all_alarms true'
 ```