Skip to content
Snippets Groups Projects

Koji upgrade review

Merged Daniel Juarez Gonzalez requested to merge koji_upgrade into master
1 unresolved thread
+ 92
8
@@ -13,20 +13,39 @@ The standard process to upgrade koji is:
The below instructions are tailored for kojitest - adaption will be required to execute against koji production
!!! Note ""
Upgrading koji production will require the usual ITSSB notifications
Upgrading koji production will require the usual ITSSB notifications. Take [OTG0069160](https://cern.service-now.com/service-portal?id=outage&n=OTG0069160)
as an example. Try to schedule them with at least one week in advance.
## Build new rpm and tag appropriately
see [https://gitlab.cern.ch/linuxsupport/rpms/koji](https://gitlab.cern.ch/linuxsupport/rpms/koji)
See [https://gitlab.cern.ch/linuxsupport/rpms/koji](https://gitlab.cern.ch/linuxsupport/rpms/koji)
**Before you tag the rpm to `qa` make sure all koji.cern.ch nodes are on the production environment and all kojitest.cern.ch nodes are on the qa environment.**
If you do not do so, you may have prod nodes with qa repos which will make versions inconsistent across the LSB nodes.
!!! Note ""
Don't forget to tag the build as needed **before** shutting down Koji!
# Test it
The following process applies for both test and prod nodes, so please first do it on <kojitest.cern.ch> involved nodes, adapting the commands when necessary.
Once you have tested a few builds, tags, image-builds or else, you will have validated that the built rpm works as it should and you can announce the OTG.
You may want to install the test rpm from <linuxsoft.cern.ch/internal/repos/linuxsupport8s-qa/> (or the corresponding one for other distros) locally to do this. Please be aware of your `/etc/koji.conf` content, take it from `aiadm.cern.ch` if you need to adjust it.
## Disable alerts
Make sure Roger knows something is going on:
```
roger update --all_alarms false --message OTG0062870 --duration 2h koji{10,11,20,21,22,30,31,32,33,34,35,40,41}
# roger status changes must be ran with sudo even when ssh'ing as root
# test
wassh -l root -c lsb/test2 'sudo roger update --all_alarms false --message OTGXXXXXX --duration 2h'
# prod (limit it to prod nodes, not the whole hostgroup)
wassh -l root -c lsb/hub 'sudo roger update --all_alarms false --message OTGXXXXXX --duration 2h'
wassh -l root -c lsb/web 'sudo roger update --all_alarms false --message OTGXXXXXX --duration 2h'
wassh -l root -c lsb/builder 'sudo roger update --all_alarms false --message OTGXXXXXX --duration 2h'
```
(If you change the appstate, the machines would be removed from the LB alias and the intervention will take longer)
@@ -36,30 +55,72 @@ roger update --all_alarms false --message OTG0062870 --duration 2h koji{10,11,20
*puppet (disable)*
```
# test
mco puppet disable "koji upgrade OTGXXXXXX" --dm puppetdb -T lsb -F 'hostgroup_1=test2'
# prod (limit it to prod nodes, not the whole hostgroup)
mco puppet disable "koji upgrade OTGXXXXXX" --dm puppetdb -T lsb -F 'hostgroup_1=hub'
mco puppet disable "koji upgrade OTGXXXXXX" --dm puppetdb -T lsb -F 'hostgroup_1=web'
mco puppet disable "koji upgrade OTGXXXXXX" --dm puppetdb -T lsb -F 'hostgroup_1=builder'
# If mco does not work for you:
# test
wassh -l root -c lsb/test2 'puppet agent --disable'
# prod
wassh -l root -c lsb 'puppet agent --disable'
```
*builders*
```
# test
mco service stop kojid --dm puppetdb -T lsb -F 'hostgroup_1=test2' -F 'hostgroup_2=builder'
# prod
mco service stop kojid --dm puppetdb -T lsb -F 'hostgroup_1=builder'
# If mco does not work for you:
# test
wassh -l root -c lsb/test2/builder 'service kojid stop'
# prod
wassh -l root -c lsb/builder 'service kojid stop'
```
*hub*
```
# test
mco service stop kojira --dm puppetdb -T lsb -F 'hostgroup_1=test2' -F 'hostgroup_2=hub'
mco service stop httpd --dm puppetdb -T lsb -F 'hostgroup_1=test2' -F 'hostgroup_2=hub'
# prod
mco service stop kojira --dm puppetdb -T lsb -F 'hostgroup_1=hub'
mco service stop httpd --dm puppetdb -T lsb -F 'hostgroup_1=hub'
# If mco does not work for you:
# test
wassh -l root -c lsb/test2/hub 'service kojira stop'
wassh -l root -c lsb/test2/hub 'service httpd stop'
# prod
wassh -l root -c lsb/hub 'service kojira stop'
wassh -l root -c lsb/hub 'service httpd stop'
```
*web*
```
# test
mco service stop httpd --dm puppetdb -T lsb -F 'hostgroup_1=test2' -F 'hostgroup_2=web'
# prod
mco service stop httpd --dm puppetdb -T lsb -F 'hostgroup_1=web'
# If mco does not work for you:
# test
wassh -l root -c lsb/test2/web 'service httpd stop'
# prod
wassh -l root -c lsb/web 'service httpd stop'
```
## Backup and run migration script
*Extract migration script*
*Extract migration script **if there is one***
```
yumdownloader koji
@@ -67,6 +128,7 @@ rpm2cpio koji-1.18.1-1.el7.cern.noarch.rpm | cpio -idv ./usr/share/doc/koji-1.18
scp usr/share/doc/koji-1.18.1/docs/schema-upgrade-1.17-1.18.sql aiadm:
```
*Initialise postgresql 9.6 environment*
```
ssh aiadm
@@ -81,13 +143,28 @@ pg_dump -h $dbod.cern.ch -p $port -d $database -U $username > kojitest_1.17-`dat
psql -h $dbod.cern.ch -p $port -d $database -U $username < schema-upgrade-1.17-1.18.sql
```
!!! Note ""
You can retrieve the credentials from /etc/koji-hub/hub.conf
**Note**: You can retrieve the credentials from `/etc/koji-hub/hub.conf` or with `tbag`:
##Upgrade Koji RPMs
```
# test
tbag show koji_db_password --hg lsb/test2
# prod
tbag show koji_db_password --hg lsb
```
## Upgrade Koji RPMs
```
# test
mco shell run '/usr/bin/yum clean all && /usr/local/sbin/distro_sync.sh' --dm puppetdb -T lsb -F 'hostgroup_1=test2'
# prod
mco shell run '/usr/bin/yum clean all && /usr/local/sbin/distro_sync.sh' --dm puppetdb -T lsb
# If mco does not work for you:
# test
wassh -l root -c lsb/test2 '/usr/bin/yum clean all && /usr/local/sbin/distro_sync.sh'
# prod
wassh -l root -c lsb '/usr/bin/yum clean all && /usr/local/sbin/distro_sync.sh'
```
## Restart Koji
@@ -103,5 +180,12 @@ mco puppet runonce --dm puppetdb -T lsb -F 'hostgroup_1=test2'
```
mco puppet enable --dm puppetdb -T lsb -F 'hostgroup_1=test2'
roger update --all_alarms true koji{10,11,20,21,22,30,31,32,33,34,35,40,41}
# test
wassh -l root -c lsb/test2 'puppet agent --enable'
wassh -l root -c lsb/test2 'sudo roger update --all_alarms true'
# prod
wassh -l root -c lsb 'puppet agent --enable'
wassh -l root -c lsb 'sudo roger update --all_alarms true'
```
Loading