Mechanism to swap between scheduler backends in production without using Puppet

The current method for switching between the two scheduler backends uses Puppet, which manages two configuration options. Switching is done via a hostgroup change (could also be controlled through another "flag").

To change the scheduler backend, operators currently need to:

1. Update the hostgroup of the targetted tape servers

2. Run Puppet on the target servers

3. Restart the servers.

The current procedure in 3 simple steps for pre-production, we are going to move tpsrv020:

12:38 guenther@aiadm84:~$ ai-foreman showhost tpsrv020.cern.ch
+------------------+------------------------+-------------------+---------------+
| Name             | Hostgroup                     | Environment       | OS            |
+------------------+------------------------+-------------------+---------------+
| tpsrv020.cern.ch | cta/tape/preproduction | cta_preproduction | AlmaLinux 9.5 |
+------------------+------------------------+-------------------+---------------+

[root@tpsrv020 ~]# grep Backend /etc/cta/cta-taped-IBMLIB4-TS1160-F08C2R2.conf
# CTA schedulerBackendName needed to give a name to schedulerDB resources ([ceph|postgres][instanceName][User|Repack|Tests...]
general SchedulerBackendName cephUser
ObjectStore BackendPath rados://cta-preproduction-test-2@tapecta-preproduction:cta-preproduction-test-2

1. guenther@aiadm84:~$ ai-foreman updatehost --hostgroup "cta/tape/preproduction/repack" tpsrv020
2. [root@tpsrv020 ~]# puppet agent -tv
3. [root@tpsrv020 ~]# systemctl restart cta-taped

Confirmation of the configuration change:

12:38 guenther@aiadm84:~$ ai-foreman showhost tpsrv020.cern.ch
+------------------+-------------------------------+-------------------+---------------+
| Name             | Hostgroup                     | Environment       | OS            |
+------------------+-------------------------------+-------------------+---------------+
| tpsrv020.cern.ch | cta/tape/preproduction/repack | cta_preproduction | AlmaLinux 9.5 |
+------------------+-------------------------------+-------------------+---------------+

[root@tpsrv020 ~]# grep Backend /etc/cta/cta-taped-IBMLIB4-TS1160-F08C2R2.conf
# CTA schedulerBackendName needed to give a name to schedulerDB resources ([ceph|postgres][instanceName][User|Repack|Tests...]
general SchedulerBackendName cephRepack
ObjectStore BackendPath rados://cta-preproduction-repack@tapecta-preproduction-repack:cta-preproduction-repack

To simplify and eliminate these manual operational steps, several alternatives were discussed in https://gitlab.cern.ch/cta/operations/-/issues/1668.

Among the options considered, the following approaches appear to be the most feasible for now. Each of them assumes that the configuration file will contain configuration for both scheduler backends:

a) Use of a cta-admin option to overwrite the configuration file in place and restart the server.

• For example, by changing the value of the hypothetical "scheduler_backend_in_use" key to point to either the physics or repack backend configuration.
• In this case, we need a mechanism to prevent Puppet from reverting the configuration file to its previous state.

b) Track configuration changes in the catalogue, then use the cta-admin option to apply the change and restart the server, but it still is facing the same issue as the previous option - we need to make Puppet aware of these specific lines being changes and make it ignore it which is not really the purpose for which we are running puppet at the first place.

c) Using another parameter in puppet than hostgroup will require the operator to make changes and commits in the hostgroup repository itself (and keeping all the branches in sync, i.e. preproduction, qa and master branch) --> more work --> hostgroup change seems fastest way to do this.

d) ... ?