Helm improvements and ci script cleanup

Summary

Apologies in advance for the large MR. It contains quite a few things at once, but having to do these things one at a time and waiting for merges would slow things down enormously. Seeing as I have to check the new Catalogue schema against the production clone soon...

So I figured it's easier to ask for forgiveness than to ask for permission. Anyway, to make it manageable, I'll break it down below.

Additionally, you are highly advised to first read the updated explanation of the setup here: https://gitlab.cern.ch/cta/CTA/-/blob/892-migrate-remaining-items-to-helm/continuousintegration/orchestration/README.md

Anyway, there are a lot of line changes, but the vast majority of those are just the movement of the same functionality to other places or other small things like indentation fixes.

Below is a list of all the things this MR includes:

• #901 (closed) Converted all the .conf files from the startup scripts in /opt/run/bin into ConfigMaps. As a result, the init_library.sh, init_database.sh and init_scheduler.sh are no longer necessary
  • The exception to this is the ctaeos chart, since this will at some point be replaced by EOS chart anyway
• #897 (closed) Refactored the run_systemtest.sh and create_instance.sh scripts so that they take properly understandable arguments and refactored the CI accordingly to use these updated arguments
• Removed all of the plain kubectl commands from the create_instance.sh script so that it is purely relying on Helm to spawn things
• Separated out the Oracle unit test path from the create_instance.sh script as it should be unrelated. This is still WIP and can be improved further later. The Oracle unit tests have their own Helm chart that is spawned by the tests/unit_test_oracle.sh script
• Removed all of the old plain .yaml files from the orchestration dir
• #888 (closed) #900 (closed) Separated out the Catalogue and the Scheduler from the init chart, so that they can be redeployed/upgraded separately. They each generate the necessary configuration for the rest of the pods. The exception is that part of the scheduler config still needs to be passed to the CTA chart due to how the .conf files are constructed.
  • As a result, the init.sh script was separated into three separate parts: 1 that cleans mhvtl, 1 that cleans/sets up the catalogue and 1 that clean/sets up the scheduler
• Refactored the liquibase update script so that it also uses Helm
• Allow the user to specify their own library configuration (or have it be auto generated)
• #902 (closed) Use Helm annotations to figure out whether a library device is in use instead of relying on the PVC generated by the minikube setup
• #893 (closed) The scheduler/catalogue config is now properly based on a values.yaml. There are some local configs available in presets/. For the ci, they are expected to be in /opt/kubernetes/CTA/scheduler/ci-*-values.yaml and /opt/kubernetes/CTA/catalogue/ci-*-values.yaml. This has already been put in place on the cirunners.
• #878 (closed) Fixed the default paths for the scheduler/catalogue configs
• Separated out a common chart with useful functions for all charts
• #906 (closed) Consistent naming and introduction of values.schema.json
• #833 (closed) Improved the logging output of the ci scripts
• Made the Oracle unit tests a regular test job instead of a cirunner job
• Quickly improved the .dockerignore to reduce build context size and speed up the Docker build a bit
• #162 (closed) Added EOS report logs to the output logs (was just a quick update of the EOS report path)
• Fixed some inconsistencies related to kubectl commands. E.g. changed namespace=... to namespace ... and removed redundant -i/-t flags from kubectl exec where they are not necessary
• Fixed a bug in the repack system test where the vid of the tapes in one of the tests was still hardcoded
• #890 (closed) allow Helm to spawn multiple tape servers and make this fully configurable

Note that this also changes the spawned configuration ever so slightly; it will spawn however many tape servers as there are drives available on the machine. If we want, we can put a config in place in all the cirunner machines to only spawn e.g. 2, but I don't think this should significantly change things.

Additional Required Actions

• Requires manual tests in pre-production: NO
• Requires a documentation update: YES: https://gitlab.cern.ch/cta/eoscta-docs/-/issues/16
• This requires an update of this RunDeck job as the location of the Oracle configuration file has changed: https://rundeck-tape.cern.ch/project/CTA_operations/job/show/71c541fb-e673-41b8-b5a5-eebff1f6de63

References

Closes #892 (closed) #901 (closed) #897 (closed) #888 (closed) #900 (closed) #902 (closed) #893 (closed) #878 (closed) #906 (closed) #833 (closed) #162 (closed) #917 (closed) #910 (closed) #898 (closed) #890 (closed)