CI database cannot be wiped in some cases
Issue
See this job: https://gitlab.cern.ch/cta/CTA/-/jobs/30959036#L80
Here are the logs:
Configuring database:
Configuring oracle database
Wiping database
Aborting: isProductionSet failed: executeQuery failed: executeQuery failed for SQL statement SELECT CTA_CATALOGUE.IS_PRODUCTION AS
IS_PRODUCTION FROM CTA_CATALOGUE: ORA-00942: table or view does not exist
Database connection failed, pausing before a retry
Aborting: isProductionSet failed: executeQuery failed: executeQuery failed for SQL statement SELECT CTA_CATALOGUE.IS_PRODUCTION AS
IS_PRODUCTION FROM CTA_CATALOGUE: ORA-00942: table or view does not exist
ERROR: Could not wipe database. cta-catalogue-schema-drop /etc/cta/cta-catalogue.conf FAILED
ERROR: init pod in ErERROR: init pod in Error state. Initialization failed.
145
FAILURE: cleaning up environment
In this situation the DB cannot be wiped on this runner anymore and all jobs will fail until the DB is entirely cleared manually...
This situation could have been created by a job cancellation at a very bad moment (DB empty, partially cleared...).
Possible fix
isProductionSet
should return false if the database or table does not exist. Indeed if the schema is incomplete there is no way this database could be in production.
Once this check is fine we need to check that DB is fine.
It should be easy to reproduce deleting this specific table (or starting from an empty schema).