CI tests fail due to DB backends being wiped out too often
CI job failed with the frontend complaining about DB table or view not being available [1]. This is due to the fact that we are calling the run_systemtest.sh
script having hardcoded options -D -O
flags for the create_instance.sh
script, which will wipe out the contents for both backends, the catalogue and the scheduler backend for that particular cirunner DB account. When 2 tests runs on the same machine with the same DB account, depending on who wipes out the DB first it determined to die first as well.
Possible solutions:
- probe if we can run tests in parallel on the un-wiped backends, if so wipe out the DB only once the pipeline has finished (whatever state it did finish in)
- run tests sequentially per runner and keep the systematic DB wiping in place
[1]
240927 14:30:54 ^[[49;31mERROR^[[0m WFE:2878 protoWFEndPoint="ctafrontend:10955" protoWFResource="/ctafrontend" fullPath="/eos/ctaeos/preprod/19509cbd-0702-4a21-8313-8c0a84c7d701/0/02768" event="sync::prepare" msg="Received an error response" response="RSP_ERR_CTA" reason="executeQuery failed: executeQuery failed for SQL statement
SELECT
DISK_SYSTEM.DISK_SYSTEM_NAME AS DISK_SYSTEM_NAME,
DI...: ORA-00942: table or view does not exist
"
240927 14:30:54 ^[[49;32mINFO ^[[0m WFE:2809 protoWFEndPoint="ctafrontend:10955" protoWFResource="/ctafrontend" fullPath="/eos/ctaeos/preprod/19509cbd-0702-4a21-8313-8c0a84c7d701/0/02763" event="sync::prepare" timeSpentMs=65 msg="Sent SSI protocol buffer request"
240927 14:30:54 ^[[49;31mERROR^[[0m WFE:2878 protoWFEndPoint="ctafrontend:10955" protoWFResource="/ctafrontend" fullPath="/eos/ctaeos/preprod/19509cbd-0702-4a21-8313-8c0a84c7d701/0/02763" event="sync::prepare" msg="Received an error response" response="RSP_ERR_CTA" reason="executeQuery failed: executeQuery failed for SQL statement
SELECT
DISK_SYSTEM.DISK_SYSTEM_NAME AS DISK_SYSTEM_NAME,
DI...: ORA-00942: table or view does not exist
"
240927 14:30:54 ^[[49;32mINFO ^[[0m WFE:2809 protoWFEndPoint="ctafrontend:10955" protoWFResource="/ctafrontend" fullPath="/eos/ctaeos/preprod/19509cbd-0702-4a21-8313-8c0a84c7d701/0/02757" event="sync::prepare" timeSpentMs=50 msg="Sent SSI protocol buffer request"
240927 14:30:54 ^[[49;31mERROR^[[0m WFE:2878 protoWFEndPoint="ctafrontend:10955" protoWFResource="/ctafrontend" fullPath="/eos/ctaeos/preprod/19509cbd-0702-4a21-8313-8c0a84c7d701/0/02757" event="sync::prepare" msg="Received an error response" response="RSP_ERR_CTA" reason="executeQuery failed: executeQuery failed for SQL statement
SELECT
DISK_SYSTEM.DISK_SYSTEM_NAME AS DISK_SYSTEM_NAME,
DI...: ORA-00942: table or view does not exist
"
240927 14:30:54 ^[[49;32mINFO ^[[0m WFE:2809 protoWFEndPoint="ctafrontend:10955" protoWFResource="/ctafrontend" fullPath="/eos/ctaeos/preprod/19509cbd-0702-4a21-8313-8c0a84c7d701/0/02756" event="sync::prepare" timeSpentMs=60 msg="Sent SSI protocol buffer request"
240927 14:30:54 ^[[49;31mERROR^[[0m WFE:2878 protoWFEndPoint="ctafrontend:10955" protoWFResource="/ctafrontend" fullPath="/eos/ctaeos/preprod/19509cbd-0702-4a21-8313-8c0a84c7d701/0/02756" event="sync::prepare" msg="Received an error response" response="RSP_ERR_CTA" reason="executeQuery failed: executeQuery failed for SQL statement
SELECT
DISK_SYSTEM.DISK_SYSTEM_NAME AS DISK_SYSTEM_NAME,
DI...: ORA-00942: table or view does not exist
"
240927 14:30:54 ^[[49;32mINFO ^[[0m WFE:1616 default SYNC::PREPARE /eos/ctaeos/preprod/19509cbd-0702-4a21-8313-8c0a84c7d701/0/02792 ctafrontend:10955 fxid=00001e74 mgm.reqid="57257e3e-7ccc-11ef-877d-fef9e2dbc1a2"
240927 14:30:54 ^[[49;32mINFO ^[[0m WFE:2809 protoWFEndPoint="ctafrontend:10955" protoWFResource="/ctafrontend" fullPath="/eos/ctaeos/preprod/19509cbd-0702-4a21-8313-8c0a84c7d701/0/02792" event="sync::prepare" timeSpentMs=57 msg="Sent SSI protocol buffer request"
240927 14:30:54 ^[[49;31mERROR^[[0m WFE:2878 protoWFEndPoint="ctafrontend:10955" protoWFResource="/ctafrontend" fullPath="/eos/ctaeos/preprod/19509cbd-0702-4a21-8313-8c0a84c7d701/0/02792" event="sync::prepare" msg="Received an error response" response="RSP_ERR_CTA" reason="executeQuery failed: executeQuery failed for SQL statement
SELECT
DISK_SYSTEM.DISK_SYSTEM_NAME AS DISK_SYSTEM_NAME,
DI...: ORA-00942: table or view does not exist
"
240927 14:30:54 ^[[49;32mINFO ^[[0m WFE:1616 default SYNC::PREPARE /eos/ctaeos/preprod/19509cbd-0702-4a21-8313-8c0a84c7d701/0/02762 ctafrontend:10955 fxid=00001e4e mgm.reqid="572edea2-7ccc-11ef-83e5-fef9e2dbc1a2"
240927 14:30:54 ^[[49;32mINFO ^[[0m WFE:1616 default SYNC::PREPARE /eos/ctaeos/preprod/19509cbd-0702-4a21-8313-8c0a84c7d701/0/02776 ctafrontend:10955 fxid=00001ec8 mgm.reqid="57307230-7ccc-11ef-8b9d-fef9e2dbc1a2"
240927 14:30:54 ^[[49;32mINFO ^[[0m WFE:1616 default SYNC::PREPARE /eos/ctaeos/preprod/19509cbd-0702-4a21-8313-8c0a84c7d701/0/02766 ctafrontend:10955 fxid=00001e44 mgm.reqid="57324df8-7ccc-11ef-a238-fef9e2dbc1a2"
240927 14:30:54 ^[[49;32mINFO ^[[0m WFE:2809 protoWFEndPoint="ctafrontend:10955" protoWFResource="/ctafrontend" fullPath="/eos/ctaeos/preprod/19509cbd-0702-4a21-8313-8c0a84c7d701/0/02762" event="sync::prepare" timeSpentMs=28 msg="Sent SSI protocol buffer request"
240927 14:30:54 ^[[49;31mERROR^[[0m WFE:2878 protoWFEndPoint="ctafrontend:10955" protoWFResource="/ctafrontend" fullPath="/eos/ctaeos/preprod/19509cbd-0702-4a21-8313-8c0a84c7d701/0/02762" event="sync::prepare" msg="Received an error response" response="RSP_ERR_CTA" reason="executeQuery failed: executeQuery failed for SQL statement
SELECT
DISK_SYSTEM.DISK_SYSTEM_NAME AS DISK_SYSTEM_NAME,
``` 81953,1 91%
Edited by Jaroslav Guenther