Risk of database deadlock during bulk insert to the catalogue
It is believed that there is a risk of deadlock during bulk insert into the CTA catalogue.
For example, the appropriate specialisation of the method Catalogue::filesWrittenToTape
records a set of files, all of which have been stored on the same tape. This method may add rows to the tables ARCHIVE_FILE
and TAPE_FILE
(amongst others). A problematic case is believed to be when to two or more files are being written to different tapes at the same time, e.g. files with multiple tape copies. The cta-taped
daemons will have transactions open to the catalogue inserting similar rows to ARCHIVE_FILE
, potentially in different orders. This may give a deadlock, which is expected to be resolved by the database by having one of the transactions fail.
A test to simulate this case and demonstrate database deadlocks was added to CatalogueTest
as a part of the Postges commit (issue #355, commit CTA@02be202e56d8238185df4e19db1be78c324e8108) called concurrent_filesWrittenToTape_many_archive_files
however it is currently disabled because:
-
The
CatalogueTest
is run as a part of the unit tests using theInMemoryCatalogue
(Sqlite in memory database). TheInMemoryCatalgoue
currently would fail the test because it includes the creation of the schema as a part of the catalogue setup, and so fails independently of any deadlock issues because the test involves creating two catalogue instances. -
It is not clear if this issue should actually be solved by attempting to eliminate database deadlocks, or if it is sufficient that the system is robust against catalogue transaction failures. This decision is expected to depend on complexity of avoiding deadlocks, the frequency with which they would happen and costs associated with recovering in this situation.