[maintenance] Implement job batching for Queue Cleanup Runner
Description
-
The
RetrieveJobToAdddata structure has been moved from the RetrieveQueue class to common::dataStructures. This allows the usage of the struct outside of the RetrieveQueue. This is needed for the batching and is what causes most of the line changes of the MR. The interesting bits are in theOStoreDB.cppandRetrieveRequest.cpp -
3 new methods added to
RetrieveRequestclass:-
reclassifyRetrieveRequest: determine if the retrieve request can be requeued to a different VID. Returns astd::optional:nulloptif it cannot be requeued orstd::stringwith the VID to requeue. -
failJob: populate the necessary fields of the job to be reported and commit to the OS. -
getJobToAdd: returns the Job data structure to be added to a queue.
-
-
Modified
OStoreDB::requeueRetrieveRequestJobsto first decide what to do with each job and then operate in batches. We cover the case where not all the jobs for a tape with multiple copie have an additional copy available.
The global behaviour is the same as the garbageCollectRetrieveRequest but split into different stages so that we can group the requests together. This new stages only are executed for the cleanup of the queue any other behaviour of the system remains the same.
Checklist
-
Documentation reflects the changes made. -
Merge Request title is clear, concise, and suitable as a changelog entry. See this link
References
Closes #1174 (closed)