[maintenance] Implement job batching for Queue Cleanup Runner
Description
-
The
RetrieveJobToAdd
data structure has been moved from the RetrieveQueue class to common::dataStructures. This allows the usage of the struct outside of the RetrieveQueue. This is needed for the batching and is what causes most of the line changes of the MR. The interesting bits are in theOStoreDB.cpp
andRetrieveRequest.cpp
-
3 new methods added to
RetrieveRequest
class:-
reclassifyRetrieveRequest
: determine if the retrieve request can be requeued to a different VID. Returns astd::optional
:nullopt
if it cannot be requeued orstd::string
with the VID to requeue. -
failJob
: populate the necessary fields of the job to be reported and commit to the OS. -
getJobToAdd
: returns the Job data structure to be added to a queue.
-
-
Modified
OStoreDB::requeueRetrieveRequestJobs
to first decide what to do with each job and then operate in batches. We cover the case where not all the jobs for a tape with multiple copie have an additional copy available.
The global behaviour is the same as the garbageCollectRetrieveRequest
but split into different stages so that we can group the requests together. This new stages only are executed for the cleanup of the queue any other behaviour of the system remains the same.
Checklist
-
Documentation reflects the changes made. -
Merge Request title is clear, concise, and suitable as a changelog entry. See this link
References
Closes #1174 (closed)