Skip to content

[maintenance] Implement job batching for Queue Cleanup Runner

Pablo Oliver Cortes requested to merge 1174-qcr-job-batching into main

Description

  • The RetrieveJobToAdd data structure has been moved from the RetrieveQueue class to common::dataStructures. This allows the usage of the struct outside of the RetrieveQueue. This is needed for the batching and is what causes most of the line changes of the MR. The interesting bits are in the OStoreDB.cpp and RetrieveRequest.cpp

  • 3 new methods added to RetrieveRequest class:

    • reclassifyRetrieveRequest: determine if the retrieve request can be requeued to a different VID. Returns a std::optional: nullopt if it cannot be requeued or std::string with the VID to requeue.
    • failJob: populate the necessary fields of the job to be reported and commit to the OS.
    • getJobToAdd: returns the Job data structure to be added to a queue.
  • Modified OStoreDB::requeueRetrieveRequestJobs to first decide what to do with each job and then operate in batches. We cover the case where not all the jobs for a tape with multiple copie have an additional copy available.

The global behaviour is the same as the garbageCollectRetrieveRequest but split into different stages so that we can group the requests together. This new stages only are executed for the cleanup of the queue any other behaviour of the system remains the same.

Checklist

  • Documentation reflects the changes made.
  • Merge Request title is clear, concise, and suitable as a changelog entry. See this link

References

Closes #1174 (closed)

Edited by Pablo Oliver Cortes

Merge request reports

Loading