Problem with m_maxBatchBytes logic in RecallTaskInjector
In RecallTaskInjector::synchronousFetch()
, when we use the default m_maxBatchBytes
value = 80 GB, we fetch not the whole queue RetrieveQueueToTransferForUser-<TapeVID>
, but only a chunk of 80 GB.
auto jobsList = m_retrieveMount.getNextJobBatch(reqFiles, reqSize, m_lc);
for (auto & j: jobsList) {
m_files++;
m_bytes += j->archiveFile.fileSize;
m_jobs.emplace_back(j.release());
}
Then in RecallTaskInjector::injectBulkRecalls()
we consume and erase 80 GB of m_jobs
.
m_jobs.erase(std::remove_if(m_jobs.begin(), m_jobs.end(), [](const std::unique_ptr<cta::RetrieveJob> &jobptr) {
return jobptr.get() == nullptr;
}), m_jobs.end());
That leaved m_jobs
empty and leads to end of session:
if (m_parent.m_jobs.empty()) {
if (req.lastCall) {
m_parent.m_lc.log(cta::log::INFO,"No more file to recall: triggering the end of session.");
m_parent.signalEndDataMovement();
break;
} else {
m_parent.m_lc.log(cta::log::DEBUG,"In RecallJobInjector::WorkerThread::run(): got empty list, but not last call. NoOp.");
}
}
In this case m_parent.injectBulkRecalls()
is run only twice, so we can process only 160 GB of data.
But in the RetrieveQueue
there are more jobs, causing more tape mounts than needed.
Edited by Volodymyr Yurchenko