Skip to content

Problem with m_maxBatchBytes logic in RecallTaskInjector

In RecallTaskInjector::synchronousFetch(), when we use the default m_maxBatchBytes value = 80 GB, we fetch not the whole queue RetrieveQueueToTransferForUser-<TapeVID>, but only a chunk of 80 GB.

    auto jobsList = m_retrieveMount.getNextJobBatch(reqFiles, reqSize, m_lc);
    for (auto & j: jobsList) {
      m_files++;
      m_bytes += j->archiveFile.fileSize;
      m_jobs.emplace_back(j.release());
    }

Then in RecallTaskInjector::injectBulkRecalls() we consume and erase 80 GB of m_jobs.

  m_jobs.erase(std::remove_if(m_jobs.begin(), m_jobs.end(), [](const std::unique_ptr<cta::RetrieveJob> &jobptr) {
    return jobptr.get() == nullptr;
  }), m_jobs.end());

That leaved m_jobs empty and leads to end of session:

      if (m_parent.m_jobs.empty()) {
        if (req.lastCall) {
          m_parent.m_lc.log(cta::log::INFO,"No more file to recall: triggering the end of session.");
          m_parent.signalEndDataMovement();
          break;
        } else {
          m_parent.m_lc.log(cta::log::DEBUG,"In RecallJobInjector::WorkerThread::run(): got empty list, but not last call. NoOp.");
        }
      }

In this case m_parent.injectBulkRecalls() is run only twice, so we can process only 160 GB of data.

But in the RetrieveQueue there are more jobs, causing more tape mounts than needed.

Edited by Volodymyr Yurchenko