Skip to content
GitLab
Projects Groups Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
  • Sign in
  • CTA CTA
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
    • Locked Files
  • Issues 135
    • Issues 135
    • List
    • Boards
    • Service Desk
    • Milestones
    • Iterations
  • Merge requests 10
    • Merge requests 10
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Packages and registries
    • Packages and registries
    • Package Registry
    • Container Registry
    • Infrastructure Registry
  • Monitor
    • Monitor
    • Metrics
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Code review
    • Issue
    • Repository
  • Wiki
    • Wiki
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • ctacta
  • CTACTA
  • Issues
  • #150
Closed
Open
Issue created Sep 16, 2022 by Volodymyr Yurchenko@vyurchenOwner

Problem with m_maxBatchBytes logic in RecallTaskInjector

In RecallTaskInjector::synchronousFetch(), when we use the default m_maxBatchBytes value = 80 GB, we fetch not the whole queue RetrieveQueueToTransferForUser-<TapeVID>, but only a chunk of 80 GB.

    auto jobsList = m_retrieveMount.getNextJobBatch(reqFiles, reqSize, m_lc);
    for (auto & j: jobsList) {
      m_files++;
      m_bytes += j->archiveFile.fileSize;
      m_jobs.emplace_back(j.release());
    }

Then in RecallTaskInjector::injectBulkRecalls() we consume and erase 80 GB of m_jobs.

  m_jobs.erase(std::remove_if(m_jobs.begin(), m_jobs.end(), [](const std::unique_ptr<cta::RetrieveJob> &jobptr) {
    return jobptr.get() == nullptr;
  }), m_jobs.end());

That leaved m_jobs empty and leads to end of session:

      if (m_parent.m_jobs.empty()) {
        if (req.lastCall) {
          m_parent.m_lc.log(cta::log::INFO,"No more file to recall: triggering the end of session.");
          m_parent.signalEndDataMovement();
          break;
        } else {
          m_parent.m_lc.log(cta::log::DEBUG,"In RecallJobInjector::WorkerThread::run(): got empty list, but not last call. NoOp.");
        }
      }

In this case m_parent.injectBulkRecalls() is run only twice, so we can process only 160 GB of data.

But in the RetrieveQueue there are more jobs, causing more tape mounts than needed.

Edited Sep 16, 2022 by Volodymyr Yurchenko
Assignee
Assign to
Time tracking