Skip to content

Update scheduler to use file creation time provided by frontend

Current CTA scheduler implementation sorts the request according their submission time. This approach works well when EOS is used as the frontend storage system: the scheduler creation timestamp matches file's creation time. However, when CTA is used with dCache, the scheduler creation time matches only timestamp at which dCache have decided to flush the file. However, when flushing the scheduler still should respect file creation time. Moreover, as single dCache pool might submit multiple requests at once, then multiple cta-taped processes might decide to flush files from a single host, thus stature the network or disk IO on that host.

To overcome this limitation, the scheduler interface should be update to accept file's creation time provided by front-end and use it when request selection is made.

It looks like that the most of CTA code is already prepared for that, except two small steps:

  • update DiskFileInfo to include creation timestamp
  • update implementations of SchedulerDatabase#queueArchive to use file creation timestamp when populating ArchiveFile#creationTime field

If no objections, we are happy contribute this changes.