Skip to content
GitLab
Projects Groups Topics Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
  • Sign in
  • athena athena
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributor statistics
    • Graph
    • Compare revisions
    • Locked files
  • Jira
    • Jira
  • Merge requests 136
    • Merge requests 136
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
  • Deployments
    • Deployments
    • Releases
  • Packages and registries
    • Packages and registries
    • Container Registry
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Code review
    • Insights
    • Repository
  • Activity
  • Graph
  • Jobs
  • Commits
Collapse sidebar
  • atlasatlas
  • athenaathena
  • Merge requests
  • !35253

Byte stream output should read from MetaDataStore

  • Review changes

  • Download
  • Email patches
  • Plain diff
Merged Frank Berghaus requested to merge berghaus/athena:ByteStreamOutput into master Jul 30, 2020
  • Overview 12
  • Commits 6
  • Pipelines 1
  • Changes 9

@tsulaia pointed out that when we write raw files the we grab run parameters from the input metadata store. That is dangerous as that store is cleared each time a new input file is opened. This prompted me to work over our ByteStreamEventStorageOutputSvc.

First we nor grab the run parameters from the metadata store filled by the metadata tools. As @wlampl suggested, a warning is raised when multiple copies of the ByteStreamMetaData are found. The run parameters from the first object are selected. This is a limitation of the bytesatream format.

Run parameters are used from sources multiple sources with the following priority:

  1. The job configuration
  2. The xAOD::EventInfo
  3. The ByteStreamMetaData
  4. The ProcessingTags
  5. defaults The information is needed to set up the data writer, either on the first putEvent that is called or when MetaDataStop is called to write out an event-less file. To avoid a large number of logic paths the parameters are taking in reverse order: job configuration parameters overwrite the content the parameters set by xAOD::EventInfo, xAOD::EventInfo overwrites those from the in-file metadata, and so on. The component accumulator configuration has been adapted such that the xAOD::EventInfo, ByteStreamMetaData and ProcessingTags should be available when the job is configured for bytestream file output. Objects are retrieved via ReadHandles to help the scheduler decide when the bytestream output has the information it needs,

Events to be written are cached in a slot specific object for write out. The cache itself my be superfluous (let me know your thoughts). I could not think my way out of two things that need locking: the initialization of the DataWriter and the writing of events to file. Since the two things happen one right after the other there is a single lock making making sure only one thread does this at a time --- should we ever end up in a scenario where this service is called by a threaded output algorithm.

@gemmeren @tadej

Edited Oct 01, 2020 by Frank Berghaus
Assignee
Assign to
Reviewers
Request review from
Time tracking
Source branch: ByteStreamOutput