Draft: Introduce pipeline synchronization, Parser and Snooper filters
This branch includes improvements to synchronization of multiple pipelines and eventual writing of EoLS/EoR files by the main pipeline, which has been successfully deployed in P5 for weeks now.
It also contains more recent changes to the input stages of processing by introducing two new pipeline filters: InputParser
and InputSnooper.
The former takes much responsibility from OrbitProcessor
, parsing orbit-containing packet contents into more maintainable structures, which can then be optionally exported as JSON objects in the latter new filter. The same parsing is then leveraged by the OrbitProcessor::ProcessSliceImpl
and OrbitProcessor::FillOrbit
methods, which can now focus on actually processing. This allows flexibility for more interesting batching strategies since all data is now known ahead of processing. The Snooper
outputs the JSON objects to a JSON file; its outputs can be streamed in future to DQM dashboards and monitoring systems.
The branch is in draft mode because some kinks still need work, particularly on the {Orbit}InputWrapper
structure, its pipeline filter functor handover and its memory management (which could use a circular buffer strategy akin to Slice
to prevent reallocation).
We can prioritize the merging of the pipeline synchronization work and the introduction of the new filters, while the functional changes to the OrbitProcessor
can wait (as they are still fresh and less understood). Regardless, I am including all the work here for context.