Skip to content

Draft: Introduce pipeline synchronization, Parser and Snooper filters

This branch includes improvements to synchronization of multiple pipelines and eventual writing of EoLS/EoR files by the main pipeline, which has been successfully deployed in P5 for weeks now.

It also contains more recent changes to the input stages of processing by introducing two new pipeline filters: InputParser and InputSnooper. The former takes much responsibility from OrbitProcessor, parsing orbit-containing packet contents into more maintainable structures, which can then be optionally exported as JSON objects in the latter new filter. The same parsing is then leveraged by the OrbitProcessor::ProcessSliceImpl and OrbitProcessor::FillOrbit methods, which can now focus on actually processing. This allows flexibility for more interesting batching strategies since all data is now known ahead of processing. The Snooper outputs the JSON objects to a JSON file; its outputs can be streamed in future to DQM dashboards and monitoring systems.

The branch is in draft mode because some kinks still need work, particularly on the {Orbit}InputWrapper structure, its pipeline filter functor handover and its memory management (which could use a circular buffer strategy akin to Slice to prevent reallocation).

We can prioritize the merging of the pipeline synchronization work and the introduction of the new filters, while the functional changes to the OrbitProcessor can wait (as they are still fresh and less understood). Regardless, I am including all the work here for context.

Merge request reports

Loading