Introduce pipeline synchronization, output template and parallel sink
This branch includes improvements to synchronization of multiple pipelines and eventual writing of EoLS/EoR files by the main pipeline, which has been successfully deployed in P5 for weeks months now. It also introduces a templated sink managing an asynchronous TBB queue, which commits "files" of a given output type (which currently can only be a POSIX file, to be joined by several DAX-based output modules).
Previously, this MR contained the new intermediate filters InputParser and InputSnooper: the first would parse orbit data into a clearly defined structure before any processing and hand it over to the next stage (this is currently done in the processing step); the second filter would optionally store the parsed data in a file for debugging / monitoring purposes. These changes will become a follow-up MR because they're less crucial at this moment. Empirically, the Parser incurred a small hit on performance because of this decoupling from the processor. Although negligible, this needs to be replicated, re-measured and better understood.
Below the original description of these filters when the MR was a draft:
It also contains more recent changes to the input stages of processing by introducing two new pipeline filters:
InputParserandInputSnooper.The former takes much responsibility fromOrbitProcessor, parsing orbit-containing packet contents into more maintainable structures, which can then be optionally exported as JSON objects in the latter new filter. The same parsing is then leveraged by theOrbitProcessor::ProcessSliceImplandOrbitProcessor::FillOrbitmethods, which can now focus on actually processing. This allows flexibility for more interesting batching strategies since all data is now known ahead of processing. TheSnooperoutputs the JSON objects to a JSON file; its outputs can be streamed in future to DQM dashboards and monitoring systems.The branch is in draft mode because some kinks still need work, particularly on the
{Orbit}InputWrapperstructure, its pipeline filter functor handover and its memory management (which could use a circular buffer strategy akin toSliceto prevent reallocation).We can prioritize the merging of the pipeline synchronization work and the introduction of the new filters, while the functional changes to the
OrbitProcessorcan wait (as they are still fresh and less understood). Regardless, I am including all the work here for context.