Skip to content

Add support for independent skipping of secondary events in DoubleEventSelector

Tadej Novak requested to merge tadej/athena:des into master

In bulk production tests we realised that independent skipping of secondary events in overlay might be beneficial for production when HS hits are large (e.g. 10k events/file) but RDOs are smaller. At the moment only the whole HITS file can be processed at once.

A separate --skipSecondaryEvents argument is added which works in the following way:

  • no flag -> same skipping as HS
  • skipSecondaryEvents=0 -> disable skipping of RDOs
  • skipSecondaryEvents=N -> independently skip N events in the first RDO file

Counters are independent for each selector (not really used anywhere at the moment). Helper tools skip both events at once (also not used for overlay).

Open questions:

  • Should skipSecondaryEvents be global argument/flag but just not used in other workflows? For now it's overlay specific.
  • Logging is a bit misleading as HITS are actually using the secondary selector but using the primary skipEvents argument.
  • Seeking still needs to be updated but will probably come in a separate MR.

Fixes ATLASSIM-5376.

/cc @mnowak @gemmeren @jchapman @mduehrss

Edited by Tadej Novak

Merge request reports