MooreOnline_hlt2_pp_2024_data throughput unstable

MooreOnline_hlt2_pp_2024_data test yields unstable throughput.

The issue was noticed last Monday 30th Sep and seen in a number of not related MRs, e.g.:

• 7 Oct Rec!4040 (comment 8550034)
• 3 Oct Moore!3889 (comment 8509052) - also the largest drop -3.60%
• 30 Sep Moore!2002 (comment 8499696) - this one was repeated and fluctuated -1.99% to -0.06%

The bar plot of % fraction participation varies quite a bit from test to test, also in actual 2024-patches night builds.

• E.g. RICH for above tests is 10.33, 10.42, 10.37, while in lhcb-2024-patches.151 10.24, in lhcb-2024-patches.157 10.28.

Selected RICH as example as it seems to be above the reference in all 3 mentioned MRs but the sample is too low to say of a general trend.

Feel free to edit the entry/assign more experts.

assigned to @msaur and @pkopciew

changed the description

@cagapopo Do you remember merging anything what could have such an effect or could make throughput tests unstabel?

@ahennequ I think you are the last one who took a serious look on all the throughput tests we have. Any ideas what to check?

That is indeed surprising. I compared the logs in a fluctuation of one the MR and noticed that not only the throughput is different but the total number of processed events changed. I'm wondering if at the slow speed of HLT2 this difference in which events are processed could make a visible difference on the throughput. If it is the case we might want to setup the MooreOnline HLT2 test to have a fixed number of events instead of a set amount of time. @raaij what do you think ? is it even possible ?

For an exact number of events, the Online component that reads from disk - I think it's DiskReader - would probably need some changes to allow this to be configured. Otherwise the testbench could check if the counter that monitors the number of processed events has passed some limit and then stop running. That will not be exact.

Any progress on this? Makes evaluating some MRs rather problematic.

mentioned in issue Moore#863 (closed)

mentioned in merge request Moore!3795 (closed)

mentioned in issue Moore#866 (closed)

assigned to @zejia

unassigned @zejia

assigned to @zejia

unassigned @zejia

mentioned in issue Moore#870 (closed)

mentioned in issue Moore#877