Redefine the global event cut to not use UT information

This removes the use of UT information from the global event cut. The default cut value is not changed which as a byproduct means that the GEC in Allen will become more of a "sanity" cut removing only the really few very highest occupancy events, which probably goes in the direction we want to go long-term anyway ( @mvesteri @decianm ). This will be a mess of reference updates and I may well have forgotten something but otherwise is ready for review @dovombru @dcampora @raaij

With this MR, the following slowdown is observed due to loosening the GEC:

Device                        Throughput (kHz)    Reference Throughput (kHz)  Speedup    % change    Status
--------------------------  ------------------  ----------------------------  ---------  ----------  ---------
NVIDIA RTX A5000                        136.13                        149.7   0.91x      -9.07%      DECREASED
AMD EPYC 7502 32-Core                    16.72                         17.27  0.97x      -3.20%      OK
NVIDIA GeForce RTX 2080 Ti              110.87                        120.8   0.92x      -8.22%      DECREASED
NVIDIA GeForce RTX 3090                 176.63                        194.07  0.91x      -8.98%      DECREASED
Device-averaged speedup: 0.93x
               % change: -7.37%
                 status: DECREASED

assigned to @gligorov

added RTA label

Could you please apply the formatting?

added 1 commit

• dcf2127f - Fixed formatting

Compare with previous version

added hlt1-throughput-decreased label

added needs ref update label

/ci-test

added ci-test-triggered label

• [2022-07-31 10:26] Validation started with lhcb-master-mr#5275

removed hlt1-throughput-decreased label

approved this merge request

@gligorov on the Allen gec becoming more of a "sanity" cut, is the idea that this should now be loose enough for EW/production physics?

added 34 commits

• dcf2127f...2288eabf - 32 commits from branch master
• b4ba5480 - redefine the global event cut to not use UT information
• 98fb1e2d - Fixed formatting

Compare with previous version

@dcampora I triggered a rebase since this one needs so many ref updates (even if it factorizes from I guess every other MR quite cleanly) but I am not really sure how you want to proceed here. Just let me know if I need to do anything more.

added hlt1-throughput-decreased label

removed hlt1-throughput-decreased label

mentioned in issue Moore#465 (closed)

added 34 commits

• 98fb1e2d...0f79208d - 32 commits from branch master
• 32ba29f6 - redefine the global event cut to not use UT information
• 60834bb6 - Fixed formatting

Compare with previous version

added hlt1-throughput-decreased label

added 32 commits

• 60834bb6...d075cd5d - 30 commits from branch master
• a82f8e48 - redefine the global event cut to not use UT information
• f6157685 - Fixed formatting

Compare with previous version

added 30 commits

• f6157685...1a98d951 - 28 commits from branch master
• bd4052f2 - redefine the global event cut to not use UT information
• 91fddffa - Fixed formatting

Compare with previous version

/ci-test --merge

• [2022-08-20 11:43] Validation started with lhcb-master-mr#5459
• [2022-09-03 19:02] Validation started with lhcb-master-mr#5567

resolved all threads

changed the description

assigned to @raaij and unassigned @gligorov

mentioned in issue Moore#467 (closed)

added 5 commits

• a31aa016 - Remove HostBuffers::host_number_of_selected_events
• 8e80b63d - Better initialization
• 5f9fceff - Remove obsolete testing code
• 3f9cdfd0 - GEC works when banks are missing
• 7a1343db - Refactor GEC into separate algorithms for SciFi and UT

Compare with previous version

added 57 commits

• 7a1343db...7a0e15a7 - 49 commits from branch master
• 6aabc459 - redefine the global event cut to not use UT information
• d23ec3b0 - Fixed formatting
• e1fa143a - Remove HostBuffers::host_number_of_selected_events
• e208179f - Better initialization
• 32597511 - Remove obsolete testing code
• 25fbf4c7 - GEC works when banks are missing
• 451a17bd - Refactor GEC into separate algorithms for SciFi and UT
• 86e939ae - Fix rebase

Compare with previous version

added 1 commit

• f88b9efd - Refactor GEC into separate algorithms for SciFi and UT

Compare with previous version

added 1 commit

• 362e5d25 - No UT in matching sequence

Compare with previous version

removed hlt1-throughput-decreased label

mentioned in issue Moore#474 (closed)

added 6 commits

• b4b1dfc0 - Fix output writing when SelReports are absent
• 335c9908 - Add sequence for calo activity with lumi banks
• 5016e4f5 - Better error handling in Allen event loop
• b737c6ca - Fixed formatting
• 63ab274a - Patch PyConf to try pydotplus if pydot is not available
• 62adce8b - Remove explicit use of host_event_list from GEC algorithms

Compare with previous version

added 1 commit

• 04f4e4c5 - Remove explicit use of host_event_list from GEC algorithms

Compare with previous version

added hlt1-throughput-decreased label

• [2022-09-07 01:50] Validation started with lhcb-master-mr#5584

removed hlt1-throughput-decreased label

added 1 commit

• a4ed1f93 - Update Allen CI references

Compare with previous version

Allen CI references have been updated.

Throughput reductions are roughly as expected for the default sequence:

sequence: hlt1_pp_default
dataset:  SciFiv6_upgrade_DC19_01_MinBiasMD_retinacluster_v1

Device                        Throughput (kHz)    Reference Throughput (kHz)  Speedup    % change    Status
--------------------------  ------------------  ----------------------------  ---------  ----------  ---------
NVIDIA RTX A5000                        141.91                        150.37  0.94x      -5.62%      OK
AMD EPYC 7502 32-Core                    16.9                          17.9   0.94x      -5.58%      OK
NVIDIA GeForce RTX 2080 Ti              115.01                        121.09  0.95x      -5.02%      OK
NVIDIA GeForce RTX 3090                 170.06                        198.14  0.86x      -14.17%     DECREASED
Device-averaged speedup: 0.92x
               % change: -7.60%
                 status: DECREASED

The matching sequences are affected similarly:

sequence: hlt1_pp_matching
dataset:  SciFiv6_upgrade_DC19_01_MinBiasMD_retinacluster_v1

Device                        Throughput (kHz)    Reference Throughput (kHz)  Speedup    % change    Status
--------------------------  ------------------  ----------------------------  ---------  ----------  ---------
NVIDIA RTX A5000                        148.42                        158.52  0.94x      -6.37%      OK
AMD EPYC 7502 32-Core                    15.08                         16.46  0.92x      -8.36%      DECREASED
NVIDIA GeForce RTX 2080 Ti              114.25                        120.29  0.95x      -5.01%      OK
NVIDIA GeForce RTX 3090                 188.65                        200.11  0.94x      -5.73%      OK
Device-averaged speedup: 0.94x
               % change: -6.37%
                 status: DECREASED

added hlt1-throughput-decreased label

mentioned in commit Moore@66188963

The RefBot pipeline created the following reference update MRs: Moore!1767 (merged)

Click this to see encountered General warnings

Some MRs have changed since the launch of the ci-test slot. We will continue with the ref update. Correctness not guaranteed!

Click this to see encountered Alignment warnings

Alignment/x86_64_v2-centos7-clang12-opt: Test without ref timed out: Humboldt.align-vp-halves-prkalman-modules!

Click this to see encountered Moore warnings

Moore/x86_64_v3-centos7-gcc11-opt+g: Test job did not finish
Moore: Warning, results for test: RecoConf.hlt2_protoparticles_baseline not found for platforms: {'x86_64_v2-centos7-clang12-opt', 'x86_64_v2-centos7-gcc11-dbg'}
Moore: Warning, results for test: RecoConf.hlt2_protoparticles_fastest not found for platforms: {'x86_64_v2-centos7-clang12-opt', 'x86_64_v2-centos7-gcc11-dbg'}

Click this to see encountered MooreAnalysis warnings

MooreAnalysis/x86_64_v2-centos7-gcc11-opt: Test without ref timed out: HltEfficiencyChecker.test_hlt1_allen_wizard_effs!

mentioned in merge request Moore!1767 (merged)

resolved all threads

merged

mentioned in commit 960eec07

mentioned in commit Moore@d148b64a