Remove GEC from default HLT1 sequence

Closes #395 (closed)

Renders !664 (closed) and !784 (closed) obsolete

Test with Moore!2631 (merged) and MooreOnline!278 (merged)

@dovombru

Want to remove Global Event Cuts (GEC) from the default HLT1 sequence. All HLT1 would then run without these cuts being implemented at the beginning of data processing.

Also want to study the impact on the selection efficiency for

$Z\rightarrow\mu\mu$

,

$W\rightarrow\mu\nu$

, and other processes impacted by the GEC

Changes:

• Set the default hlt1 node to NOT include GEC (EnableGEC=False in HLT1.py)
• Remove GEC from all sequence files in configuration/python/AllenSequences
• Allow GEC to be configurable in validator_node function

- Added a new function called make_composite_node to replace the currrently used function make_composite_node_with_gec which is featured in a couple of sequence files (is there a better way to do this?) -->Unnecessary

Next plans:

• Study the impact on the selection efficiency for
  $Z\rightarrow\mu\mu$
• Study the impact on the selection efficiency for
  $W\rightarrow\mu\nu$
• Confirm that the impact on the throughput is reasonable
• Remove all instances where we use GECs (except one sequence hlt1_pp_default_gec)
• Adjust throughput tests to run over correct sequences (now without GECs by default)

Throughput:

Latest throughput values obtained from the "AllenPR throughput" mattermost channel:

TL;DR: all throughputs are fine, apart from the one from hlt1_pp_no_ut which is slightly worse than before:

• hlt1_pp_default --> with GECs:

• hlt1_pp_no_gec --> without GECs:

• hlt1_pp_matching --> with GECs:

• hlt1_pp_matching_no_gec --> without GECs and without ut:

• hlt1_pp_no_ut --> without GECs:

---

Efficiencies:

Studied the efficiencies for Z/W decays using HltEfficiencyChecker and 2 yaml files: hlt1_z_eff.yaml hlt1_w_eff.yaml

Can just run the command: MooreAnalysis/run MooreAnalysis/HltEfficiencyChecker/scripts/hlt_eff_checker.py hlt1_w_eff.yaml from the stack home directory

TL;DR: both W and Z efficiencies increase without GEC as expected (0.914 --> 0.941 and 0.935 --> 0.977 respectively)

W before (master branch): w_eff_check_master_branch.txt

Integrated HLT efficiencies for the lines with denominator: CanRecoChildren
------------------------------------------------------------------------------------
Line:	 Hlt1SingleHighPtMuonDecision         	 Efficiency:	 0.914 +/- 0.009
Line:	 W_Hlt1SingleHighPtMuonDecisionTrueSim	 Efficiency:	 0.914 +/- 0.009
------------------------------------------------------------------------------------
Finished printing integrated HLT efficiencies for denominator: CanRecoChildren
------------------------------------------------------------------------------------

W after (this branch): w_eff_check_new_branch.txt

Integrated HLT efficiencies for the lines with denominator: CanRecoChildren
------------------------------------------------------------------------------------
Line:	 Hlt1SingleHighPtMuonDecision         	 Efficiency:	 0.941 +/- 0.008
Line:	 W_Hlt1SingleHighPtMuonDecisionTrueSim	 Efficiency:	 0.940 +/- 0.008
------------------------------------------------------------------------------------
Finished printing integrated HLT efficiencies for denominator: CanRecoChildren
------------------------------------------------------------------------------------

Z before (master branch): z_eff_check_master_branch.txt

Integrated HLT efficiencies for the lines with denominator: CanRecoChildren
------------------------------------------------------------------------------------
Line:	 Hlt1SingleHighPtMuonDecision          	 Efficiency:	 0.935 +/- 0.015
Line:	 Z0_Hlt1SingleHighPtMuonDecisionTrueSim	 Efficiency:	 0.935 +/- 0.015
------------------------------------------------------------------------------------
Finished printing integrated HLT efficiencies for denominator: CanRecoChildren
------------------------------------------------------------------------------------

Z after (this branch): z_eff_check_new_branch.txt

Integrated HLT efficiencies for the lines with denominator: CanRecoChildren
------------------------------------------------------------------------------------
Line:	 Hlt1SingleHighPtMuonDecision          	 Efficiency:	 0.977 +/- 0.009
Line:	 Z0_Hlt1SingleHighPtMuonDecisionTrueSim	 Efficiency:	 0.977 +/- 0.009
------------------------------------------------------------------------------------
Finished printing integrated HLT efficiencies for denominator: CanRecoChildren
------------------------------------------------------------------------------------

I also checked with the default HLTEfficiencyChecker example: MooreAnalysis/run MooreAnalysis/HltEfficiencyChecker/scripts/hlt_eff_checker.py MooreAnalysis/HltEfficiencyChecker/options/hlt1_eff_example.yaml and confirmed that these lines were not affected (as expected): default_eff_check_master_branch.txtdefault_eff_check_new_branch.txt

Efficiencies plots:

Got some plots using HltEfficiencyChecker:

Efficiencies__Hlt1SingleHighPtMuonDecision__CanRecoChildren__PT:

• W before:
• W after:
• Z before:
• Z after:

added RTA label

requested review from @dovombru

assigned to @ahabdelm

added 4 commits

• 07d424b4 - Set the default hlt1 node to NOT include GEC
• 905e7d12 - Added new default sequence with GEC enabled to keep testing it in the nightlies
• 48b19084 - Removed GEC from most sequences (simpler ones)
• 4ef2773f - Added 'make_composite_node' function as a replacement to...

Compare with previous version

mentioned in merge request !664 (closed)

mentioned in merge request !784 (closed)

added 10 commits

• 4ef2773f...4833e6e1 - 9 commits from branch master
• 93e069b7 - Merge branch 'master' into ahabdelm-remove-gec-from-hlt1

Compare with previous version

added 1 commit

• e2296546 - Fix 'make_composite_node' function

Compare with previous version

changed the description

changed the description

@dovombru Hello! Are these tests to the efficiencies and the throughput enough, or do you think we require some more tests?

In terms of physics efficiencies it would be interesting to see where in phase space we gain, i.e. showing some of the plots produced with the HLTEfficiency Checker.

Are the gains as expected @mvesteri ? Especially for the W?

added 35 commits

• e2296546...12fc92de - 34 commits from branch master
• dc5e88f5 - Merge branch 'master' into ahabdelm-remove-gec-from-hlt1

Compare with previous version

added 3 commits

• 166c45e4 - removed deprecated sequence file as part of cleanup
• e5603a9a - removed redundant function
• 5c81ec7d - Added 'hlt1_pp_matching_no_gec' to full CI gitlab pipeline to test throughput

Compare with previous version

added 1 commit

• ed4fa9f3 - fix minor mistakes

Compare with previous version

added hlt1-throughput-decreased label

Hi @dovombru. Sorry, this has been going at a snail's pace. I implemented your suggestions and was trying to get the throughput for hlt1_pp_matching_no_gec by including it to the full gitlab CI pipeline like you suggested. I made my attempt here by including hlt1_pp_matching_no_gec to the list of other sequences we use for throughputs. This doesn't seem to work. Gitlab doesn't produce the throughput for this sequence. I tried asking a few people, but no one knows the answer. Can you please suggest where I went wrong? Thanks :)

added 132 commits

• ed4fa9f3...d8fe0870 - 130 commits from branch master
• 298b6d30 - Merge branch 'master' into ahabdelm-remove-gec-from-hlt1
• 3500caa4 - Add throughput test addition in the correct file

Compare with previous version

removed hlt1-throughput-decreased label

added hlt1-throughput-decreased label

changed the description

@dovombru Okay I think managed to do everything you requested. The only thing worth noting is the fact that we get a slight decrease of the throughput for the hlt1_pp_no_ut sequence. Not sure if this to be expected. You have all the throughput test results as well as the efficiency results with some plots. Let me know if there is more work to be done :)

marked the checklist item Study the impact on the selection efficiency for Z\rightarrow\mu\mu as completed

marked the checklist item Study the impact on the selection efficiency for W\rightarrow\mu\nu as completed

marked the checklist item Study the impact on the selection efficiency for Z\rightarrow\mu\mu as incomplete

marked the checklist item Study the impact on the selection efficiency for W\rightarrow\mu\nu as incomplete

changed the description

@mvesteri Let me know if you had any comments on the changes in efficiency for the W and Z.

added 1 commit

• 4ec791b8 - Enabling GECs again for hlt1_pp_default and hlt1_pp_matching to test for throughput

Compare with previous version

removed hlt1-throughput-decreased label

added hlt1-throughput-decreased label

changed the description

marked the checklist item Study the impact on the selection efficiency for Z\rightarrow\mu\mu as completed

marked the checklist item Study the impact on the selection efficiency for W\rightarrow\mu\nu as completed

marked the checklist item Confirm that the impact on the throughput is reasonable as completed

added 1 commit

• e46ef664 - Removed/renamed all no_gec sequences and removed all traces of GECs (imports, functions, etc.)

Compare with previous version

removed hlt1-throughput-decreased label

added 1 commit

• 987636aa - Adjust throughput tests to run over correct sequences

Compare with previous version

resolved all threads

changed the description

changed the description

added 16 commits

• 987636aa...f00ea23a - 15 commits from branch master
• 4b471aca - Merge branch 'master' into ahabdelm-remove-gec-from-hlt1

Compare with previous version

added 1 commit

• 476085ae - Fix throughput test by adding correct (updated) sequence name

Compare with previous version

added hlt1-throughput-decreased label

added 1 commit

• 58906fdf - Removed repeated throughput test

Compare with previous version

removed hlt1-throughput-decreased label