Implementing prescaling for trigger validation in TMMT

Addressing ATR-21613.

Adding writing/using of prescale sets (@mark ):

• !34964 (e4113401) - L1Menu has to accept that the menuName now will contain the prescale from the new menu name LS2_v1_TriggerValidation_mc_prescale, also added to the menu mapping to know which menu to build.

• !34964 (36e90a6f) - Add prescales to chains in GenerateMenuMT.py (1 by default) and then update using applyHLTPrescale in MenuUtil.py

• !34964 (ec2017ab) - move the default HLTPrescale JSON creation createHLTPrescalesFileFromMenu from TrigConfigSvcCfg.py to HLTPrescaleJSON.py in TriggerMenuMT. Also some code commented that probably can just be removed

• !34964 (c57d03e0) - now actually set chain prescale and enabled flag based on chain dictionary

• mark/athena@60642111 - fixes issue for finding L1xml file name

Setting which triggers to disable (@tvazquez )

• Disabled triggers starts by using a list disable_chain_list with triggers explicitly defined, work in progress moving to use a group in the menu.

---

Status chains now disabled in the menu LS2_v1_TriggerValidation_mc_prescale will be disabled when running. The relevant tests that are affected in ATR-21613 will need the menu changing to this rather than LS2_v1.

Future work will be adding stream prescale information ATR-21324 and currently remaking whole menus where only the name changes so to avoid duplicates should neaten this procedure to use the same names (will follow up on ATR-20878)

added 1 commit

• e4113401 - changes from Mark S to allow compilation

Compare with previous version

added 4 commits

• 36e90a6f - Add prescale to chain object
• ec2017ab - Move PrescaleJSON creation to TMMT from TrigConfigSvc
• c57d03e0 - Set enabled and prescale as required into HLTPrescaleJSON
• 828a0a0e - Minor comments/whitespace

Compare with previous version

changed the description

added 1093 commits

• 828a0a0e...6ae35b53 - 1090 commits from branch atlas:master
• d8856a79 - using Online groups
• 51f59246 - resolving conflict
• 2c6ad0dc - resolving more conflicts

Compare with previous version

Hi @tvazquez

Please add the commit: mark/athena@60642111

This then allows the L1 xml filename to be found correctly and the job can then successfully run. TrigSignatureMoniMT shows HLT_timeburner_L1All is disabled (see the log in athenaHLT:01.out)

Cheers, Mark

changed the description

added 530 commits

• 2c6ad0dc...3e3bbeda - 527 commits from branch atlas:master
• c73f9018 - merging with master,r15
• 2b2dccdb - Switch input L1Menu to just use menu base name (aka remove prescale definition) for XML reading
• cafebbe8 - adding configflags to work with r15

Compare with previous version

added 1 commit

• d328ca40 - some remaining changes

Compare with previous version

added 1 commit

• f646a930 - fixing a wrong merge

Compare with previous version

added 1 commit

• 500fe43e - reverting wrong merge

Compare with previous version

added 1 commit

• f1fd9336 - returning Prescale, fully working version for disabling chains in Online group for trigvalid

Compare with previous version

added 1 commit

• ea36fa38 - cleaning up white-spaces and others

Compare with previous version

unmarked as a Work In Progress

changed title from WIP: implementing prescaling for trigger validation in TMMT to Implementing prescaling for trigger validation in TMMT

WARNING: big files (>100K) are found in the changeset

148K in file Trigger/TriggerCommon/TriggerMenuMT/python/L1/Config/ItemDef.py

This merge request affects 3 packages:

• Trigger/TrigConfiguration/TrigConfigSvc
• Trigger/TriggerCommon/TriggerJobOpts
• Trigger/TriggerCommon/TriggerMenuMT

Adding @bernius ,@mark ,@orlando ,@cmorenom ,@asonay as watchers

added Trigger TriggerMenu master review-pending-level-1 labels

CI Result FAILURE (hash ea36fa38)

	Athena	AthSimulation	AthGeneration	AnalysisBase
externals
cmake
make
required tests
optional tests

Full details available on this CI monitor view
Athena: number of compilation errors 0, warnings 0
AthSimulation: number of compilation errors 0, warnings 0
AthGeneration: number of compilation errors 0, warnings 0
AnalysisBase: number of compilation errors 0, warnings 0
For experts only: Jenkins output [CI-MERGE-REQUEST-CC7 19156]

Jenkins please retry a build

WARNING: big files (>100K) are found in the changeset

148K in file Trigger/TriggerCommon/TriggerMenuMT/python/L1/Config/ItemDef.py

This merge request affects 3 packages:

• Trigger/TrigConfiguration/TrigConfigSvc
• Trigger/TriggerCommon/TriggerJobOpts
• Trigger/TriggerCommon/TriggerMenuMT

Adding @bernius ,@mark ,@orlando ,@cmorenom ,@asonay as watchers

CI Result FAILURE (hash ea36fa38)

	Athena	AthSimulation	AthGeneration	AnalysisBase
externals
cmake
make
required tests
optional tests

Full details available on this CI monitor view
Athena: number of compilation errors 0, warnings 0
AthSimulation: number of compilation errors 0, warnings 0
AthGeneration: number of compilation errors 0, warnings 0
AnalysisBase: number of compilation errors 0, warnings 0
For experts only: Jenkins output [CI-MERGE-REQUEST-CC7 19241]

added review-user-action-required label and removed review-pending-level-1 label

added 3 commits

• fef52d89 - Fixes for ctests
• 47e839ad - Set default prescale during generation for new JO
• 96e70e52 - Add prescale to fix full_menu_cf ctest

Compare with previous version

WARNING: big files (>100K) are found in the changeset

148K in file Trigger/TriggerCommon/TriggerMenuMT/python/L1/Config/ItemDef.py

This merge request affects 4 packages:

• Trigger/TrigConfiguration/TrigConfigSvc
• Trigger/TrigSteer/DecisionHandling
• Trigger/TriggerCommon/TriggerJobOpts
• Trigger/TriggerCommon/TriggerMenuMT

Adding @tamartin ,@asonay ,@bernius ,@orlando ,@cmorenom ,@mark as watchers

added review-pending-level-1 label and removed review-user-action-required label

CI Result FAILURE (hash 96e70e52)

	Athena	AthSimulation	AthGeneration	AnalysisBase
externals
cmake
make
required tests
optional tests

Full details available on this CI monitor view
Athena: number of compilation errors 0, warnings 0
AthSimulation: number of compilation errors 0, warnings 0
AthGeneration: number of compilation errors 0, warnings 0
AnalysisBase: number of compilation errors 0, warnings 0
For experts only: Jenkins output [CI-MERGE-REQUEST-CC7 19284]

It looks like the failure in DataQuality_required is now /var/lib/jenkins/workspace/CI-test-driver/ci_test_driver/ci_test_wrapper.sh: line 45: 30699 Killed $@

I will rerun CI tests...

Magdalena (L1)

Jenkins please retry a build

WARNING: big files (>100K) are found in the changeset

148K in file Trigger/TriggerCommon/TriggerMenuMT/python/L1/Config/ItemDef.py

This merge request affects 4 packages:

• Trigger/TrigConfiguration/TrigConfigSvc
• Trigger/TrigSteer/DecisionHandling
• Trigger/TriggerCommon/TriggerJobOpts
• Trigger/TriggerCommon/TriggerMenuMT

Adding @tamartin ,@asonay ,@bernius ,@orlando ,@cmorenom ,@mark as watchers

CI Result FAILURE (hash 96e70e52)

	Athena	AthSimulation	AthGeneration	AnalysisBase
externals
cmake
make
required tests
optional tests

Full details available on this CI monitor view
Athena: number of compilation errors 0, warnings 0
AthSimulation: number of compilation errors 0, warnings 0
AthGeneration: number of compilation errors 0, warnings 0
AnalysisBase: number of compilation errors 0, warnings 0
For experts only: Jenkins output [CI-MERGE-REQUEST-CC7 19317]

resolved all threads

@maslawin @rbielski @jpanduro @okumura

The data quality test is now failing, I assume because the menu CI tests are taking so long (seeing as they now run to completion) so we now hit a timeout in the rest of the CI? Is there a way to get around this limit or show that the changes don't affect this test (they are unrelated).

Cheers, Mark

Hi @mark, we've been seeing these DQ errors sporadically for the past days. We can retry again... usually it clears up.

Jenkins please retry a build

WARNING: big files (>100K) are found in the changeset

148K in file Trigger/TriggerCommon/TriggerMenuMT/python/L1/Config/ItemDef.py

This merge request affects 4 packages:

• Trigger/TrigConfiguration/TrigConfigSvc
• Trigger/TrigSteer/DecisionHandling
• Trigger/TriggerCommon/TriggerJobOpts
• Trigger/TriggerCommon/TriggerMenuMT

Adding @tamartin ,@asonay ,@bernius ,@orlando ,@cmorenom ,@mark as watchers

resolved all threads

resolved all threads

CI Result SUCCESS (hash 96e70e52)

	Athena	AthSimulation	AthGeneration	AnalysisBase
externals
cmake
make
required tests
optional tests

Full details available on this CI monitor view
Athena: number of compilation errors 0, warnings 0
AthSimulation: number of compilation errors 0, warnings 0
AthGeneration: number of compilation errors 0, warnings 0
AnalysisBase: number of compilation errors 0, warnings 0
For experts only: Jenkins output [CI-MERGE-REQUEST-CC7 19340]

Tests pass, and most changes look good, I'm only wondering if we can remove the functions that are commented out before approving. Thanks! -Mike (L1 shifter)

added review-user-action-required label and removed review-pending-level-1 label

added 1 commit

• ee0d9ca4 - removing commented out code

Compare with previous version

WARNING: big files (>100K) are found in the changeset

148K in file Trigger/TriggerCommon/TriggerMenuMT/python/L1/Config/ItemDef.py

This merge request affects 4 packages:

• Trigger/TrigConfiguration/TrigConfigSvc
• Trigger/TrigSteer/DecisionHandling
• Trigger/TriggerCommon/TriggerJobOpts
• Trigger/TriggerCommon/TriggerMenuMT

Adding @tamartin ,@asonay ,@bernius ,@orlando ,@cmorenom ,@mark as watchers

added review-pending-level-1 label and removed review-user-action-required label

resolved all threads

CI Result SUCCESS (hash ee0d9ca4)

	Athena	AthSimulation	AthGeneration	AnalysisBase
externals
cmake
make
required tests
optional tests

Full details available on this CI monitor view
Athena: number of compilation errors 0, warnings 0
AthSimulation: number of compilation errors 0, warnings 0
AthGeneration: number of compilation errors 0, warnings 0
AnalysisBase: number of compilation errors 0, warnings 0
For experts only: Jenkins output [CI-MERGE-REQUEST-CC7 19396]

• Previously reviewed by L1, comments responded to.
• Changes look good.
• Trigger code, but already reviewed by trigger experts
• Passing to L2 for a quick check as changes are reasonably extensive. (L1 shifter)

added review-pending-level-2 label and removed review-pending-level-1 label

I spotted a copyright year that needs to updates, but otherwise, did not see anything jump out. Marking approved. (L2)

added review-user-action-required label and removed review-pending-level-2 label

added review-pending-level-2 label and removed review-user-action-required label

added review-approved review-user-action-required labels

removed review-pending-level-2 label

removed review-user-action-required label

resolved all threads

merged

mentioned in commit 8dac4313

added sweep:ignore label

mentioned in merge request !36415 (merged)