Add new parallel HLTBeginSeq for L1Decoder and CacheCreators, add HLTEndSeq for summary algorithms

Also put L1Decoder into the sequence at the correct point already in runHLT_standalone.py. Addresses ATR-21657, and in general means we can run parallel jobs more stably.

This merge request affects 2 packages:

• Trigger/TriggerCommon/TriggerJobOpts
• Trigger/TriggerCommon/TriggerMenuMT

Adding @bernius as watcher

added Trigger TriggerMenu master review-pending-level-1 labels

While I like the idea and the implementation, I don't like the generic, meaningless and misleading name. I don't like HLTTop either to be honest.

Maybe we could mirror the athena sequence names and have e.g. HLTBeginSeq and HLTEndSeq? We could also call the middle main part HLTAlgSeq but maybe that one doesn't make as much sense. We already have HLTAllSteps so we could use that and move one level up.

I would see the trigger job CF like this:

HLTBeginSeq
  L1TriggerByteStreamDecoder
  L1Decoder
  MuonCacheCreator
  MuonPRDCacheCreator
  InDetCacheCreatorTrig
  CaloBCIDAvgAlg
  TrigOpMonitor
HLTAllSteps
  Step1_filter
  Step1_reco
  // and all the other HLTCF steps here
HLTEndSeq
  TriggerSummaryFinal
  DecisionSummaryMakerAlg
  TrigSignatureMoniMT
  EDMCreatorAlg

What do you think?

Hi Rafal,

I agree with you, these are nice naming choices. I actually wasn't keen on the name but forgot about this when I was just happy that the solution worked.

I can implement this tomorrow.

CI Result FAILURE (hash f4891bdb)

	Athena	AthSimulation	AnalysisBase	AthGeneration
externals
cmake
make
required tests
optional tests

Full details available on this CI monitor view
Athena: number of compilation errors 0, warnings 1
AthSimulation: number of compilation errors 0, warnings 0
AnalysisBase: number of compilation errors 0, warnings 0
AthGeneration: number of compilation errors 0, warnings 0
For experts only: Jenkins output [CI-MERGE-REQUEST-CC7 17282]

Putting back the user action required label for the name changes, and to fix the test failures.

added review-user-action-required label and removed review-pending-level-1 label

added 4 commits

• 8371020e - Remove unused import
• e429e14c - Add sequence name argument to triggerRunCfg
• a9033a14 - Set sequence to AthMasterSeq: otherwise it defaults to AlgSeq
• 76fd6b52 - Establish HLTBeginSeq and HLTEndSeq

Compare with previous version

This merge request affects 3 packages:

• Trigger/TrigValidation/TrigUpgradeTest
• Trigger/TriggerCommon/TriggerJobOpts
• Trigger/TriggerCommon/TriggerMenuMT

Adding @okumura ,@bernius ,@jpanduro as watchers

added review-pending-level-1 label and removed review-user-action-required label

CI Result FAILURE (hash 76fd6b52)

	Athena	AthSimulation	AnalysisBase	AthGeneration
externals
cmake
make
required tests
optional tests

Full details available on this CI monitor view
Athena: number of compilation errors 0, warnings 0
AthSimulation: number of compilation errors 0, warnings 0
AnalysisBase: number of compilation errors 0, warnings 0
AthGeneration: number of compilation errors 0, warnings 0
For experts only: Jenkins output [CI-MERGE-REQUEST-CC7 17322]

The CI tests are still failing with errors like:

File "/build2/ci-builds/master/Athena/build/Athena/x86_64-centos7-gcc8-opt/python/TriggerMenuMT/HLTMenuConfig/Menu/HLTCFConfig.py", line 132, in makeHLTTree l1decoder = [ d for d in hltBeginSeq.getChildren() if d.getType() == "L1Decoder" ] AttributeError: 'NoneType' object has no attribute 'getChildren' Py:Athena INFO leaving with code 8: "an unknown exception occurred"

Marking for user action. (L2)

added review-user-action-required label and removed review-pending-level-1 label

added 349 commits

• 76fd6b52...52722fb6 - 347 commits from branch atlas:master
• 5a1cf5ee - Merge remote-tracking branch 'upstream/master' into new_HLT_bootstrap_sequence
• 43aeaa84 - Adjust to new L1Decoder location in tests

Compare with previous version

This merge request affects 3 packages:

• Trigger/TrigValidation/TrigUpgradeTest
• Trigger/TriggerCommon/TriggerJobOpts
• Trigger/TriggerCommon/TriggerMenuMT

Adding @okumura ,@bernius ,@jpanduro as watchers

added review-pending-level-1 label and removed review-user-action-required label

CI Result FAILURE (hash 43aeaa84)

	Athena	AthSimulation	AnalysisBase	AthGeneration
externals
cmake
make
required tests
optional tests

Full details available on this CI monitor view
Athena: number of compilation errors 0, warnings 0
AthSimulation: number of compilation errors 0, warnings 0
AnalysisBase: number of compilation errors 0, warnings 0
AthGeneration: number of compilation errors 0, warnings 0
For experts only: Jenkins output [CI-MERGE-REQUEST-CC7 17386]

Jenkins please retry a build

This merge request affects 3 packages:

• Trigger/TrigValidation/TrigUpgradeTest
• Trigger/TriggerCommon/TriggerJobOpts
• Trigger/TriggerCommon/TriggerMenuMT

Adding @okumura ,@bernius ,@jpanduro as watchers

CI Result FAILURE (hash 43aeaa84)

	Athena	AthSimulation	AnalysisBase	AthGeneration
externals
cmake
make
required tests
optional tests

Full details available on this CI monitor view
Athena: number of compilation errors 0, warnings 0
AthSimulation: number of compilation errors 0, warnings 0
AnalysisBase: number of compilation errors 0, warnings 0
AthGeneration: number of compilation errors 0, warnings 0
For experts only: Jenkins output [CI-MERGE-REQUEST-CC7 17411]

changed title from Add new parallel HLTBootstrap sequence for L1Decoder and CacheCreators. to Add new parallel HLTBeginSeq for L1Decoder and CacheCreators, add HLTEndSeq for summary algorithms

changed the description

added 2 commits

• 10463aaa - Add extra argument to triggerRunCfg call
• 7387b424 - Set separate working directory for unit tests - otherwise they can interfere...

Compare with previous version

mentioned in merge request !34938 (merged)

This merge request affects 3 packages:

• Trigger/TrigValidation/TrigUpgradeTest
• Trigger/TriggerCommon/TriggerJobOpts
• Trigger/TriggerCommon/TriggerMenuMT

Adding @okumura ,@bernius ,@jpanduro as watchers

CI Result FAILURE (hash 7387b424)

	Athena	AthSimulation	AnalysisBase	AthGeneration
externals
cmake
make
required tests
optional tests

Full details available on this CI monitor view
Athena: number of compilation errors 0, warnings 0
AthSimulation: number of compilation errors 0, warnings 0
AnalysisBase: number of compilation errors 0, warnings 0
AthGeneration: number of compilation errors 0, warnings 0
For experts only: Jenkins output [CI-MERGE-REQUEST-CC7 17448]

This has the known error and some additional ones that are my fault. I will work on these.

added 2 commits

• 57d615af - Add rmdir and mkdir steps, combine two similar tests into a cmake foreach
• a80e59a8 - Remove attempts to add duplicates of hltTop sequence

Compare with previous version

This merge request affects 3 packages:

• Trigger/TrigValidation/TrigUpgradeTest
• Trigger/TriggerCommon/TriggerJobOpts
• Trigger/TriggerCommon/TriggerMenuMT

Adding @okumura ,@bernius ,@jpanduro as watchers

resolved all threads

CI Result FAILURE (hash a80e59a8)

	Athena	AthSimulation	AnalysisBase	AthGeneration
externals
cmake
make
required tests
optional tests

Full details available on this CI monitor view
Athena: number of compilation errors 0, warnings 0
AthSimulation: number of compilation errors 0, warnings 0
AnalysisBase: number of compilation errors 0, warnings 0
AthGeneration: number of compilation errors 0, warnings 0
For experts only: Jenkins output [CI-MERGE-REQUEST-CC7 17488]

Nearly there - apart from the known issue I see

 TrigSignatureMoniMT                                INFO HLT_mu6_mu6noL1_L1MU6 #451489897
-TrigSignatureMoniMT                                INFO -- #451489897 Events          10         10         10         10         10         10         6          4          4
-TrigSignatureMoniMT                                INFO -- #451489897 Features                              14         13         16         21         9          7
+TrigSignatureMoniMT                                INFO -- #451489897 Events          10         10         10         10         10         10         5          4          4
+TrigSignatureMoniMT                                INFO -- #451489897 Features                              14         13         16         21         8          7

so some small count changes in HLT_mu6_mu6noL1_L1MU6. Any ideas anyone?

Unrelated, see !34954 (comment 3645525)

OK thanks. L1/L2 shifters: this has had expert attention (@tbold, @rbielski) although you are welcome to go through your own review.

Thanks for the expert reviews. I don't see any particular issue with the new configurations. Indeed the failures are related to a different issue. Approving - L1

added review-approved label and removed review-pending-level-1 label

merged

mentioned in commit 0b187948

added sweep:ignore label