Trying to minimize CPU load mentioned ATR-22328

This MR is to address CPU load discussed in ATR-22328. I did some correction in this MR and would like to check locally if this is improving the cpu/time. I am using TrigCostAnalysis_JobOptions.py and data_test.00339070.unknown_SingleStream.daq.RAW._lb0149._Athena._0001.data in /afs/cern.ch/work/d/dbakshig/public/ATR_21211_etCut_1/run for cost calculation. The file data_test.00339070.unknown_SingleStream.daq.RAW._lb0149._Athena._0001.data is generated by running

athena.py --imf --perfmon --threads 1 --concurrent-events 1 --filesInput '/eos/home-d/dbakshig/data17_13TeV.00339070.physics_Main/data17_13TeV.00339070.physics_Main.merge.DRAW_EGZ.f887_m1831._0152.1' --evtMax 3 --skipEvents 0 -c 'isOnline=True;forceEnableAllChains=True;setGlobalTag="CONDBR2-HLTP-2018-01"' TriggerJobOpts/runHLT_standalone.py 2>&1 |tee electron_ManyEvents.log

and to calculate cost I ran

athena TrigCostAnalysis_JobOptions.py

but its crashing, the TrigCostAnalysis_JobOptions.py I have edited can be seen in /afs/cern.ch/work/d/dbakshig/public/ATR_21211_etCut_1/run. Adding @tamartin , @fernando , @cjmeyer , @stelzer

Thanks, Debo

Hello @dbakshig

These changes won't make it any faster - but they're good to get in nonetheless.

TrigCostAnalysis_JobOptions.py has a crash in finalise. But it still produces the output ROOT file. You should find the alg in here.

added 1 commit

• 7fc43d1a - Adding some preselection to TrigEgammaFastElectronFexMTConfig.py

Compare with previous version

added 270 commits

• 7fc43d1a...81511495 - 267 commits from branch atlas:master
• 10cea362 - removing threshold dependency from ElectronFex_Hard
• 38b34f69 - Merge tag 'nightly/master/2020-11-18T2101' into tryingLoadOffCPU
• adbd7bc4 - Turning on DEBUG messages

Compare with previous version

added 83 commits

• adbd7bc4...c2ca84a5 - 65 commits from branch atlas:master
• ba6b58be - First iteratation towards using an updated RoI for egamma HLT processing
• 8831b182 - Merge remote-tracking branch 'upstream/master' into betterROIForEgamma
• a0a8cd84 - Update to reflect true ownership model
• 6f082b5a - Merge remote-tracking branch 'upstream/master' into betterROIForEgamma
• 55e5dffc - Update ViewCreatorCentredOnClusterROITool, configure its initial size
• 890d1f8c - Whitespace
• a561d670 - Allow ViewCreatorPreviousROITool to make a copy of the ROIs which it uses
• 2593e92e - Merge remote-tracking branch 'upstream/master' into betterROIForEgamma
• 1257fca7 - Update tracking input maker, update collection names
• c6accfc4 - Update counts using new centred egamma ROIs
• 7ad84e36 - Update docs
• 19f80a8a - Merge remote-tracking branch 'upstream/master' into betterROIForEgamma
• dfcd8c38 - update refs
• f5884e7c - Merge remote-tracking branch 'upstream/master' into betterROIForEgamma
• 0ac34ba4 - Update refs following MR 38392
• 6885bfd0 - Merge tag 'nightly/master/2020-11-19T2101' into tryingLoadOffCPU
• 2e898f21 - Merge remote-tracking branch 'dtim/betterROIForEgamma' into tryingLoadOffCPU
• d161f0be - c++/python sanitization in...

Compare with previous version

Hi @tamartin , @fernando , @cjmeyer ,

The changes I have made in Fex gives a reduction in step2 (that is 1 here; since step starts from 0) features

und 32 chains with count differences:
chainComp INFO       HLT_2e17_etcut_L12EM15VH:
chainComp INFO         stepFeatures:
chainComp INFO           1: 314 -> 106
chainComp INFO       HLT_2e3_etcut_L12EM3:
chainComp INFO         stepFeatures:
chainComp INFO           1: 3730 -> 2214
chainComp INFO           2: 796 -> 784
chainComp INFO           3: 304 -> 300
chainComp INFO       HLT_e17_lhloose_mu14_L1EM15VH_MU10:
chainComp INFO         stepFeatures:
chainComp INFO           1: 43 -> 11
chainComp INFO       HLT_e17_lhvloose_nod0_L1EM15VH:
chainComp INFO         stepFeatures:
chainComp INFO           1: 80 -> 22
chainComp INFO       HLT_e24_lhvloose_L1EM20VH:
chainComp INFO         stepFeatures:
chainComp INFO           1: 80 -> 22
chainComp INFO       HLT_e26_etcut_L1EM22VHI:
chainComp INFO         stepFeatures:
chainComp INFO           1: 132 -> 47
chainComp INFO       HLT_e26_lhloose_L1EM15VH:
chainComp INFO         stepFeatures:
chainComp INFO           1: 80 -> 22

but cpu/time for precisionElectronHypo doesn't improve. Should I go ahead and merge this branch?

Thanks, Debo.

For the > 3 GeV items, it is entirely contained within the FTF step. Good. Please do Debo.

Hi @tamartin ,

I am currently having excess warning that's cannot merge this branch

INFO     -----------------------------------------------------------------------------------------------------------------------------------------------
INFO     | test_trigAna_RDOtoRDOTrig_v1Dev_build.py | Reco_tf: 0, CheckLog: 0, MessageCount: 3, RootComp: 175, CountRefComp: 0, ZeroCounts: 0, exit: 0 |
INFO     | test_trigP1_v1Dev_build.py               | athenaHLT: 0, CheckLog: 0, MessageCount: 2, RootComp: 201, ZeroCounts: 0, exit: 0                |
INFO     | test_trig_data_newJO_build.py            | athena: 0, ExecSummary: 0, CheckLog: 0, exit: 0                                                  |
INFO     | test_trig_data_v1Dev_build.py            | athena: 0, CheckLog: 0, MessageCount: 1, RootComp: 25, CountRefComp: 0, ZeroCounts: 0, exit: 1   |
INFO     -----------------------------------------------------------------------------------------------------------------------------------------------
INFO     3 tests succeeded out of 4 executed
ERROR    ==================================================
ERROR    The following 1 tests failed:
ERROR        test_trig_data_v1Dev_build.py
ERROR    ==================================================

all test results are here /afs/cern.ch/work/d/dbakshig/public/ATR_21211_etCut_1/run/runTrigART one type of WARNING seems egamma specific , can it be rectified?

IMfastCalo                                 3   0 WARNING Have no decisions in output handle HLTNav_IMfastCalo. Handle is valid but container is empty.

Thanks, Debo.

BTW technically I am not sure if its failing due to WARNING but I am not seeing any other reason since athena.log shows successful run

There is https://its.cern.ch/jira/browse/ATR-22306 open for Test trig_data_v1Dev_build finished with code 1 because the following required steps failed: ['MessageCount'] - it won't invalidate the CI test

added 112 commits

• d161f0be...93e95499 - 110 commits from branch atlas:master
• 8f33340b - Merge remote-tracking branch 'upstream/master' into tryingLoadOffCPU
• 04a28e6f - Updating references and removing DEBUG messages

Compare with previous version

unmarked as a Work In Progress

This merge request affects 5 packages:

• Trigger/TrigHypothesis/TrigEgammaHypo
• Trigger/TrigValidation/TrigAnalysisTest
• Trigger/TrigValidation/TriggerTest
• Trigger/TriggerCommon/TrigEDMConfig
• Trigger/TriggerCommon/TriggerMenuMT

Affected files list will not be printed in this case

Adding @sutt ,@amorley ,@dbakshig ,@okumura ,@hrussell ,@fernando ,@bernius ,@jmeyer ,@okuprash ,@ckaldero ,@jpanduro as watchers

added Egamma Trigger TriggerEDM TriggerMenu changes-trigger-counts master review-pending-level-1 labels

CI Result SUCCESS (hash 04a28e6f)

	Athena	AthSimulation	AthGeneration	AnalysisBase
externals
cmake
make
required tests
optional tests

Full details available on this CI monitor view
Athena: number of compilation errors 0, warnings 0
AthSimulation: number of compilation errors 0, warnings 0
AthGeneration: number of compilation errors 0, warnings 0
AnalysisBase: number of compilation errors 0, warnings 0
For experts only: Jenkins output [CI-MERGE-REQUEST-CC7 23889]

resolved all threads

Dear Shifter, can it be passed for further review?

Thanks, Debo.

Hi, I don't want to cause a delay in the thread, but I find the description somewhat unhelpful, ie what are the actual changes ? Do I need to read through the ATR ? Perhaps it would have been helpfuly to have outlined what the actual changes were and how they were expected to changed things, eg

"Changed to the selection in the Egamma hypo show things down, so this changes the order of the loop"

or something. I did in fact reading through the Jira but it was not clear even from there exactly what the changes were. I see a change in the number of counts, so presumably it was some algorithmic change ?

I just would have found it helpful to have included some summary of that information.

Cheers Mark