Adding towers as ghosts

This MR adds towers as ghosts to PFlow jets and stores the towers in JETM1. This allows the Jet/Etmiss group to study e.g. q/g tagging or PU jet suppression in the forward region as well as new rho definitions using towers. This MR shouldn't have an effect on any other derivations, including DAOD_PHYS because GhostTower is not added to the smart slimming list.

Tagging @mswiatlo @camacho @schaarsc @sawyer @hpacey @miholzbo @mspalla @dportill for their information.

Restarting after network problems resolution

Jenkins please retry a build

This merge request affects 2 packages:

• PhysicsAnalysis/DerivationFramework/DerivationFrameworkJetEtMiss
• Reconstruction/Jet/JetRec

Affected files list will not be printed in this case

Adding @goetz ,@jmellent ,@jveatch ,@nelsonc ,@ispokhar as watchers

added Derivation JetEtmiss Reconstruction analysis-review-required master review-pending-level-1 labels

CI Result SUCCESS (hash eb779798)

	Athena	AthSimulation	AthGeneration	AnalysisBase	AthAnalysis	DetCommon
externals
cmake
make
required tests
optional tests

Full details available on this CI monitor view
Athena: number of compilation errors 0, warnings 0
AthSimulation: number of compilation errors 0, warnings 0
AthGeneration: number of compilation errors 0, warnings 0
AnalysisBase: number of compilation errors 0, warnings 0
AthAnalysis: number of compilation errors 0, warnings 0
DetCommon: number of compilation errors 0, warnings 0
For experts only: Jenkins output [CI-MERGE-REQUEST-CC7 40489]

All looks fine, approving. Cheers, L1

added review-approved label and removed review-pending-level-1 label

added analysis-review-approved label and removed analysis-review-required label

merged

mentioned in commit c5a6d934

added sweep:ignore label

This MR seems to change the ESD content of the standard q431:

xAOD::Init                INFO    Environment initialised for data access
Py:diff-root         INFO comparing tree [CollectionTree] in files:
Py:diff-root         INFO  old: [./ref-2021-10-18T2101/myESD.pool.root]
Py:diff-root         INFO  new: [./myESD.pool.root]
Py:diff-root         INFO branches of interest: 
Py:diff-root         INFO ignore  leaves:       ['RecoTimingObj_p1_HITStoRDO_timings', 'RecoTimingObj_p1_RAWtoESD_mems', 'RecoTimingObj_p1_RAWtoESD_timings', 'RAWtoESD_mems', 'RAWtoESD_timings', 'ESDtoAOD_mems', 'ESDtoAOD_timings', 'HITStoRDO_timings', 'RAWtoALL_mems', 'RAWtoALL_timings', 'RecoTimingObj_p1_RAWtoALL_mems', 'RecoTimingObj_p1_RAWtoALL_timings', 'RecoTimingObj_p1_EVNTtoHITS_timings', 'EVNTtoHITS_timings', 'RecoTimingObj_p1_Bkg_HITStoRDO_timings', 'index_ref']
Py:diff-root         INFO enforce leaves:       ('BCID',)
Py:diff-root         INFO leaves prefix:        
Py:diff-root         INFO hacks:                ('m_athenabarcode', 'm_token')
Py:diff-root         INFO entries:              20
Py:diff-root         INFO mode:                 semi-detailed
Py:diff-root         INFO error mode:           resilient
Py:diff-root         INFO order trees:          True
Py:diff-root         INFO comparing over [20] entries...
Py:diff-root      WARNING the following variables exist only in the new file !
Py:diff-root      WARNING  - [xAOD::JetAuxContainer_v1_AntiKt4EMPFlowJetsAuxDyn.GhostTower]
Py:diff-root      WARNING  - [xAOD::JetAuxContainer_v1_AntiKt4EMPFlowJetsAuxDyn.GhostTowerCount]
Py:diff-root      WARNING  - [xAOD::JetAuxContainer_v1_AntiKt4EMPFlowJetsAuxDyn.GhostTowerPt]
Py:diff-root         INFO comparing [2400] leaves over entries...
Py:diff-root         INFO Found [60364005] identical leaves
Py:diff-root         INFO Found [7] different leaves
Py:diff-root         INFO  [xAOD::JetAuxContainer_v1_AntiKt4EMPFlowJetsAuxDyn.EMFrac]: 2 leaves differ
Py:diff-root         INFO  [xAOD::JetAuxContainer_v1_AntiKt4EMPFlowJetsAuxDyn.EnergyPerSampling]: 3 leaves differ
Py:diff-root         INFO  [xAOD::JetAuxContainer_v1_AntiKt4EMPFlowJetsAuxDyn.FracSamplingMax]: 1 leaves differ
Py:diff-root         INFO  [xAOD::JetAuxContainer_v1_AntiKt4EMPFlowJetsAuxDyn.HECFrac]: 1 leaves differ
Py:diff-root        ERROR files differ!

Here is the log from the ART: https://atlas-art-data.web.cern.ch/atlas-art-data/grid-output/master/Athena/x86_64-centos7-gcc8-opt/2021-10-19T2101/Tier0ChainTests/test_q431/tarball_logs/payload.stdout

Hi @amete ,

Thanks for letting me know. The three extra variables are expected. I guess I will have to add some reference file then. For the other four variables I don't see how this is related to my MR but will investigate if I can spot anything. The towers should in principle not have any impact on jet finding and thus any variables related to it.

Cheers, Chris

Thanks a lot @cdelitzs. The addition of a few variables in the ESD is probably not a problem on its own (unless PROCs think otherwise). I just wanted to highlight it to make sure there is no unintended side effects (i.e. changes trickling down to other downstream formats and/or affecting other variables).

Just to understand-- is it expected that the ghosts are added to the outputs of the ESD? I guess the ESD contains jets still, but the AOD doesn't, and that's what we're seeing?

I guess when we remake jets in the AOD->DAOD stage, we'll also get these variables, but most derivations (including PHYS) will drop them by default.

Hi @mswiatlo ,

The easiest implementation was to ghost-associate the towers per default to the jets and not just for specific use cases. This means that they would appear in the ESDs where we decided to keep the jets for now. I could maybe suppress writing those moments to the ESDs however I think some people like @loch would be happy to have those in the ESDs for some further studies :) I think in the past we were discussing with @delsart to suppress in general the pT distribution of ghosts, i.e. the GhostXXXPt variables because that's not really used (so this would drop at least one variable and GhostTower are just links) but I might misremember this discussion at the moment.

For AOD -> DAOD we would just not add those variables to the smart slimming list but having the ghost-association run per default makes it just significantly easier for the JETMX derivation formats and any other formats who would like to study them for whatever reason :) I thought about a few alternatives so that we only schedule them in JETMX derivations but I didn't find that particularly appealing but maybe someone else has a more beautiful solution but indeed in principle for most formats that content shouldn't change as you mentioned.

Cheers, Chris

Thanks for the followup Chris! This seems fine to me, I just wanted to make sure I understood the change.

Hi @amete ,

Sorry to bother you with this but could you remind me what the exact command is for the q431 test? I simply used Reco_tf.py --AMI q431 but I couldn't reproduce the problem with that but I think it might be just that I'm not running over enough events.

I was also wondering if it's possible to print out the event number or something like that in the diff when the leaves disagree in a certain branch to take a closer look. I could write a short script to compare those but thought I would ask :)

Cheers, Chris

Hi Chris,

This is the script that runs on ART. If you run diff-root with --mode detailed, then for each difference it should tell you the event's TTree index, the name of the branch and the leaf, as well as the index of the entry within there (to print all differences you should do --error-mode resilient). Once you know the TTree index, you can use dump-event-from-file.py script to dump a human-readable version of the data that diff-root compares for that event. There are other scripts that do similar things but this should do the trick.