Migrate DerivationFrameworkInDet to data handles

Should only have an impact on RAWtoALL production jobs, or if special ID DAOD outputs are requested. I compared DAODs (DAOD_IDTIDE, _PIXELVALID, DAOD_SCTVALID, DAOD_IDTRKVALID, DAOD_IDTRKLUMI) requested as additional outputs when running q221 and q431. The DAODs of the unpatched and patched release agree. Though, attempts to produce DAOD_IDTIDE in q431 fails for both version with identical errors, and to produce DAOD_IDTRKLUMI in the unpatched release, a small configuration change is needed.

I tried to produce other DAODs which use some of the tools of DerivationFrameworkInDet (EGAM1, HIGG.., TAUP1, ...) but all attempts fail due to unrelated problems.

Are there any other tests people like to see (@stsuno, @oda, @npetters, @goblirsc) ? Is there any validation DAOD used for TRT validation (@hansenph) I should try to produce ?

I noticed that the UnassociatedHitsGetter does not consider ganged pixels and TRT outliers i.e. . Is that desired (@stsuno, @hansenph, @npetters, @goblirsc, ) ?

This merge request affects 3 packages:

InnerDetector/InDetEventCnv/InDetPrepRawDataToxAOD
InnerDetector/InDetExample/InDetRecExample
PhysicsAnalysis/DerivationFramework/DerivationFrameworkInDet

Adding @goetz ,@sroe as watchers

added Derivation InnerDetector master review-pending-level-1 labels

Hello @goetz We run something like this script to produce TRT DxAOD #!/bin/bash

export MyInput=/cvmfs/atlas-nightlies.cern.ch/repo/data/data-art/Tier0ChainTests/data18_13TeV.00348618.physics_Main.daq.RAW._lb0220._SFO-1._0001.data

Reco_tf.py --inputDRAW_EMUFile=$MyInput --ignoreErrors 'True' --maxEvents '100' --conditionsTag 'CONDBR2-BLKPA-2018-14' --geometryVersion 'ATLAS-R2-2016-01-00-01' --preExec 'all:from InDetPhysValMonitoring.InDetPhysValJobProperties import InDetPhysValFlags; InDetPhysValFlags.doValidateTightPrimaryTracks.set_Value_and_Lock(True); from InDetPrepRawDataToxAOD.InDetDxAODJobProperties import InDetDxAODFlags; InDetDxAODFlags.DumpPixelInfo.set_Value_and_Lock(False); InDetDxAODFlags.DumpSctInfo.set_Value_and_Lock(False); InDetDxAODFlags.DumpTrtInfo.set_Value_and_Lock(True); InDetDxAODFlags.DumpTriggerInfo.set_Value_and_Lock(False); InDetDxAODFlags.ThinTrackSelection.set_Value_and_Lock("InDetTrackParticles.pt > 2.0*GeV && abs(InDetTrackParticles.eta) < 1.8"); InDetFlags.doTRTGlobalOccupancy.set_Value_and_Lock(True); InDetFlags.doSlimming.set_Value_and_Lock(False);' --steering 'RAWtoESD:in+ESD' --outputDAOD_IDTRKVALIDFile=InDetDxAOD_new.pool.root

cheers Peter

Hi again,

Here is a more up-to-date TRT DxAOD script using the latest alignment:

pathena --trf "Reco_tf.py
--inputBSFile='%IN'
--ignoreErrors 'True'
--skipEvents='%SKIPEVENTS'
--maxEvents '250'
--AMI 'f938'
--preExec 'all:from InDetPrepRawDataToxAOD.InDetDxAODJobProperties import InDetDxAODFlags ; InDetDxAODFlags.DumpPixelInfo.set_Value_and_Lock(False); InDetDxAODFlags.DumpSctInfo.set_Value_and_Lock(False); InDetDxAODFlags.DumpTrtInfo.set_Value_and_Lock(True);InDetDxAODFlags.DumpTriggerInfo.set_Value_and_Lock(True); InDetFlags.doTRTGlobalOccupancy.set_Value_and_Lock(True); InDetFlags.doSlimming.set_Value_and_Lock(False); InDetDxAODFlags.DumpLArCollisionTime.set_Value_and_Lock(False); rec.OutputLevel.set_Value_and_Lock(ERROR); rec.doTau.set_Value_and_Lock(False); rec.doBTagging.set_Value_and_Lock(False); rec.doPerfMon.set_Value_and_Lock(True);'
--postExec 'all:from IOVDbSvc.CondDB import conddb; conddb.addOverride("/Indet/AlignL1/ID", "IndetAlignL1ID-R2dynamic_2018_Part1_ReAlign_Initial"); conddb.addOverride("/Indet/AlignL2/PIX", "IndetAlignL2PIX-R2dynamic_2018_Part1_ReAlign_Initial"); conddb.addOverride("/Indet/AlignL2/SCT", "IndetAlignL2SCT-R2dynamic_2018_Part1_ReAlign_Initial"); conddb.addOverride("/Indet/IBLDist", "IndetIBLDist-R2dynamic_2018_Part1_ReAlign_Initial"); conddb.addOverride("/TRT/AlignL1/TRT", "TRTAlignL1-R2dynamic_2018_Part1_ReAlign_Initial"); conddb.addOverride("/Indet/AlignL3", "IndetAlignL3_2018_Part1_ReAlign_Jan-2020"); conddb.addOverride("/TRT/AlignL2", "TRTAlignL2_2018_Part1_ReAlign_Jan-2020"); conddb.addOverride("/TRT/Calib/ToT/ToTValue","TRTCalibToTDataValue-000-03") ; conddb.addOverride("/TRT/Calib/ToT/ToTVectors","TRTCalibToTDataVectors-000-03")'
--steering 'RAWtoESD:in+ESD'
--outputDAOD_IDTRKVALIDFile=%OUT.InDetDxAOD.pool.root"
--nEventsPerJob=250
--official
--nCore=8
--osMatching
--inDS=data18_13TeV.00352448.physics_Main.merge.DRAW_ZMUMU.f938_m1831
--outDS=group.det-indet.data18_13TeV.00352448.physics_Main.merge.DRAW_ZMUMU.f938_m1831.TRTxAOD_NewAlign_trt115

cheers, Peter

CI Result SUCCESS (hash 5dd37f41)

	Athena	AthSimulation	AnalysisBase	AthGeneration
externals
cmake
make
required tests
optional tests

Full details available on this CI monitor view
Athena: number of compilation errors 0, warnings 0
AthSimulation: number of compilation errors 0, warnings 0
AnalysisBase: number of compilation errors 0, warnings 0
AthGeneration: number of compilation errors 0, warnings 0
For experts only: Jenkins output [CI-MERGE-REQUEST-CC7 12399]

I may overlook something but the cmake issues do not seem to be related to the changes of this MR.

added 6 commits

4ce115f0 - Replace deprecated PRD association tool by PRDtoTrackMap.
d949b743 - Removed duplicated truth decoration.
ed6cb374 - Migrate TrackToVertexWrapper to data handles.
afce1969 - Converted end-of-line from Dos to unix in UnassociatedHitsDecorator
6c2f653f - Migrate UnassociatedHitsDecorator to data handles.
721614a7 - Migrate UnassociatedHitsGetterTool to data handles.

Compare with previous version

Concerning the above changes:

I corrected one commit to allow producing IDTRKVALID with the DumpTrtInfo flag being set, and at the same time
removed a configuration snippet which was duplicating the setup of a truth decoration algorithm

I reran all tests i.e. q221 and q431 with extra DAOD outputs as mentioned further above plus the IDTRKVALID with TRT info and see agreement between the patched and unpatched release (where unpatched is not fully correct since some patches are needed to get the jobs with the extra outputs working).

resolved all threads

This merge request affects 3 packages:

InnerDetector/InDetEventCnv/InDetPrepRawDataToxAOD
InnerDetector/InDetExample/InDetRecExample
PhysicsAnalysis/DerivationFramework/DerivationFrameworkInDet

Adding @goetz ,@sroe as watchers

CI Result FAILURE (hash 721614a7)

	Athena	AthSimulation	AnalysisBase	AthGeneration
externals
cmake
make
required tests
optional tests

Full details available on this CI monitor view
Athena: number of compilation errors 0, warnings 0
AthSimulation: number of compilation errors 0, warnings 0
AnalysisBase: number of compilation errors 0, warnings 0
AthGeneration: number of compilation errors 0, warnings 0
For experts only: Jenkins output [CI-MERGE-REQUEST-CC7 12421]

Retrying the build since the Tier0 test seems to has failed due to network issues PyRoot: Error in <TNetXNGFile::Open>: [ERROR] Server responded with an error: [3014] Unable to open file /eos/atlas/atlascerngroupdisk/data-art/grid-input/Tier0ChainTests/overlay-d1498/22.0/v30/myRDO.pool.root; Network is unreachable

Jenkins please retry a build

This merge request affects 3 packages:

InnerDetector/InDetEventCnv/InDetPrepRawDataToxAOD
InnerDetector/InDetExample/InDetRecExample
PhysicsAnalysis/DerivationFramework/DerivationFrameworkInDet

Adding @goetz ,@sroe as watchers

CI Result SUCCESS (hash 721614a7)

	Athena	AthSimulation	AnalysisBase	AthGeneration
externals
cmake
make
required tests
optional tests

Full details available on this CI monitor view
Athena: number of compilation errors 0, warnings 0
AthSimulation: number of compilation errors 0, warnings 0
AnalysisBase: number of compilation errors 0, warnings 0
AthGeneration: number of compilation errors 0, warnings 0
For experts only: Jenkins output [CI-MERGE-REQUEST-CC7 12444]