Merge branch 'add_trig_edm_docs' into 'main'

Draft: Dedicated Trigger Developer Guide's Trigger EDM section

See merge request !129

docs/athena/trigger/developers/algos.md

@@ -44,18 +44,16 @@ ready) [fast e/gamma cluster maker algorithm]({{data.athena_git_url}}/blob/{{dat
 
 
 ## Saving EDM objects
-If the reconstructed objects that are created by algorithms
-need to be recorded in the output file these steps should be followed:
+The following software modifications are needed to record reconstructed objects created by HLT algorithms to file:
 
-1. contact trigger EDM coordinator [found on trigger Core s/w twiki](https://twiki.cern.ch/twiki/bin/view/Atlas/TriggerCoreSoftware)
 1. in the python configuration of the algorithm an instance of the
-  `recordable` function from [TrigEDMConfig.TriggerEDMRun3]({{data.athena_git_url}}/blob/{{data.branchTier0Short}}/Trigger/TriggerCommon/TrigEDMConfig/python/TriggerEDMRun.py#L34) needs
+  `recordable` function from [TrigEDMConfig.TriggerEDM]({{data.athena_git_url}}/blob/{{data.branchTier0Short}}/Trigger/TriggerCommon/TrigEDMConfig/python/TriggerEDM.py#L34) needs
   to be used when setting the name of the collection. It assures that
   the naming convention of the HLT EDM is followed and no trivial typos sneak in between name of the container .
-1. The collection (type and name)  has to be listed in 
-  [TrigEDMConfig.TriggerEDMRun3]({{data.athena_git_url}}/blob/{{data.branchTier0Short}}/Trigger/TriggerCommon/TrigEDMConfig/python/TriggerEDMRun3.py)
+1. The collection (type and name)  has to be listed in
+  [TrigEDMConfig.TriggerEDMRun3]({{data.athena_git_url}}/blob/{{data.branchTier0Short}}/Trigger/TriggerCommon/TrigEDMConfig/python/TriggerEDMRun3.py) (or `TriggerEDMRun4` for upgrade studies)
   together with the required output file types in which it should be recorded. Typically the object should be stored
-  at least in `BS`, `ESD` and `AOD`. Consult the Trigger EDM coordinator about this. Especially about the `AOD` formats where the size is constrained.
+  at least in `BS` and `ESD` (`RDO_TRIG` files - the equivalent of "raw" files for MC - contain all `ESD`-labelled content).
 1. Ensure that in
   [HLTEDMCreator.h]({{data.athena_git_url}}/blob/{{data.branchTier0Short}}/Trigger/TrigSteer/TrigOutputHandling/src/HLTEDMCreator.h)
   and
@@ -69,7 +67,9 @@ need to be recorded in the output file these steps should be followed:
 Once the above steps are done the object should appear in the relevant output files.
 For POOL files (ESD, AOD) the content can be checked with one of the following scripts:
 `checkxAOD.py filename` `checkSG.py filename` `checkFile.py filename`.
-  
+
+See the [trigger EDM guidelines](edm.md#procedure-for-modifying-trigger-edm) on the full procedure for adding new trigger EDM content (testing, verification and approval).
+
 
 ## Troubleshooting
 Typically a reconstruction algorithm uses many input data

docs/athena/trigger/developers/edm.md

+# Trigger EDM
+
+The Trigger Event Data Model (EDM) defines the trigger reconstructed content that is written out to RAW files and data or MC AODs.
+The Trigger EDM content for Run 3 is defined in [TrigEDMConfig.TriggerEDMRun3]({{data.athena_git_url}}/blob/{{data.branchTier0Short}}/Trigger/TriggerCommon/TrigEDMConfig/python/TriggerEDMRun3.py) (the same folder contains similar files for other periods).
+Each EDM entry is a tuple containing all EDM details for a specific interface or Aux container. An example of two (interface + aux) EDM entries:
+```python
+('xAOD::CaloClusterContainer#HLT_CaloEMClusters_Electron',               'BS ESD AODFULL AODSLIM', 'Egamma', [InViews('precisionCaloElectronViews')]),
+('xAOD::CaloClusterTrigAuxContainer#HLT_CaloEMClusters_ElectronAux.',    'BS ESD AODFULL AODSLIM', 'Egamma'),
+
+('xAOD::CaloClusterContainer#HLT_TopoCaloClustersLC',                             'BS ESD AODFULL', 'Tau', [InViews('tauCaloMVAViews'), allowTruncation]),   
+('xAOD::CaloClusterTrigAuxContainer#HLT_TopoCaloClustersLCAux.nCells.CENTER_MAG', 'BS ESD AODFULL', 'Tau', [allowTruncation]),
+
+```
+Every EDM entry will at minimum define the container type and container name, the "EDM targets" and trigger signature. Any additional dynamic Aux variables
+to be recorded are specified as part of the Aux container name string.
+The EDM targets define the output type to which the containers will be recorded in the case of `RAW`, `RDO_TRIG`, `ESD` and `AOD` files, as well as PHYSVAL `DAOD` files. The `ESD` level is used for MC `RDO_TRIG` (equivalent to data `RAW` files for MC) production. We use these to restrict the HLT content when not all HLT content is needed,
+e.g. in MC AODs for physics analysis derivations. The trigger signature is used to categorise total EDM size by signature in the [Trigger EDM monitoring](validation.md#trigger-edm-size-monitoring).
+
+## EDM targets
+
+The EDM target labels (2nd entry in an EDM entry) are mapped to the following output:
+
+| Label | Output | Purpose |
+|-------------|-------------|-------------|
+|`BS`| ByteStream - Any content that is to make it out of Point 1 and be recorded to RAW file. | Required for recording to data AOD after or kept in BS for debugging in case of data taking issues.|
+|`ESD`| ESD files | Low-level reconstructed information such as HLT calorimeter clusters for specialised reconstruction studies, also used for `RDO_TRIG` (MC equivalent of a RAW file) output.|
+|`AODFULL`| data AOD at Tier0, trigger developer AOD production requests, HLT Reprocessing AODs. | Required for regular trigger efficiency studies in data or validation during commissioning.|
+|`AODSLIM`| bulk MC production AODs for physics analysis, SampleT productions. | Required for physics analysis. Examples: Final precision HLT leptons/b-jets needed for trigger matching and trigger scale factors.|
+|`PhysicsTLA`/`EgammaPEBTLA`/`DarkJetPEBTLA`/`FTagPEBTLA`| respective RAW output of a Trigger-Level Analysis (TLA) or TLA Partial Event Building stream| Content required for a trigger-level physics analysis or TLA with partial event building (PEB) information.|
+
+**Note**: SampleA trigger validation productions record all `ESD` labelled content.
+
+**Important**: Think hard about what targets you need. The HLT content in total constitutes 25% of a physics_Main AOD and 20% of a RAW file, and we have far outblown our initial pledge for Run 3 disk space.
+
+- Valid reasons to keep trigger content in `AODFULL`: If needed for regular trigger efficiency checks in recorded data AOD or if it is a new implementation that needs a few months of commissioning.
+- Valid reasons to keep trigger content in `AODSLIM`: If needed for physics analysis, namely trigger matching and scale factor derivations.
+
+
+## Optional EDM details
+
+There are several additional EDM details that can be specified as a list constituting the last entry in the tuple.
+
+### Views
+
+In the case of reconstruction in a limited region of the detector, this names the event view in which this collection is reconstructed, so that additional information allowing the items to be accessed in event views is recorded. For more details, see [the section on Event Views](eventviews.md).
+
+### allowTruncation
+
+This is only valid for data taking. It is a flag to indicate that this result can be dropped from an event, if the event data is larger than some threshold, which would have otherwise caused it to go into the debug stream. It is not necessary for a developer to worry about setting this EDM detail. However, for technical reasons, "allowTruncation" entries need to appear at the end of the EDM list. Therefore, a developer should add any new EDM content before the last "allowTruncation" entries - preferably within the relevant signature section.
+
+### Aliases
+
+A name with suffix `ShallowCopy` is set for all `ShallowAuxContainer` type containers to indicate that they need special treatment by the [HLTEDMCreator]({{data.athena_git_url}}/blob/{{data.branchTier0Short}}/Trigger/TrigSteer/TrigOutputHandling/src/HLTEDMCreator.h) given that the Aux store name differs from the interface name.
+
+## EDM manipulation
+
+The following configuration flag setting will override the "output level" for any AOD output file:
+```python
+flags.Trigger.AODEDMSet='ESD'
+```
+i.e. this will record all `ESD` labelled containers to the output AOD file.
+
+
+The following configuration flag settings will override or add a new EDM entry if not already found:
+```python
+flags.Trigger.ExtraEDMList = [
+    ('xAOD::CaloClusterContainer#HLT_TopoCaloClustersFS',                  'BS ESD AODFULL', 'Calo'),
+    ('xAOD::CaloClusterTrigAuxContainer#HLT_TopoCaloClustersFSAux.nCells', 'BS ESD AODFULL', 'Calo'),
+    ('xAOD::ElectronContainer#HLT_NewParticles',                                'BS ESD AODFULL', 'EGamma', [InViews('NewViews')]),
+    ('xAOD::ElectronAuxContainer#HLT_NewParticlesAux.',                'BS ESD AODFULL', 'EGamma'),
+]
+```
+
+This command will update the targets for the already existing `HLT_TopoCaloClustersFS` and `HLT_TopoCaloClustersFSAux` containers, as well as add new `HLT_NewParticles` and `HLT_NewParticlesAux` containers.
+
+
+## unit tests
+
+TODO!
+
+Two unit tests exist that check for:
+
+- Validity of EDM entries.
+- Ordering of EDM entries.
+
+
+## Procedure for modifying trigger EDM
+
+### Adding content
+
+The following steps should be followed if you would like to **add** new trigger EDM content:
+
+1. Do the necessary software modications to record the new content as described in [section Reconstruction Algorithms: Saving EDM objects](algos.md#saving-edm-objects).
+1. Test and validate: Check content is recorded in the expected output (data AOD, MC AOD etc) and the content is correct. See the subsection below!
+1. Estimate the size increase. See the subsection below!
+1. Contact the trigger EDM coordinator [found on trigger Core s/w twiki](https://twiki.cern.ch/twiki/bin/view/Atlas/TriggerCoreSoftware), providing:
+    (a) a JIRA ticket motivating the new implementation, new EDM content and EDM target, as well as a post of the results of the 2 previous steps.
+    (b) (if ready) a link to the draft Merge Request.
+1. The trigger EDM coordinator will confirm whether the EDM targets are appropriate and sufficiently motivated.
+1. If the change is implemented during a data taking period (e.g. Run 3) to the current Tier0 branch ({{data.branchTier0Short}}), the change requires "PROC" sign-off (Prompt Reconstruction Operation Coordinator) due to the _Frozen Tier0 Policy_ which tries to prevent EDM changes during data taking periods or major MC campaigns -- see details on the [FrozenTier0 twiki](https://twiki.cern.ch/twiki/bin/viewauth/AtlasComputing/FrozenTier0Policy).
+If relevant, you should tag or email PROC after sign-off from the trigger group (FIXME: Check what actual procedure should be). You can find the current coordinators under the [ATLAS Data Preparation organisation page](https://atlaspo.cern.ch/public/ATLASOrganisation/index.html?value=DATAPREP) or on the twiki.
+1. Once the MR is undrafted, there are a couple of CI tests that may fail due to breaking the Frozen Tier0 policy during data taking, namely `CITest_RecoRun3Data_Checks-test` and `CITest_RecoRun3MC-test`. The ERROR will include
+```sh
+ERROR    Py:diff-root        ERROR files differ!
+ERROR    Your change breaks the frozen tier0 policy in test q449.
+ERROR    Please make sure this has been discussed in the correct meeting (RIG or Simulation) meeting and approved by the relevant experts.
+```
+along with details of the diff. The failure is due to outdated AOD/ESD references, which PROC manages. Double-check that the differences are expected and tag the current PROC in your MR.
+The MR will only be merged once the ref updates have been done by PROC (once all tests are green). (TODO: Check - twiki says developers can update references themselves.)
+
+**NOTE** Notifying the Trigger EDM coordinator is very important. Besides reviewing size increases, the trigger EDM coordinator also needs to, post-MR, update the bytestream schema, so that new container types can be serialised as well as ensure that Athena releases for P1/Tier0 and for GlobalMonitoring are synchronised!
+
+**Advice** Usually a new EDM implementation comes along with new trigger chains which need to be motivated in a presentation at the Trigger General Meeting if they are ready to go into the Physics menu. This presentation should also include material motivating the EDM content and the expected EDM size increase.
+
+### Removing content
+
+If you would like to **remove** trigger EDM content, then get in touch with the [trigger EDM coordinator](https://twiki.cern.ch/twiki/bin/view/Atlas/TriggerCoreSoftware) directly, so they can make sure that the content is not used by anyone. **Note** A trigger EDM entry should never be completely removed; the EDM targets should be removed instead.
+
+## Test and validate
+
+The following should be cross-checked for new EDM content:
+
+1. New EDM containers appear in RAW or relevant AOD output.
+1. New EDM containers are populated for events that seed the relevant reconstruction (this is not only events where the relevant chain has passed).
+1. Dynamic values can be accessed and make sense: Are not default values, zeroed, etc.
+1. The size increase is as expected and not exhorbitant.
+
+The results should be documented in JIRA.
+
+!!! warning
+    All EDM containers listed in `TriggerEDMRunX` _will_ appear in the relevant output specified by the EDM target, _regardless_ if they were reconstructed at the HLT: We run a "GapFiller" each time we process produce AODs, which creates empty HLT containers for an event if they don't already exist to ensure a consistent EDM throughout all events. Thus, seeing a container listed doesn't mean it's populated!
+
+### Cross-check: Container recording
+
+The following tests will output AODs after rerunning the HLT on a given menu:
+
+| test | data/MC | EDM target | Menu |
+|-------------|-------------|-------------|-------------|
+| test_trigAna_RDOtoAOD_v1MC_build | ttbar MC | AODSLIM | `"Physics_pp_run3"` |
+| test_trigAna_RDOtoAOD_v1Dev_build | ttbar MC | AODFULL | `"Dev_pp_run3"`|
+| test_trigP1_v1PhysP1_T0Mon_build | data | AODFULL | `"Physics_pp_run3"`|
+
+Each test will output `AOD.pool.root.checkFile` and `AOD.pool.root.checkFileTrigSize` files, produced from
+`checkFile.py AOD.pool.root` and `checkFileTrigSize.py AOD.pool.root`, respectively.
+
+Your container should appear in `AOD.pool.root.checkFile`. Each file provides details on container or total component sizes.
+
+#### Tips on running own trigger tests
+
+When you run tests like `test_trigAna_RDOtoAOD_v1MC_build.py`, `test_trigP1_v1PhysP1_T0Mon_build.py` or `test_trigP1_v1PhysP1_T0MonTrf_build.py`, files are automatically created containing the executables: `command.json` for the former (MC reco) and `runargs.RAWtoALL.py` for the latter. Alternatively, you copy and edit the test directly or check the log file for the executed command. You can use this output as a reference for running your own trigger test with modified settings (different menu, number of events, EDM target etc).
+
+### Cross-check: Container content
+
+First, you want to check that your containers are populated by your chains. A very useful cross-check is to run a trigger test with only your chains present by setting the `Trigger.selectedChains=["chain1", "chain2"]` flag, e.g.:
+
+```sh
+athenaHLT.py -o output --imf --perfmon --threads=4 --concurrent-events=4 --nprocs=1 --number-of-events=50 --file=/cvmfs/atlas-nightlies.cern.ch/repo/data/data-art/TrigP1Test/data24_13p6TeV.00475321.physics_EnhancedBias.merge.RAW._lb0231._SFO-11._0001.1 --file=/cvmfs/atlas-nightlies.cern.ch/repo/data/data-art/TrigP1Test/data24_13p6TeV.00475321.physics_EnhancedBias.merge.RAW._lb0231._SFO-12._0001.1 TriggerJobOpts.runHLT Trigger.triggerMenuSetup="PhysicsP1_pp_run3_v1_HLTReprocessing_prescale" Trigger.selectChains=[\"HLT_j250_a10sd_cssk_ditauOmni0Trk99_pf_jes_ftf_preselj200_L1jJ160\",\"HLT_tau200_mediumGNTau_L1eTAU140\"] Trigger.doLVL1=True Exec.FPE=1 >athenaHLT.log 2>&1
+```
+TODO: This is for data. Easiest way to get skimmed MC on limited menu?
+
+This ensures that your output contains only events that passed your chain, meaning containers should be populated
+and ensures that all reconstruction is performed by _your_ chains
+(relevant if you are adding new chains that use already existing reconstructed objects).
+
+Use an analysis script to retrieve the containers and plot variables to histogram or print to terminal to check that variables look correct.
+
+### Estimate size increase
+
+This can be tricky, especially if your chains have a low selection rate. A size estimate can be given by comparing between the latest Athena branch and your implementation:
+
+- Size of produced RAW (data) files between the current release and your implementation.
+- AOD file size difference quoted in AOD.pool.root.checkFile or summed size of new containers.
+- Trig EDM size difference quoted in AOD.pool.root.checkFileTrigSize.
+
+TODO: Ran out of time. Needs more detail.
+
+Once the implementation is merged to Athena, ART (nightly) tests monitor the trigger EDM size on larger amount (~1000) of MC & data events. These numbers can be referenced in the JIRA. See [the section on trigger EDM size monitoring](validation.md#trigger-edm-size-monitoring) on obtaining the size estimates.

mkdocs.yml

@@ -149,6 +149,7 @@ nav:
       - athena/trigger/developers/hypos.md
       - athena/trigger/developers/combohypos.md
       - athena/trigger/developers/navigation.md
+      - athena/trigger/developers/edm.md
       - athena/trigger/developers/monitoring.md
       - athena/trigger/developers/menu.md
       - athena/trigger/developers/validation.md