diff --git a/docs/athena/trigger/developers/algos.md b/docs/athena/trigger/developers/algos.md index ff9e7d9c85b0ef10bf36979b33cfb563f99351d4..abfb3b4a0e171a6dedbf938e137ac7a133b8e0b1 100644 --- a/docs/athena/trigger/developers/algos.md +++ b/docs/athena/trigger/developers/algos.md @@ -44,18 +44,16 @@ ready) [fast e/gamma cluster maker algorithm]({{data.athena_git_url}}/blob/{{dat ## Saving EDM objects -If the reconstructed objects that are created by algorithms -need to be recorded in the output file these steps should be followed: +The following software modifications are needed to record reconstructed objects created by HLT algorithms to file: -1. contact trigger EDM coordinator [found on trigger Core s/w twiki](https://twiki.cern.ch/twiki/bin/view/Atlas/TriggerCoreSoftware) 1. in the python configuration of the algorithm an instance of the - `recordable` function from [TrigEDMConfig.TriggerEDMRun3]({{data.athena_git_url}}/blob/{{data.branchTier0Short}}/Trigger/TriggerCommon/TrigEDMConfig/python/TriggerEDMRun.py#L34) needs + `recordable` function from [TrigEDMConfig.TriggerEDM]({{data.athena_git_url}}/blob/{{data.branchTier0Short}}/Trigger/TriggerCommon/TrigEDMConfig/python/TriggerEDM.py#L34) needs to be used when setting the name of the collection. It assures that - the naming convention of the HLT EDM is followed and no trivial typos sneak in between name of the container . -1. The collection (type and name) has to be listed in - [TrigEDMConfig.TriggerEDMRun3]({{data.athena_git_url}}/blob/{{data.branchTier0Short}}/Trigger/TriggerCommon/TrigEDMConfig/python/TriggerEDMRun3.py) + the naming convention of the HLT EDM is followed and no trivial typos sneak in. +1. The collection (type and name) has to be listed in + [TrigEDMConfig.TriggerEDMRun3]({{data.athena_git_url}}/blob/{{data.branchTier0Short}}/Trigger/TriggerCommon/TrigEDMConfig/python/TriggerEDMRun3.py) (or `TriggerEDMRun4` for upgrade studies) together with the required output file types in which it should be recorded. Typically the object should be stored - at least in `BS`, `ESD` and `AOD`. Consult the Trigger EDM coordinator about this. Especially about the `AOD` formats where the size is constrained. + at least in `BS` and `ESD` (`RDO_TRIG` files - the equivalent of "raw" files for MC - contain all `ESD`-labelled content). 1. Ensure that in [HLTEDMCreator.h]({{data.athena_git_url}}/blob/{{data.branchTier0Short}}/Trigger/TrigSteer/TrigOutputHandling/src/HLTEDMCreator.h) and @@ -69,7 +67,9 @@ need to be recorded in the output file these steps should be followed: Once the above steps are done the object should appear in the relevant output files. For POOL files (ESD, AOD) the content can be checked with one of the following scripts: `checkxAOD.py filename` `checkSG.py filename` `checkFile.py filename`. - + +See the [trigger EDM guidelines](edm.md#procedure-for-modifying-trigger-edm) on the full procedure for adding new trigger EDM content (testing, verification and approval). + ## Troubleshooting Typically a reconstruction algorithm uses many input data diff --git a/docs/athena/trigger/developers/edm.md b/docs/athena/trigger/developers/edm.md new file mode 100644 index 0000000000000000000000000000000000000000..94f5b7677588100c7e2e69415eccea591607a129 --- /dev/null +++ b/docs/athena/trigger/developers/edm.md @@ -0,0 +1,241 @@ +# Trigger EDM + +The Trigger Event Data Model (EDM) defines the trigger reconstructed content that is written out to RAW files and data or MC AODs. +The Trigger EDM content for Run 3 is defined in [TrigEDMConfig.TriggerEDMRun3]({{data.athena_git_url}}/blob/{{data.branchTier0Short}}/Trigger/TriggerCommon/TrigEDMConfig/python/TriggerEDMRun3.py) (the same folder contains similar files for other periods). +Each EDM entry is a tuple containing all EDM details for a specific interface or Aux container. An example of two (interface + aux) EDM entries: +```python +('xAOD::CaloClusterContainer#HLT_CaloEMClusters_Electron', 'BS ESD AODFULL AODSLIM', 'Egamma', [InViews('precisionCaloElectronViews')]), +('xAOD::CaloClusterTrigAuxContainer#HLT_CaloEMClusters_ElectronAux.', 'BS ESD AODFULL AODSLIM', 'Egamma'), + +('xAOD::CaloClusterContainer#HLT_TopoCaloClustersLC', 'BS ESD AODFULL', 'Tau', [InViews('tauCaloMVAViews'), allowTruncation]), +('xAOD::CaloClusterTrigAuxContainer#HLT_TopoCaloClustersLCAux.nCells.CENTER_MAG', 'BS ESD AODFULL', 'Tau', [allowTruncation]), + +``` +Every EDM entry will at minimum define the container type and container name, the "EDM targets" and trigger signature. Any additional dynamic Aux variables +to be recorded are specified as part of the Aux container name string. +The EDM targets define the output type to which the containers will be recorded in the case of `RAW`, `RDO_TRIG`, `ESD` and `AOD` files, as well as PHYSVAL `DAOD` files. The `ESD` level is used for MC `RDO_TRIG` (equivalent to data `RAW` files for MC) production. We use these to restrict the HLT content when not all HLT content is needed, +e.g. in MC AODs for physics analysis derivations. The trigger signature is used to categorise total EDM size by signature in the [Trigger EDM monitoring](validation.md#trigger-edm-size-monitoring). + +## EDM targets + +The EDM target labels (2nd entry in an EDM entry) are mapped to the following output: + +| Label | Output | Purpose | +|-------------|-------------|-------------| +|`BS`| ByteStream - Any content that is to make it out of Point 1 and be recorded to RAW file. | Required for recording to data AOD after or kept in BS for debugging in case of data taking issues.| +|`ESD`| ESD files, RDO_TRIG files, SampleA productions | Low-level reconstructed information such as HLT calorimeter clusters for specialised reconstruction studies. Also used for `RDO_TRIG` (MC equivalent of a RAW file) output.| +|`AODFULL`| data AOD at Tier0, trigger developer AOD production requests, HLT Reprocessing AODs. | Required for regular trigger efficiency studies in data or validation during commissioning.| +|`AODSLIM`| bulk MC production AODs for physics analysis, SampleT productions. | Required for physics analysis. Examples: Final precision HLT leptons/b-jets needed for trigger matching and trigger scale factors.| +|`PhysicsTLA`/`EgammaPEBTLA`/`DarkJetPEBTLA`/`FTagPEBTLA`| respective RAW output of a Trigger-Level Analysis (TLA) or TLA Partial Event Building stream| Content required for a trigger-level physics analysis or TLA with partial event building (PEB) information.| + +**Important**: Think hard about what targets you need, as the HLT content in total has been taking up a signficant fraction of disk space. + +Valid reasons to keep trigger content in `AODFULL`: If needed for regular trigger efficiency checks in recorded data AOD or if it is a new implementation that needs a few months of commissioning. + +Valid reasons to keep trigger content in `AODSLIM`: If needed for physics analysis, namely trigger matching and scale factor derivations. + +### Note on AODSLIM: Further reductions + +One can specify Run 3 container dynamic variables that are not needed in `AODSLIM`, i.e. variables that are not needed by physics analyses. This is set via the `varToRemoveFromAODSLIM` list in [TrigEDMConfig.TriggerEDMRun3]({{data.athena_git_url}}/blob/{{data.branchTier0Short}}/Trigger/TriggerCommon/TrigEDMConfig/python/TriggerEDMRun3.py). + + +## Optional EDM details + +There are several additional EDM details that can be specified as a list constituting the last entry in the tuple. + +### InViews + +In the case of reconstruction in a limited region of the detector, this names the event view in which this collection is reconstructed, so that additional information allowing the items to be accessed in event views is recorded. For more details, see [the section on Event Views](eventviews.md). + +### allowTruncation + +This flag is only valid for data taking and is used to indicate that this container can be dropped from an event if the event data is larger than some threshold that would have resulted in the event data going to the debug stream. A developer need not worry about setting this EDM detail. However, for technical reasons, "allowTruncation" entries need to appear at the end of the EDM list. Therefore, a developer should add any new EDM content _before_ the last "allowTruncation" entries and, more specifically, within the relevant signature section. + +### Aliases + +A name with suffix `ShallowCopy` is set for all `ShallowAuxContainer` type containers to indicate that they need special treatment by the [HLTEDMCreator]({{data.athena_git_url}}/blob/{{data.branchTier0Short}}/Trigger/TrigSteer/TrigOutputHandling/src/HLTEDMCreator.h) given that the Aux store name differs from the interface name. + +## unit tests + +Unit tests exist that check the validity, ordering and manipulation of EDM entries. + +## EDM manipulation + +The following configuration flag setting will override the "output level" for any AOD output file: +```python +flags.Trigger.AODEDMSet='ESD' +``` +i.e. this will record all `ESD` labelled containers to the output AOD file. + + +The following configuration flag settings will override or add a new EDM entry if not already found: +```python +flags.Trigger.ExtraEDMList = [ + ('xAOD::CaloClusterContainer#HLT_TopoCaloClustersFS', 'BS ESD AODFULL', 'Calo'), + ('xAOD::CaloClusterTrigAuxContainer#HLT_TopoCaloClustersFSAux.nCells', 'BS ESD AODFULL', 'Calo'), + ('xAOD::ElectronContainer#HLT_NewParticles', 'BS ESD AODFULL', 'EGamma', [InViews('NewViews')]), + ('xAOD::ElectronAuxContainer#HLT_NewParticlesAux.', 'BS ESD AODFULL', 'EGamma'), +] +``` + +This command will update the targets for the already existing `HLT_TopoCaloClustersFS` and `HLT_TopoCaloClustersFSAux` containers, as well as add new `HLT_NewParticles` and `HLT_NewParticlesAux` containers. + +## Procedure for modifying trigger EDM + +### Adding content + +The following steps should be followed if you would like to **add** new trigger EDM content: + +1. Do the necessary software modications to record the new content as described in [section Reconstruction Algorithms: Saving EDM objects](algos.md#saving-edm-objects). +1. Test and validate: Check content is recorded in the expected output (data AOD, MC AOD etc) and the content is correct. See the subsection below! +1. Estimate the size increase. See the subsection below! +1. Contact the trigger EDM coordinator [found on the trigger Core s/w twiki](https://twiki.cern.ch/twiki/bin/view/Atlas/TriggerCoreSoftware), providing: + 1. (likely already existing) JIRA ticket motivating the new implementation, in which is posted the motivation for the new EDM content and EDM target and the results of the 2 previous steps. + 1. (if ready) a link to the draft Merge Request. +1. The trigger EDM coordinator will confirm whether the EDM targets are appropriate and sufficiently motivated. +1. Submit/undraft the MR + 1. If an MR changes the EDM during a data taking period (e.g. Run 3) to the current Tier0 branch ({{data.branchTier0Short}}), the MR will require "PROC" sign-off (Prompt Reconstruction Operation Coordinator) due to the _Frozen Tier0 Policy_ which tries to prevent EDM changes during data taking periods or major MC campaigns -- see details on the [FrozenTier0 twiki](https://twiki.cern.ch/twiki/bin/viewauth/AtlasComputing/FrozenTier0Policy). There will be failing CI tests due to this policy, namely `CITest_RecoRun3Data_Checks-test` and `CITest_RecoRun3MC-test`: +```sh +ERROR Py:diff-root ERROR files differ! +ERROR Your change breaks the frozen tier0 policy in test q449. +ERROR Please make sure this has been discussed in the correct meeting (RIG or Simulation) meeting and approved by the relevant experts. +``` +along with details of the diff. **Double-check that the differences are expected** and tag the current PROC in your MR, providing the JIRA link with motivation (see [ATLAS Data Preparation organisation page](https://atlaspo.cern.ch/public/ATLASOrganisation/index.html?value=DATAPREP) or [twiki](https://twiki.cern.ch/twiki/bin/viewauth/AtlasComputing/FrozenTier0Policy) for current coordinators). +The MR will only be merged once the AOD/ESD references have been updated with your changes by PROC (once all tests are green). +1. When requested to give a presentation of your new implementation at the Trigger General Meeting, provide 1-2 slides motivating the trigger EDM changes and size estimates. + +!!! note + Notifying the Trigger EDM coordinator is very important. Besides reviewing size increases, the trigger EDM coordinator may also need to, post-MR, update the bytestream schema, so that new container types can always be serialised, as well as ensure that Athena releases for P1/Tier0 and for GlobalMonitoring are synchronised! + +### Removing content + +If you would like to **remove** trigger EDM content, then get in touch with the [trigger EDM coordinator](https://twiki.cern.ch/twiki/bin/view/Atlas/TriggerCoreSoftware) directly, so they can make sure that the content is not used by anyone. **Note** A trigger EDM entry should never be completely removed; the EDM targets should be removed instead. + +## Test and validate + +The following should be cross-checked for new EDM content: + +1. New EDM containers appear in RAW or relevant AOD output. +1. New EDM containers are populated by your trigger reconstruction (Note: Containers will be filled with the online reconstruction for testing your trigger hypothesis, even if your trigger didn't pass). +1. Dynamic values can be accessed and make sense: Are not default values, etc. +1. The size increase is as expected and not exorbitant. + +The results should be documented in JIRA. + +#### Useful trigger tests + +The following tests will output AODs after rerunning the HLT on a given menu: + +| test | data/MC | EDM target | Menu | +|-------------|-------------|-------------|-------------| +| test_trigAna_RDOtoAOD_v1MC_build | ttbar MC | AODSLIM | `"Physics_pp_run3"` | +| test_trigAna_RDOtoAOD_v1Dev_build | ttbar MC | AODFULL | `"Dev_pp_run3"`| +| test_trigP1_v1PhysP1_T0Mon_build | data | AODFULL | `"Physics_pp_run3"`| + +Each test will output `AOD.pool.root.checkFile` and `AOD.pool.root.checkFileTrigSize` files, produced from +`checkFile.py AOD.pool.root` and `checkFileTrigSize.py AOD.pool.root`, respectively: `checkFile` output provides sizes per container element, `checkFileTrigSize` provides EDM sizes for the total _trigger_ EDM as well as individual trigger signatures. + + +#### Tips on running own trigger tests + +When you run tests like `test_trigAna_RDOtoAOD_v1MC_build.py` or `test_trigP1_v1PhysP1_T0Mon_build.py`, files are automatically created containing the executables: `commands.json`, as well as `runargs.RAWtoALL.py` for the latter. Alternatively, you copy and edit the test directly. You can use this output as a reference for running your own trigger test with modified settings (different menu, number of events, EDM target etc). + +### Cross-check: Container recording + +Your container should appear in `AOD.pool.root.checkFile` for a relevant ("AODFULL", "AODSLIM" ...) AOD file. + +!!! warning + All EDM containers listed in `TriggerEDMRunX` _will_ appear in the relevant output specified by the EDM target, _regardless_ if they were reconstructed at the HLT: We run a "GapFiller" each time we produce AODs, which creates empty HLT containers for an event if they don't already exist to ensure a consistent EDM throughout all events. Thus, seeing a container listed doesn't mean it's populated! + +### Cross-check: Container content + +You should verify that new containers _are_ being filled and that they are populated by _your_ chains. A very useful cross-check is to run a trigger test on a limited menu containing your chains only by setting `flags.Trigger.selectedChains=["my_new_chain"]`. +This ensures that all reconstruction and hence container content is the result of _your_ chains (especially relevant if you are adding new chains that write to already existing containers). +Ideally, you run the trigger task on an input MC RDO file containing a good fraction of events that pass your triggers. The ttbar MC RDO used in `test_trigAna_RDOtoAOD_v1MC_build.py` may be sufficient. + +Use an analysis script to retrieve and plot EDM variables to histogram or open the AOD or RDO_TRIG file in interactive ROOT or ROOT browser to check that variable distributons look correct. + +### Estimate size increase + +The size increase that primarily matters to us are for: + +- `test_trigAna_RDOtoAOD_v1MC_build` AOD output ("AODSLIM" content), run on the MC menu +- `test_trigP1_v1PhysP1_T0Mon_build` RAW and AOD output ("AODFULL" content), run on the Physics menu. + +It is useful to have the size increase quoted as percentage relative to: total file size, total trigger EDM size (from `checkFileTrigSize`) and relevant trigger signature size (from `checkFileTrigSize`). + +One can measure the size increase by: + +- Comparing output file sizes (e.g. `AOD.pool.root`, `*.physics_Main*._athenaHLT*.data`) between release {{data.branchTier0Short}} and development branch with your trigger chains included in the physics menu. +- Comparing numbers printed in `checkFile` and `checkFileTrigSize` output: e.g. `AOD.pool.root.checkFile` provides total file size + size for individual EDM elements, `AOD.pool.root.checkFileTrigSize` provides sizes for total trigger EDM and for individual signatures. + +Example terminal output comparing total file, trigger EDM and individual trigger signature sizes: + +```console +du 24.0/AOD.pool.root +4554 24.0/AOD.pool.root +du my_dev_branch/AOD.pool.root +4573 my_dev_branch/AOD.pool.root +... +head 24.0/AOD.pool.root.checkFile +PyRoot: Styles set by rootlogin.C +## opening file [AOD.pool.root]... +## importing ROOT... +## importing ROOT... [DONE] +## opening file [OK] +File:AOD.pool.root +Size: 4553.341 kb +Nbr Events: 20 + +head my_dev_branch/AOD.pool.root.checkFile +PyRoot: Styles set by rootlogin.C +## opening file [AOD.pool.root]... +## importing ROOT... +## importing ROOT... [DONE] +## opening file [OK] +File:AOD.pool.root +Size: 4572.145 kb +Nbr Events: 20 + +... +grep triggerTau 24.0/AOD.pool.root.checkFileTrigSize +triggerTau 3.184 +grep triggerTau my_dev_branch/AOD.pool.root.checkFileTrigSize +triggerTau 4.067 +... +grep "Total Trigger Size" 24.0/AOD.pool.root.checkFileTrigSize +Total Trigger Size 51.414 +grep "Total Trigger Size" my_dev_branch/AOD.pool.root.checkFileTrigSize +Total Trigger Size 52.311 +``` + +Example terminal output looking at the "Size/Evt" of individual EDM additions in `AOD.pool.root.checkFile`and comparing to other similar containers - the "Size/Evt" for other containers can be retrieved from the "checkFile" output if run on the full MC/Physics menu or from the [trigger EDM size monitoring webpage](https://test-atrvshft.web.cern.ch/test-atrvshft/TriggerEDMSizeMonitoring/webpage/) for the MC and PhysP1 tests mentioned above: + +```console +================================================================================ + Mem Size Disk Size Size/Evt MissZip/Mem items (X) Container Name (X=Tree|Branch) +================================================================================ +grep HLT_TrigTauRecMerged_MVA my_dev_branch/AOD.pool.root.checkFile + 1.049 kb 0.256 kb 0.013 kb 0.000 20 (B) HLT_TrigTauRecMerged_MVA + 1.121 kb 0.316 kb 0.016 kb 0.000 20 (B) HLT_TrigTauRecMerged_MVAAuxDyn.GNTau_Loose + 1.121 kb 0.316 kb 0.016 kb 0.000 20 (B) HLT_TrigTauRecMerged_MVAAuxDyn.GNTau_Tight + 1.126 kb 0.317 kb 0.016 kb 0.000 20 (B) HLT_TrigTauRecMerged_MVAAuxDyn.GNTau_Medium + 1.141 kb 0.324 kb 0.016 kb 0.000 20 (B) HLT_TrigTauRecMerged_MVAAuxDyn.GNTau_VeryLoose + 1.939 kb 0.338 kb 0.017 kb 0.000 20 (B) HLT_TrigTauRecMerged_MVAAuxDyn.viewIndex + 1.426 kb 0.480 kb 0.024 kb 0.000 20 (B) HLT_TrigTauRecMerged_MVAAuxDyn.GNTau_Score + 1.465 kb 0.484 kb 0.024 kb 0.000 20 (B) HLT_TrigTauRecMerged_MVAAuxDyn.GNTau_ScoreSigTrans + 144.458 kb 20.145 kb 1.007 kb 0.000 20 (B) HLT_TrigTauRecMerged_MVAAux. +grep HLT_TrigTauRecMerged_LLP my_dev_branch/AOD.pool.root.checkFile + 1.049 kb 0.256 kb 0.013 kb 0.000 20 (B) HLT_TrigTauRecMerged_LLP + 1.541 kb 0.350 kb 0.017 kb 0.000 20 (B) HLT_TrigTauRecMerged_LLPAuxDyn.viewIndex + 88.093 kb 12.688 kb 0.634 kb 0.000 20 (B) HLT_TrigTauRecMerged_LLPAux. +grep HLT_TrigTauRecMerged_MVA my_dev_branch/AOD.pool.root.checkFile | awk '{SIZE=$5;SUM+=SIZE}END{print SUM}' +1.149 +grep HLT_TrigTauRecMerged_LLP my_dev_branch/AOD.pool.root.checkFile | awk '{SIZE=$5;SUM+=SIZE}END{print SUM}' +0.664 +grep triggerTau my_dev_branch/AOD.pool.root.checkFileTrigSize +triggerTau 4.067 +``` + +!!! note + It can be difficult obtaining a good size estimate if your triggers have a low selection rate. However, once your EDM changes are merged to the official Athena branch, ART (nightly) tests monitor the trigger EDM size on a larger amount (~1000) of MC & data events, which will give us a more accurate idea of the size increase. It is good to reference these numbers in your JIRA ticket once available (absolute "size/evt" values as well as relative size increase). See [the section on trigger EDM size monitoring](validation.md#trigger-edm-size-monitoring) on obtaining these size estimates. diff --git a/mkdocs.yml b/mkdocs.yml index be30c479a023df1cca37626a74ee1c674b087cf5..bb4353d9c97bb24a4e99eb9c91795477e0e6cf51 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -149,6 +149,7 @@ nav: - athena/trigger/developers/hypos.md - athena/trigger/developers/combohypos.md - athena/trigger/developers/navigation.md + - athena/trigger/developers/edm.md - athena/trigger/developers/monitoring.md - athena/trigger/developers/menu.md - athena/trigger/developers/validation.md