athena merge requestshttps://gitlab.cern.ch/atlas/athena/-/merge_requests2024-03-28T15:32:02+01:00https://gitlab.cern.ch/atlas/athena/-/merge_requests/70196Valkyrie: delete legacy job options and executable2024-03-28T15:32:02+01:00Frank WinklmeierValkyrie: delete legacy job options and executableDelete the legacy job options to configure `ValgrindSvc`. Also delete
the (hopefully) unused `valkyrie.py` script. Our transforms support
running valgrind so this should no longer be needed.
Update documentation and apply some name chan...Delete the legacy job options to configure `ValgrindSvc`. Also delete
the (hopefully) unused `valkyrie.py` script. Our transforms support
running valgrind so this should no longer be needed.
Update documentation and apply some name changes.https://gitlab.cern.ch/atlas/athena/-/merge_requests/70188Cleanup string/token writing code in APR Containers2024-03-28T14:28:46+01:00Marcin NowakCleanup string/token writing code in APR ContainersCode cleanup as side effect of the investigation into ROOT string writing bug (in ROOT HEAD)
No functionality changes expectedCode cleanup as side effect of the investigation into ROOT string writing bug (in ROOT HEAD)
No functionality changes expectedhttps://gitlab.cern.ch/atlas/athena/-/merge_requests/701872024-03-28: merge of 24.0 into main2024-03-28T14:34:52+01:00Edward Moyse2024-03-28: merge of 24.0 into mainThis sweep contains the following MRs:
* !70158 athena.py: allow --config-only without storing config (legacy) ~Core
* !70181 Fix bug in han configuration merging ~DQ
* !70149 Update collisions_run.config - add the new han configurati...This sweep contains the following MRs:
* !70158 athena.py: allow --config-only without storing config (legacy) ~Core
* !70181 Fix bug in han configuration merging ~DQ
* !70149 Update collisions_run.config - add the new han configuration for all the l1calo monitoring changes ~DQ, ~L1Calo
* !70151 athena.py: remove cppyy_minvmem command line option ~Core
* !70155 TileMonitoring: Update Tile online monitoring script for cosmics ~DQ, ~Tile
* !70103 Optimization of the Trigger Menu generation: move CA generation after the alignment (ATR-29039) ~JetEtmiss, ~Trigger, ~TriggerMenu
* !69406 24.0 TRT high threshold tune ~Digitization, ~InnerDetector, ~Overlay, ~Run2-DataOverlay-output-changed, ~Run2-MCOverlay-output-changed, ~Run2-MCReco-output-changed, ~Run3-MCOverlay-output-changed, ~Run3-MCReco-output-changed, ~Tools, ~frozen-tier0-violating
* !70127 Improvements in Run2 to Run 3 nav conversion ~Trigger
* !70135 LArConfiguration: delete legacy job option fragments ~LAr
* !70134 LArROD: remove legacy configuration ~LAr
* !70130 TileSimAlgs: Remove legacy JO configurations (ATLASRECTS-7939) ~Digitization, ~Tile
* !70085 remove old taggers and add GN2v01 in PhysVal monitoring ~Analysis, ~BTagging, ~Derivation, ~JetEtmiss
* !70138 Add HLT_noalg_L1MU14EOF to express ~Trigger, ~TriggerMenu
* !70083 Dev new histograms ~DQ, ~LAr, ~Trigger
* !70105 gFEX update jwoj threshold ~Trigger, ~TriggerMenu, ~changes-trigger-counts
* !70121 TileMonitoring: Remove legacy JO configurations (ATLASRECTS-7939) ~DQ, ~Tile
* !70126 TileMuId: Remove legacy JO configurations (ATLASRECTS-7939) ~Tile
* !70128 TileRecAlgs: Remove legacy JO configurations (ATLASRECTS-7939) ~Tile
* !69231 Implementing the MuonSelectionTool for selecting muon tracks. ~Analysis, ~BTagging, ~Derivation, ~JetEtmiss
* !70125 remove obsolete job-config files from CaloRec/share ~Calorimeter
* !70124 TileMonitoring: Remove legacy config from TileMBTSMonitorAlgorithm (ATLASRECTS-7939) ~DQ, ~Tile
* !70122 TileMonitoring: Remove legacy config from TileDQFragMonitorAlgorithm (ATLASRECTS-7939) ~DQ, ~Tile
There was one conflict (in `BTaggingValidationPlots.cxx`). As discussed in !70085, I resolved this by taking the `24.0` changes.https://gitlab.cern.ch/atlas/athena/-/merge_requests/70179Clean up Event/xAOD legacy configs2024-03-28T09:58:33+01:00Thomas StreblerClean up Event/xAOD legacy configs`xAODEventFormatCnv` still to be cleaned up as CI tests need to be migrated to CA
FYI @maszyman`xAODEventFormatCnv` still to be cleaned up as CI tests need to be migrated to CA
FYI @maszymanhttps://gitlab.cern.ch/atlas/athena/-/merge_requests/70156Draft: write in-file metadata using RootKeyContainer2024-03-27T17:50:55+01:00Maciej Pawel SzymanskiDraft: write in-file metadata using RootKeyContainerAn experiment to check what's needed to switch metadata storage technologyAn experiment to check what's needed to switch metadata storage technologyhttps://gitlab.cern.ch/atlas/athena/-/merge_requests/70150PyUtils: Make ##Params storage technology explicit2024-03-28T10:28:42+01:00Grigori RybkinPyUtils: Make ##Params storage technology explicitTools/PyUtils/python/AthFileLite.py: Extract file GUID in a more Pythonic way
Tools/PyUtils/python/MetaReader.py: idem
Related to ATEAM-960Tools/PyUtils/python/AthFileLite.py: Extract file GUID in a more Pythonic way
Tools/PyUtils/python/MetaReader.py: idem
Related to ATEAM-960https://gitlab.cern.ch/atlas/athena/-/merge_requests/70111CaloMonitoring: delete legacy job options and ROOT macros2024-03-28T14:37:46+01:00Frank WinklmeierCaloMonitoring: delete legacy job options and ROOT macrosDelete legacy job options, root macros and updates link dependencies.Delete legacy job options, root macros and updates link dependencies.Pavol StrizenecPavol Strizenechttps://gitlab.cern.ch/atlas/athena/-/merge_requests/70106Draft: ActsGeometryContext - Non const access2024-03-27T08:20:11+01:00Johannes Junggeburthjohannes.josef.junggeburth@cern.chDraft: ActsGeometryContext - Non const accesshttps://gitlab.cern.ch/atlas/athena/-/merge_requests/69793AthContainers, etc: Move Accessor, etc. classes out of AuxElement.2024-03-28T09:53:38+01:00Scott SnyderAthContainers, etc: Move Accessor, etc. classes out of AuxElement.Accessor and related classes have been members of AuxElement, the original
thinking being that derived classes of AuxElement may want to redefine them.
However, in retrospect, there does not seem to be any circumstance in which
that woul...Accessor and related classes have been members of AuxElement, the original
thinking being that derived classes of AuxElement may want to redefine them.
However, in retrospect, there does not seem to be any circumstance in which
that would actually make sense. Further, it prevents having any specializations
for these classes. So move these classes out of AuxElement into the SG
namespace (but with the names brought into AuxElement via using declarations
for backwards compatibility).https://gitlab.cern.ch/atlas/athena/-/merge_requests/69701Adding sTGC xAOD PRD output2024-03-28T15:18:05+01:00Edward MoyseAdding sTGC xAOD PRD outputAs with !69514, this adds the ability for the RDO to PRD convertor to output xAOD PRDs directly, and adds them to the test.
There is some missing functionality, since:
- we do not have access to the ReadoutElement
- I do not yet know ho...As with !69514, this adds the ability for the RDO to PRD convertor to output xAOD PRDs directly, and adds them to the test.
There is some missing functionality, since:
- we do not have access to the ReadoutElement
- I do not yet know how to handle merging xAOD PRDs
This will come in follow up MRs. This is all disabled by default.
cc @jojunggehttps://gitlab.cern.ch/atlas/athena/-/merge_requests/69664Draft: athena.py: implement native support for CA2024-03-28T10:37:33+01:00Frank WinklmeierDraft: athena.py: implement native support for CAThis MR is a refactor of the functionality of `ThinCAWrapper.sh` and `CARunner.py` into the main `athena` executable. After the initial shell-phase, the process remains in python. This is achieved using the `runpy` module to directly lau...This MR is a refactor of the functionality of `ThinCAWrapper.sh` and `CARunner.py` into the main `athena` executable. After the initial shell-phase, the process remains in python. This is achieved using the `runpy` module to directly launch the CA script via the `__main__` entry point.
This MR is the first step in making the `--CA` option obsolete.
#### User visible changes:
* The interactive mode now requires an optional \<stage\> argument also for legacy athena. If this is directly followed by the job options, the arguments need to be terminated by `--`. E.g all of the following are equivalent:
```plaintext
> athena AthExHelloWorld/HelloWorldOptions.py -i
> athena -i -- AthExHelloWorld/HelloWorldOptions.py
> athena -i init AthExHelloWorld/HelloWorldOptions.py
```
* If the user tries to run a CA script in legacy mode, the following will be printed:
```plaintext
AthenaCommon.Include.IncludeError: include file AthExHelloWorld/HelloWorldConfig.py can not be found
If the script is a ComponentAccumulator configuration, add a shebang (#!) to its first line or run with the --CA option.
```
#### Implementation details:
* Use the `AthOptionsParser` for both legacy athena and CA-based job/athena. If used within `athena.py` the athena-specific command line options are added.
* The shell-based CLI-parsing is now restricted to what is necessary to setup the preloading.
* Implement the file path resolution in Python using `unixtools.FindFile`. The "shebang" CA detection now also works for file paths that are resolved at runtime.
* The `CARunner.py` use-case is replaced by directly loading the pkl file (legacy/CA pickles are automatically distinguished). I dropped the support of CA pkls without MainServicesCfg. Could be added again if really needed.
* Migrate the `--tracelevel` feature into `AthenaCommon.Debugging`. We should probably unify this with the existing `--trace` option in a second iteration.
For the moment, we still support the `--CA` flag. But if the proper shebang is used, the following commands all work transparently without the need to use `--CA`:
```sh
athena.py AthExHelloWorld/HelloWorldOptions.py
athena.py --config-only legacy.pkl AthExHelloWorld/HelloWorldOptions.py
athena.py legacy.pkl
athena.py AthExHelloWorld/HelloWorldConfig.py
athena.py --config-only cfg.pkl AthExHelloWorld/HelloWorldConfig.py
athena.py cfg.pkl
```
cc @wlampl @tadej @ssnyder
TODO:
* [ ] Delete `CARunner.py` after migrating the few uses to `athena.py`
* [ ] Backport to ~"24.0" once validated in ~mainhttps://gitlab.cern.ch/atlas/athena/-/merge_requests/69592Draft: Removal of H6 TB simu2024-03-08T08:59:33+01:00Pavol StrizenecDraft: Removal of H6 TB simuThis should remove H6 TB simulations, as agreed on https://its.cern.ch/jira/browse/ATLASSIM-7104This should remove H6 TB simulations, as agreed on https://its.cern.ch/jira/browse/ATLASSIM-7104https://gitlab.cern.ch/atlas/athena/-/merge_requests/69569MetaReader: print warning when ignoring all but first entry from /TagInfo fields2024-03-11T14:57:18+01:00Maciej Pawel SzymanskiMetaReader: print warning when ignoring all but first entry from /TagInfo fieldsFollow-up from https://gitlab.cern.ch/atlas/athena/-/merge_requests/68982/ on request from @tstreble in [ATEAM-958](https://its.cern.ch/jira/browse/ATEAM-958?focusedId=6302527&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acommen...Follow-up from https://gitlab.cern.ch/atlas/athena/-/merge_requests/68982/ on request from @tstreble in [ATEAM-958](https://its.cern.ch/jira/browse/ATEAM-958?focusedId=6302527&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-6302527).https://gitlab.cern.ch/atlas/athena/-/merge_requests/69513Use config flags in analysis config2024-03-28T13:34:35+01:00Tadej Novaktadej.novak@cern.chUse config flags in analysis configMake config flags the main way to steer analysis config. This will help people working on analysis and Athena to feel at home and prevent accumulation of arguments of the `ConfigAccumulator`.
The naming is not fixed, I will probably als...Make config flags the main way to steer analysis config. This will help people working on analysis and Athena to feel at home and prevent accumulation of arguments of the `ConfigAccumulator`.
The naming is not fixed, I will probably also convert more items to flags. This is mainly to start the discussion and freeze the API as soon asp possible. I will probably then break this into multiple MRs.
Also some core flags should be renamed (see https://its.cern.ch/jira/browse/ATEAM-964).
Tagging @krumnack, @jolamber, @gwatts, @ekourlit for core AMG. Also tagging interested parties @ravinab, @omajersk, @tstreble, @khoo, @jchapman.https://gitlab.cern.ch/atlas/athena/-/merge_requests/69417Draft: Remove Legacy to CA conversion utilities2024-03-08T12:37:40+01:00Tomasz BoldDraft: Remove Legacy to CA conversion utilitiesThis MR is a test to see how far we are from fully relying on CA job options.This MR is a test to see how far we are from fully relying on CA job options.https://gitlab.cern.ch/atlas/athena/-/merge_requests/68758MuonTrackingGeometry fix memory bugs2024-03-28T09:12:51+01:00Johannes Junggeburthjohannes.josef.junggeburth@cern.chMuonTrackingGeometry fix memory bugsRework the memory management of the muon tracking geometry and the Geo -> Trk::Volume converter. Cone clean up etc. Few bug fixes.
Validation tests perfromed on simulation Run2 & Run 3
![image](/uploads/a6228ba4e68dfa1ae2c86416e850e7...Rework the memory management of the muon tracking geometry and the Geo -> Trk::Volume converter. Cone clean up etc. Few bug fixes.
Validation tests perfromed on simulation Run2 & Run 3
![image](/uploads/a6228ba4e68dfa1ae2c86416e850e73c/image.png)
![image](/uploads/2ea2da76b97d84b97364d60fff524b30/image.png)
![image](/uploads/11c1818a9a4511ba42503932d9f32bdf/image.png)
![image](/uploads/cc765f3ff84e76aed07300dae506493a/image.png)
![image](/uploads/edc265d2f6d791370dcddfced77be744/image.png)
Further 250k data events from DRAW_ZMUMU have been reprocessed
![image](/uploads/ddac5de860349f24cb60a48d5be657b4/image.png)
![image](/uploads/ba8ba82ebaeb84eb2fa3a36a1c7cb07a/image.png)
![image](/uploads/4846fdd866a27bb8ec1db75caac31e90/image.png)
![image](/uploads/5efd30d930e56420e452877f78ac413d/image.png)
![image](/uploads/4ccb61e8b448e6de80732baf284fc6bb/image.png)
Tagging @todorova, @pscholer, @zyan, @nstyles, @jcatmore, @jchapman, @kluit
Closes ATLASRECTS-7876
The full plot bock is attached to the JIRA ticket because the file size limit there is larger than herehttps://gitlab.cern.ch/atlas/athena/-/merge_requests/68582Draft: [ATR-28536][ATR-28717] Working towards a trigger EDM refactor and inte...2024-02-19T10:57:11+01:00Tim MartinDraft: [ATR-28536][ATR-28717] Working towards a trigger EDM refactor and integrating R4 into the EDM (part B)Follows from !68370
I would like some input on these proposed changes, cc @cantel, @okuprash, @wiedenma, @fwinkl before un-drafting (also I am trampling over some bjet changes I need to undo before un-drafting)
Here in part B, I aim to...Follows from !68370
I would like some input on these proposed changes, cc @cantel, @okuprash, @wiedenma, @fwinkl before un-drafting (also I am trampling over some bjet changes I need to undo before un-drafting)
Here in part B, I aim to make the following changes
* Change the signature of `getTriggerEDMList` to require the config flags (but make it such that it doesn't crash if you supply `None` for some e.g. standalone test use cases). Update all clients.
* Introduce a new function, `getRawTriggerEDMList` which takes `flags`. Every user of the `TriggerEDMRun3.TriggerHLTListRun3` should obtain this list via this function such that every client gets a consistent copy. Updated all clients.
* Mark the internal functions inside `TriggerEDM` with an initial underscore.
* Removed unused functions inside `TriggerEDM`.
* Move the `addExtraCollectionsToEDMList` function from the `TriggerEDMRun3` file to the `TriggerEDM` file.
To follow in the next part:
* Attempt to refactor `addHLTNavigationToEDMList` using the new `flags.Trigger.ExtraEDMFunction` - this will be hard...
* Add caching to the `getRawTriggerEDMList` function.
* Make it harder to obtain `TriggerEDMRun3.TriggerHLTListRun3` without going via `getRawTriggerEDMList(flags, 3)`, maybe wrap this list in a simple class.
* Connect up the Run 4 EDM filehttps://gitlab.cern.ch/atlas/athena/-/merge_requests/67630Draft: Use PropertyProxy from Gaudi2024-02-19T11:20:40+01:00Frank WinklmeierDraft: Use PropertyProxy from GaudiExperimental MR to migrate to Gaudi's `PropertyProxy` (and `ConfigurableDb`). In principle there are only very few differences and this would avoid having to maintain our own copy each time a new property type gets added.
Ideally, we wo...Experimental MR to migrate to Gaudi's `PropertyProxy` (and `ConfigurableDb`). In principle there are only very few differences and this would avoid having to maintain our own copy each time a new property type gets added.
Ideally, we would do the same for the (legacy) `Configurable` class but that requires more cleanup first (i.e. removing the support of the CA behavior once we no longer need it).