athena merge requestshttps://gitlab.cern.ch/atlas/athena/-/merge_requests2024-03-28T18:56:30+01:00https://gitlab.cern.ch/atlas/athena/-/merge_requests/701872024-03-28: merge of 24.0 into main2024-03-28T18:56:30+01:00Edward Moyse2024-03-28: merge of 24.0 into mainThis sweep contains the following MRs:
* !70158 athena.py: allow --config-only without storing config (legacy) ~Core
* !70181 Fix bug in han configuration merging ~DQ
* !70149 Update collisions_run.config - add the new han configurati...This sweep contains the following MRs:
* !70158 athena.py: allow --config-only without storing config (legacy) ~Core
* !70181 Fix bug in han configuration merging ~DQ
* !70149 Update collisions_run.config - add the new han configuration for all the l1calo monitoring changes ~DQ, ~L1Calo
* !70151 athena.py: remove cppyy_minvmem command line option ~Core
* !70155 TileMonitoring: Update Tile online monitoring script for cosmics ~DQ, ~Tile
* !70103 Optimization of the Trigger Menu generation: move CA generation after the alignment (ATR-29039) ~JetEtmiss, ~Trigger, ~TriggerMenu
* !69406 24.0 TRT high threshold tune ~Digitization, ~InnerDetector, ~Overlay, ~Run2-DataOverlay-output-changed, ~Run2-MCOverlay-output-changed, ~Run2-MCReco-output-changed, ~Run3-MCOverlay-output-changed, ~Run3-MCReco-output-changed, ~Tools, ~frozen-tier0-violating
* !70127 Improvements in Run2 to Run 3 nav conversion ~Trigger
* !70135 LArConfiguration: delete legacy job option fragments ~LAr
* !70134 LArROD: remove legacy configuration ~LAr
* !70130 TileSimAlgs: Remove legacy JO configurations (ATLASRECTS-7939) ~Digitization, ~Tile
* !70085 remove old taggers and add GN2v01 in PhysVal monitoring ~Analysis, ~BTagging, ~Derivation, ~JetEtmiss
* !70138 Add HLT_noalg_L1MU14EOF to express ~Trigger, ~TriggerMenu
* !70083 Dev new histograms ~DQ, ~LAr, ~Trigger
* !70105 gFEX update jwoj threshold ~Trigger, ~TriggerMenu, ~changes-trigger-counts
* !70121 TileMonitoring: Remove legacy JO configurations (ATLASRECTS-7939) ~DQ, ~Tile
* !70126 TileMuId: Remove legacy JO configurations (ATLASRECTS-7939) ~Tile
* !70128 TileRecAlgs: Remove legacy JO configurations (ATLASRECTS-7939) ~Tile
* !69231 Implementing the MuonSelectionTool for selecting muon tracks. ~Analysis, ~BTagging, ~Derivation, ~JetEtmiss
* !70125 remove obsolete job-config files from CaloRec/share ~Calorimeter
* !70124 TileMonitoring: Remove legacy config from TileMBTSMonitorAlgorithm (ATLASRECTS-7939) ~DQ, ~Tile
* !70122 TileMonitoring: Remove legacy config from TileDQFragMonitorAlgorithm (ATLASRECTS-7939) ~DQ, ~Tile
There was one conflict (in `BTaggingValidationPlots.cxx`). As discussed in !70085, I resolved this by taking the `24.0` changes.https://gitlab.cern.ch/atlas/athena/-/merge_requests/70156Draft: write in-file metadata using RootKeyContainer2024-03-27T17:50:55+01:00Maciej Pawel SzymanskiDraft: write in-file metadata using RootKeyContainerAn experiment to check what's needed to switch metadata storage technologyAn experiment to check what's needed to switch metadata storage technologyhttps://gitlab.cern.ch/atlas/athena/-/merge_requests/69793AthContainers, etc: Move Accessor, etc. classes out of AuxElement.2024-03-28T17:43:29+01:00Scott SnyderAthContainers, etc: Move Accessor, etc. classes out of AuxElement.Accessor and related classes have been members of AuxElement, the original
thinking being that derived classes of AuxElement may want to redefine them.
However, in retrospect, there does not seem to be any circumstance in which
that woul...Accessor and related classes have been members of AuxElement, the original
thinking being that derived classes of AuxElement may want to redefine them.
However, in retrospect, there does not seem to be any circumstance in which
that would actually make sense. Further, it prevents having any specializations
for these classes. So move these classes out of AuxElement into the SG
namespace (but with the names brought into AuxElement via using declarations
for backwards compatibility).https://gitlab.cern.ch/atlas/athena/-/merge_requests/69664Draft: athena.py: implement native support for CA2024-03-28T10:37:33+01:00Frank WinklmeierDraft: athena.py: implement native support for CAThis MR is a refactor of the functionality of `ThinCAWrapper.sh` and `CARunner.py` into the main `athena` executable. After the initial shell-phase, the process remains in python. This is achieved using the `runpy` module to directly lau...This MR is a refactor of the functionality of `ThinCAWrapper.sh` and `CARunner.py` into the main `athena` executable. After the initial shell-phase, the process remains in python. This is achieved using the `runpy` module to directly launch the CA script via the `__main__` entry point.
This MR is the first step in making the `--CA` option obsolete.
#### User visible changes:
* The interactive mode now requires an optional \<stage\> argument also for legacy athena. If this is directly followed by the job options, the arguments need to be terminated by `--`. E.g all of the following are equivalent:
```plaintext
> athena AthExHelloWorld/HelloWorldOptions.py -i
> athena -i -- AthExHelloWorld/HelloWorldOptions.py
> athena -i init AthExHelloWorld/HelloWorldOptions.py
```
* If the user tries to run a CA script in legacy mode, the following will be printed:
```plaintext
AthenaCommon.Include.IncludeError: include file AthExHelloWorld/HelloWorldConfig.py can not be found
If the script is a ComponentAccumulator configuration, add a shebang (#!) to its first line or run with the --CA option.
```
#### Implementation details:
* Use the `AthOptionsParser` for both legacy athena and CA-based job/athena. If used within `athena.py` the athena-specific command line options are added.
* The shell-based CLI-parsing is now restricted to what is necessary to setup the preloading.
* Implement the file path resolution in Python using `unixtools.FindFile`. The "shebang" CA detection now also works for file paths that are resolved at runtime.
* The `CARunner.py` use-case is replaced by directly loading the pkl file (legacy/CA pickles are automatically distinguished). I dropped the support of CA pkls without MainServicesCfg. Could be added again if really needed.
* Migrate the `--tracelevel` feature into `AthenaCommon.Debugging`. We should probably unify this with the existing `--trace` option in a second iteration.
For the moment, we still support the `--CA` flag. But if the proper shebang is used, the following commands all work transparently without the need to use `--CA`:
```sh
athena.py AthExHelloWorld/HelloWorldOptions.py
athena.py --config-only legacy.pkl AthExHelloWorld/HelloWorldOptions.py
athena.py legacy.pkl
athena.py AthExHelloWorld/HelloWorldConfig.py
athena.py --config-only cfg.pkl AthExHelloWorld/HelloWorldConfig.py
athena.py cfg.pkl
```
cc @wlampl @tadej @ssnyder
TODO:
* [ ] Delete `CARunner.py` after migrating the few uses to `athena.py`
* [ ] Backport to ~"24.0" once validated in ~mainhttps://gitlab.cern.ch/atlas/athena/-/merge_requests/69513Use config flags in analysis config2024-03-28T13:34:35+01:00Tadej Novaktadej.novak@cern.chUse config flags in analysis configMake config flags the main way to steer analysis config. This will help people working on analysis and Athena to feel at home and prevent accumulation of arguments of the `ConfigAccumulator`.
The naming is not fixed, I will probably als...Make config flags the main way to steer analysis config. This will help people working on analysis and Athena to feel at home and prevent accumulation of arguments of the `ConfigAccumulator`.
The naming is not fixed, I will probably also convert more items to flags. This is mainly to start the discussion and freeze the API as soon asp possible. I will probably then break this into multiple MRs.
Also some core flags should be renamed (see https://its.cern.ch/jira/browse/ATEAM-964).
Tagging @krumnack, @jolamber, @gwatts, @ekourlit for core AMG. Also tagging interested parties @ravinab, @omajersk, @tstreble, @khoo, @jchapman.https://gitlab.cern.ch/atlas/athena/-/merge_requests/69417Draft: Remove Legacy to CA conversion utilities2024-03-08T12:37:40+01:00Tomasz BoldDraft: Remove Legacy to CA conversion utilitiesThis MR is a test to see how far we are from fully relying on CA job options.This MR is a test to see how far we are from fully relying on CA job options.https://gitlab.cern.ch/atlas/athena/-/merge_requests/69382Adding new AthenaExamples package - AthExBasics2024-03-29T00:57:09+01:00James CatmoreAdding new AthenaExamples package - AthExBasicsThis MR adds a new package to AthenaExamples. It is intended to contain examples of tools, algorithms and config scripts for doing very basic real-world tasks such as reading the trigger decision, accessing and selecting xAOD objects, ru...This MR adds a new package to AthenaExamples. It is intended to contain examples of tools, algorithms and config scripts for doing very basic real-world tasks such as reading the trigger decision, accessing and selecting xAOD objects, running trigger matching &c. It is hoped developers will add to these over time.
Obviously this will have no impact on any existing workflows.
This is part of the lead-up to the documentation week in April.https://gitlab.cern.ch/atlas/athena/-/merge_requests/69308Draft: Add AthXRT service and AthExXRT example (ATLINFR-5117)2024-03-08T10:57:10+01:00Quentin BerthetDraft: Add AthXRT service and AthExXRT example (ATLINFR-5117)This MR add the AthXRT service and AthExXRT example to Athena.
This service is a prototype to interact with AMD FPGA accelerators from Athena. It is quite basic and can only be used to enumerate and program the FPGA with a configuration...This MR add the AthXRT service and AthExXRT example to Athena.
This service is a prototype to interact with AMD FPGA accelerators from Athena. It is quite basic and can only be used to enumerate and program the FPGA with a configuration file (bitstream) at the moment. It is used for the exploration of EFCalo topoclustering FPGA acceleration, and might be useful for other Run4 FPGA acceleration prototypes (EFTracking have also used this code). The structure is heavily inspired from AthCUDA and AthExCUDA and is intended to be the common entry point for AMD FPGA accelerators in Athena.
The AthExXRT example illustrates how to use the AthXRT service to run a dummy vector addition kernel from Athena. An HLS code and build procedure is also provided to produce the required bitstream (Note that some public Apache 2.0 licensed code from AMD/Xilinx is used there).
This code requires the XRT library and it is currently not provided in any Atlas standard environments (See [this](https://its.cern.ch/jira/browse/ATLINFR-5117) ticket for the discussion on XRT inclusion in AtlasExternals). Because of this, these new packages are not compiled if XRT library is not available.
Note to reviewer: this is my first MR, and the first code that I ever written for Athena, so I expect modifications to be required before merging.Quentin BerthetQuentin Berthethttps://gitlab.cern.ch/atlas/athena/-/merge_requests/68960Draft: Use std::format instead of boost::format2024-02-27T09:58:01+01:00Andrii VerbytskyiDraft: Use std::format instead of boost::formatUse std::format instead of boost::formatUse std::format instead of boost::formathttps://gitlab.cern.ch/atlas/athena/-/merge_requests/67630Draft: Use PropertyProxy from Gaudi2024-02-19T11:20:40+01:00Frank WinklmeierDraft: Use PropertyProxy from GaudiExperimental MR to migrate to Gaudi's `PropertyProxy` (and `ConfigurableDb`). In principle there are only very few differences and this would avoid having to maintain our own copy each time a new property type gets added.
Ideally, we wo...Experimental MR to migrate to Gaudi's `PropertyProxy` (and `ConfigurableDb`). In principle there are only very few differences and this would avoid having to maintain our own copy each time a new property type gets added.
Ideally, we would do the same for the (legacy) `Configurable` class but that requires more cleanup first (i.e. removing the support of the CA behavior once we no longer need it).https://gitlab.cern.ch/atlas/athena/-/merge_requests/67388Try to use C++20 starts_with2024-03-21T13:16:05+01:00Andrii VerbytskyiTry to use C++20 starts_withTry to use C++20 starts_withTry to use C++20 starts_withhttps://gitlab.cern.ch/atlas/athena/-/merge_requests/66502Draft: MR for triggering jobs for CI training and tests on EL9, please never ...2024-01-02T19:56:34+01:00Alexander UndrusDraft: MR for triggering jobs for CI training and tests on EL9, please never acceptThis MR is created to trigger test jobs in the CI system. Please do not accept it.This MR is created to trigger test jobs in the CI system. Please do not accept it.Alexander UndrusAlexander Undrushttps://gitlab.cern.ch/atlas/athena/-/merge_requests/61052Allow overriding of compression algorithm and level2024-02-19T10:51:49+01:00Tadej Novaktadej.novak@cern.chAllow overriding of compression algorithm and levelAllow overriding of compression algorithm and level similar to what was implemented for the auto flush.
Fixes ATLASRECTS-7504.
/cc @gemmeren @ameteAllow overriding of compression algorithm and level similar to what was implemented for the auto flush.
Fixes ATLASRECTS-7504.
/cc @gemmeren @ametehttps://gitlab.cern.ch/atlas/athena/-/merge_requests/60463Draft: MR for triggering jobs for CI training and tests, please never accept2024-03-16T04:57:48+01:00Alexander UndrusDraft: MR for triggering jobs for CI training and tests, please never acceptThis MR is created to trigger test jobs in the CI system. Please do not accept it.This MR is created to trigger test jobs in the CI system. Please do not accept it.Alexander UndrusAlexander Undrus