athena merge requestshttps://gitlab.cern.ch/atlas/athena/-/merge_requests2024-03-04T15:51:43+01:00https://gitlab.cern.ch/atlas/athena/-/merge_requests/59680Draft: Implement asynchronous I/O in HltEventLoopMgr2024-03-04T15:51:43+01:00Rafal Bielskirafal.bielski@cern.chDraft: Implement asynchronous I/O in HltEventLoopMgrChange the implementation of the main event loop in HLT from synchronous jumping between input handling and output handling on one thread to independent asynchronous I/O threads.
The threading / synchronisation boilerplate code is put i...Change the implementation of the main event loop in HLT from synchronous jumping between input handling and output handling on one thread to independent asynchronous I/O threads.
The threading / synchronisation boilerplate code is put into a separate header file and is now shared by three threads in the HltEventLoopMgr: the timeout monitoring thread (almost no change to its implementation) and the new input and output handling threads. The main thread of the application only starts the three loop threads and sleeps until the end of the event loop.
Like before this MR, the input and output parts of the HltEventLoopMgr (previously on the main thread, now in separate threads) still don't do much work. They only define, prepare and schedule a TBB Task which is then executed by TBB worker threads. This keeps the CPU cost of the HltEventLoopMgr threads at near zero, and all the work is done by the pool of TBB worker threads, which is more natural and predictable in performance profiling. The only functional change in this MR is that the TBB tasks for I/O handling are now launched asynchronously and don't block each other. Previously, we could not fill a free Gaudi Scheduler slot while waiting for a finished event, and similarly could not pop a finished event from the Gaudi Scheduler while waiting for a new event from input source.
The new implementation required a somewhat significant change in error handling. Since the input and output handling are now asynchronous, the concept of "draining all Scheduler slots" mid-run and then continuing to fill them again isn't so trivially possible. Implementing such behaviour would require the output thread to "pause" the input thread and then after clearing all slots, "resume" it. With the experience of 2022 P1 operations I can say the drain-all-slots procedure wasn't really needed in the first place. If things go wrong on the framework side, it is fine to exit the event loop with a failure and restart the entire process from scratch. This will now happen in a wider range of errors (most of which have never been encountered in practice) which previously attempted the "drain all slots and continue" approach.
Jira: ATR-26285
FYI @fwinkl, @wiedenma, @mark
Simplified call diagrams (skipping the start/end of loop conditions):
**before this MR:**
<img src="/uploads/dda223b50afc2a91520e5f53de7f3ecf/HltEventLoopMgr-sync-io.png" height="150" />
**after this MR:**
<img src="/uploads/7222b092b2cd738028bb798dfa6c0320/HltEventLoopMgr-async-io.png" height="150" />https://gitlab.cern.ch/atlas/athena/-/merge_requests/64191Adding flags to enable/disable conversion of space point containers (InDet to...2023-07-12T14:23:08+02:00Noemi CalaceAdding flags to enable/disable conversion of space point containers (InDet to xAOD)Self explanatory title.
FYI: @cvarniSelf explanatory title.
FYI: @cvarniNoemi CalaceNoemi Calacehttps://gitlab.cern.ch/atlas/athena/-/merge_requests/63358Speedup computation of Pixel distortion correction2023-07-05T18:21:12+02:00Goetz GayckenSpeedup computation of Pixel distortion correctionIn a data22 athena job, std::pow shows up as the the 6th most used function in vtune (50 s accumulated of 1751 s total Algorithm::sysExecute). This function is primarily called by PixelDistortionData::bernstein_grundpolynom which compute...In a data22 athena job, std::pow shows up as the the 6th most used function in vtune (50 s accumulated of 1751 s total Algorithm::sysExecute). This function is primarily called by PixelDistortionData::bernstein_grundpolynom which computes
sum_i=0^n sum_j=0^n P_{n;i}(u) P_{n;j}(v) a_{ij}
with P_{n;i}(u) = (n over i) * u^i * (1-u)^{n-i}, and n=20. In this nested sum std::pow is called for each evaluation 2*20^2 times.
To speed up the computation, the commits implement the following optimisations:
1) compute u^{i}, (1-u)^i, v^{i} and (1-v)^{i} outside the nested sum for i=0..20, use the pre-computed factors inside the nested sum. This replaces 400 calls of std::pow by array accesses and 4*20 products.
2) multiply the parameters a_{ij} when creating the conditions data with the binomial coefficients (n_over_i) * (n_over_j), which removes two multiplications from the nested sum.
3) pass the distortion correction parameter vectors by reference rather than by value which avoid allocation and memory copies.
In synthetic benchmarks this shows a speedup by more than a factor 50. The athena data22 jobs show in average an increase in the number of events processed per second by more than ~1% (an improvement from 1.11 to 1.12 events/s), which is below the naive expectation of ~3% (taking the vtune results as truth: 1751s -> 1706s).https://gitlab.cern.ch/atlas/athena/-/merge_requests/58907Draft: Add the Voxel Density Optimization to the postInclude.G4Optimizations ...2023-07-05T10:04:27+02:00Marilena BandieramonteDraft: Add the Voxel Density Optimization to the postInclude.G4Optimizations script for Run3https://gitlab.cern.ch/atlas/athena/-/merge_requests/62164Draft: Proposing faster eta calculation for `Amg::Vector3D`2023-07-05T10:03:36+02:00Nuno Dos Santos FernandesDraft: Proposing faster eta calculation for `Amg::Vector3D`During the changes included for the Calo GPU Clustering in !62160, an expression to calculate η from the Cartesian coordinates of a vector was deduced.
Mathematically, this will be equivalent to the current implementation, which calls `t...During the changes included for the Calo GPU Clustering in !62160, an expression to calculate η from the Cartesian coordinates of a vector was deduced.
Mathematically, this will be equivalent to the current implementation, which calls `tan(atan2(sqrt(x*x + y*y), z)/2)`, without evaluating the rather expensive trigonometric functions, with the only expected differences coming from floating point accuracy.
The mathematical reasoning is explained in the comment added to the relevant section of the code, but the gist of it is that we can leverage the half-tangent formula and the geometrical interpretation of the trigonometric functions to express the quantity we want as just an algebraic function of x, y and z.
Though no specific performance measurement has been carried out yet, it is expected that this change will result in some gains given that we are side-stepping the evaluation of two rather expensive functions.
The purpose of the present merge request is, above all, to gauge receptiveness to this change, given its potentially far-reaching consequences.
cc @wlampl @ssnyder @mhodgkin @christoshttps://gitlab.cern.ch/atlas/athena/-/merge_requests/56535Draft: HGTD holes on track tool2023-07-05T10:02:15+02:00Valentina RaskinaDraft: HGTD holes on track toolThe tool is added for the holes on track search between the last hit in ITk and the first layer of HGTD. @aleopoldThe tool is added for the holes on track search between the last hit in ITk and the first layer of HGTD. @aleopoldValentina RaskinaValentina Raskinahttps://gitlab.cern.ch/atlas/athena/-/merge_requests/64083Config delta X and Y cuts2023-07-04T10:54:14+02:00Luisa CarvalhoConfig delta X and Y cutsConfigure cuts in delta X and Y for AFP+dijet algorithm. Job options change only. Y and absolute distance cuts very wide since we know that with current parameterization and lack of global alignment Y matching probably won't work.Configure cuts in delta X and Y for AFP+dijet algorithm. Job options change only. Y and absolute distance cuts very wide since we know that with current parameterization and lack of global alignment Y matching probably won't work.https://gitlab.cern.ch/atlas/athena/-/merge_requests/64065Set FRONTIER as default database server instead of ORACLE in all TileCal pyth...2023-06-30T23:44:35+02:00Sanya SolodkovSet FRONTIER as default database server instead of ORACLE in all TileCal python scriptsUpdating all python scripts which works with COOL DB - set FRONTIER as default database serverUpdating all python scripts which works with COOL DB - set FRONTIER as default database serverhttps://gitlab.cern.ch/atlas/athena/-/merge_requests/63983Draft: Test vertex cache reset2023-06-29T14:19:33+02:00Carlo Varnicarlo.varni@cern.chDraft: Test vertex cache resetjust a test to check the impactjust a test to check the impacthttps://gitlab.cern.ch/atlas/athena/-/merge_requests/63927Update NSWPRDValAlg output filenames2023-06-26T14:48:32+02:00Marten Zefanja BarelUpdate NSWPRDValAlg output filenamesSmall update the the names of the output files generated by the NSWPRDValAlg python scripts to match the name of the script generating the fileSmall update the the names of the output files generated by the NSWPRDValAlg python scripts to match the name of the script generating the filehttps://gitlab.cern.ch/atlas/athena/-/merge_requests/52154Draft: Combine Herwig LHEF config files for versions 7.1.x and 7.2.x into one.2023-06-20T14:23:10+02:00Yoran YehDraft: Combine Herwig LHEF config files for versions 7.1.x and 7.2.x into one.This MR includes some cleaning up of the common Herwig7_i repository on the master branch (rel22).
Two separate LHEF configuration files (for Herwig versions 7.1.x and 7.2.x respectively) are merged into a single file that fetches the He...This MR includes some cleaning up of the common Herwig7_i repository on the master branch (rel22).
Two separate LHEF configuration files (for Herwig versions 7.1.x and 7.2.x respectively) are merged into a single file that fetches the Herwig version from the `$HERWIG7VER` environment variable.
The EvtGen file is renamed to avoid confusion for which Herwig version it is suitable. (It is suitable for both 7.1.x and 7.2.x, but the previous name suggested otherwise.)
Tagging @dhirschhttps://gitlab.cern.ch/atlas/athena/-/merge_requests/6382423.0-coverity-FPGATrackSimMaps2023-06-20T13:13:48+02:00Shaun Roe23.0-coverity-FPGATrackSimMapsFix various coverity defects: uninitialised member variables, use of uninitialised variable, memory leak. One identified bug (arguments reversed) will have JIRA ticket opened, not part of this MR.Fix various coverity defects: uninitialised member variables, use of uninitialised variable, memory leak. One identified bug (arguments reversed) will have JIRA ticket opened, not part of this MR.https://gitlab.cern.ch/atlas/athena/-/merge_requests/63380Draft:Fix for ATR-27420 causing error in GenerateMenuMT stage of trig_mc_newJ...2023-06-20T09:54:00+02:00John BainesDraft:Fix for ATR-27420 causing error in GenerateMenuMT stage of trig_mc_newJO_ITk_buildFix for ATR-27420 causing error in GenerateMenuMT stage of trig_mc_newJO_ITk_build
This MR adds a switch between ITkTracking and InDetTracking flags in generateMuon.py, generateTau.py and TrigInDetConfig/utils.py
e.g.
muonflagsCBid = mu...Fix for ATR-27420 causing error in GenerateMenuMT stage of trig_mc_newJO_ITk_build
This MR adds a switch between ITkTracking and InDetTracking flags in generateMuon.py, generateTau.py and TrigInDetConfig/utils.py
e.g.
muonflagsCBid = muonflagsCB.cloneAndReplace("Tracking.ActiveConfig", "Trigger.ITkTracking.muon" if flags.Detector.GeometryITk else "Trigger.InDetTracking.muon")
it also removes the cloneAndReplace from TrigInDetConfig/TrigIndetConfigITk.py as this is no wperformed higher up.
tagging @jmasik @ewattonhttps://gitlab.cern.ch/atlas/athena/-/merge_requests/63778Improvements and fixes to PhysLite, primarily as seen from the H4l analysis...2023-06-18T18:59:36+02:00R D SchafferImprovements and fixes to PhysLite, primarily as seen from the H4l analysis...Improvements and fixes to PhysLite (and Phys), primarily as seen from the H4l analysis perspective.
The changes include:
- Implementation of the isolationCloseBy tool in the running of Phys and PhysLite. This allows closeBy to be availa...Improvements and fixes to PhysLite (and Phys), primarily as seen from the H4l analysis perspective.
The changes include:
- Implementation of the isolationCloseBy tool in the running of Phys and PhysLite. This allows closeBy to be available for all analyses, and avoids extra information to be added to either so that one could do it when reading Phys/PhysLite and included LLP1. Corrected iso variables are <iso_value>_OR. For the moment, the 'extra' closeBy information is left in Phys and LLP1 to allow cross check. But this can be removed, eventually. - [ATLASG-2520](https://its.cern.ch/jira/browse/ATLASG-2520)
- Added in config to PhysLite for H4l vertex calculation [ATLASG-2518](https://its.cern.ch/jira/browse/ATLASG-2518)
- Added in STXS information to PhysLite for Higgs analyses, fixing units [ATLASG-2515](https://its.cern.ch/jira/browse/ATLASG-2515)
- Reducing the electron id requirement from LH Loose to SiHits for PhysLite [ATLASG-2537](https://its.cern.ch/jira/browse/ATLASG-2537)R D SchafferR D Schafferhttps://gitlab.cern.ch/atlas/athena/-/merge_requests/63718modify tau smear tool according to R22 rec2023-06-15T12:30:39+02:00Mingming Xiamodify tau smear tool according to R22 recadd "Campaign" property for tau smear tool for AnalysisTopadd "Campaign" property for tau smear tool for AnalysisTophttps://gitlab.cern.ch/atlas/athena/-/merge_requests/60401Annealing for robust fit2023-06-14T23:51:26+02:00Vadim KostyukhinAnnealing for robust fitUse small annealing in vertex fits with robust option depending on the fit iteration number.
This should speed up convergence and help to avoid fake minima.Use small annealing in vertex fits with robust option depending on the fit iteration number.
This should speed up convergence and help to avoid fake minima.https://gitlab.cern.ch/atlas/athena/-/merge_requests/63688Draft: Test long long2023-06-14T17:44:11+02:00Carlo Varnicarlo.varni@cern.chDraft: Test long longhttps://gitlab.cern.ch/atlas/athena/-/merge_requests/48284Draft: e-tag dataset and parser tool2023-06-13T17:32:55+02:00Miha Muskinjamiha.muskinja@cern.chDraft: e-tag dataset and parser toolFollowing the discussion in ATEAM-787 and core-sw meeting, I created a .csv dataset and a parsing tool for e-tags. The dataset is created from the PMG TWiki page: https://twiki.cern.ch/twiki/bin/view/AtlasProtected/PmgMcSoftware.
The to...Following the discussion in ATEAM-787 and core-sw meeting, I created a .csv dataset and a parsing tool for e-tags. The dataset is created from the PMG TWiki page: https://twiki.cern.ch/twiki/bin/view/AtlasProtected/PmgMcSoftware.
The tool return a `map<string, string>` object given an e-tag string with the information from the TWiki page. This can be coupled with the info from the in-file metadata to find out the specific version of the MC generator software (e.g. Sherpa version). The tool has both a python and a C++ interface.
Currently the tool has no clients, but in the future it can be used to update the in-file metadata e.g. at the start of derivation jobs. Example usage via CLI:
```
> e-tag-parser.py e8276
EvtGen : 1.7.0
HepMC : 2.6.11
Herwig7 : 7.2.1
Lhapdf : 6.2.3
MadGraph : 2.8.1.atlas7
Powheg ATLAS : 00-04-05
Pythia8 : 8.244
Release : 21.6.53
Rivet : 3.1.2
Sherpa : 2.2.10p5
Starlight : r313
Superchic : 3.0.6
e-tag : e8276
```
Adding @tadej @cgutscho @mgignac and please add anyone else who might be interested.https://gitlab.cern.ch/atlas/athena/-/merge_requests/63504SUSYTools - Fix art AthAnalysis2023-06-06T16:40:53+02:00Marco RimoldiSUSYTools - Fix art AthAnalysisMarco RimoldiMarco Rimoldihttps://gitlab.cern.ch/atlas/athena/-/merge_requests/60737Draft: An interesting test with ParticleLink2023-06-06T16:40:20+02:00Andrii VerbytskyiDraft: An interesting test with ParticleLinkAn interesting test with ParticleLink
Will that work better? Will that work at all?
@tadej @jchapman @christosAn interesting test with ParticleLink
Will that work better? Will that work at all?
@tadej @jchapman @christos