Draft: ComponentAccumulator: implement async merges
- Oct 27, 2023
-
-
Frank Winklmeier authored
-
Frank Winklmeier authored
Implement asynchronous merges using the `multiprocess` module.
-
Proof-of-concept (not proposing to merge this in its current form) to implement asynchronous merging in ComponentAccumulator
, e.g.:
acc = ComponentAccumulator()
acc.async_merge(MyCfg, flags)
acc.async_merge(MyOtherCfg, flags)
...
acc.async_wait()
Changed RecoSteering
to apply this for the various reco domains. Results in speedup from 2:00 (91% CPU) minutes to 1:35 (136%) with a process Pool of size 4. This means there is unfortunately not enough parallelism to even fully utilize two cores.
Notes:
multiprocess
module (available in LCG) which uses dill
for object serialization (the default pickle
cannot handle the lamba
s in AthConfigFlags
)AccumulatorCache
and cache statics are not reported back to the main processTo reproduce one needs to setup the full LCG release:
export LCG_RELEASE_BASE=/cvmfs/sft.cern.ch/lcg/releases
asetup --noLcgReleaseBase Athena,main,latest
Implement asynchronous merges using the `multiprocess` module.