Baptiste Ravina authored 1 year ago and Walter Lampl committed 1 year ago

CPAlgorithms: distinguish between MC and data when writing output branches

Oliver Majersky authored 1 year ago and Vakhtang Tsulaia committed 1 year ago

CP algorithms: rename the jvt selection name to something more explanatory

Essentially this ensures that users don't misspell their selection
names, or use a selection name they forgot to define.

This is only for full selections, for preselections I don't do
it, as I usually ask for them before they are defined.  I also
don't do it for the baseline selection.  If that is not present
that just happens.

Oliver Majersky authored 1 year ago and Tadej Novak committed 1 year ago

Revert "Remove FTAG selectionNmaes that are actually never used."

This reverts commit 436ed695.

This is a followup to !68703 that uses that mechanism to add factories for
all components used by the common CP algorithms.  This is not necessarily
the prettiest way of doing things, but it is what can be practically done
right now with little effort.  It should probably be replaced by a better
mechanism at some point.

Command to test/run this:
```
FullCPAlgorithmsTest_eljob.py --data-type data --algorithm-memory --factory-preload 'libComponentFactoryPreloaderDict.so,CP::preloadComponentFactories' --block-config --use-flags
```

The basic idea here is that dictionaries are fairly heavy when used as a factory.  Registering factory functions instead makes the jobs a lot lighter.
For the CP algorithm test job (on data) the savings are:

~600 MB in memory
~22s in startup time
one major root dependency eliminated

The first draft also included a function for preloading all the dictionaries.  I removed
it to allow the user to define their own.

Baptiste Ravina authored 1 year ago and Frank Winklmeier committed 1 year ago

CPAlgorithms: bugfix for PerEventSFBlock

James Catmore authored 1 year ago and Vakhtang Tsulaia committed 1 year ago

Removal of legacy config for DAOD production - part 4

This MR removes much of the legacy DAOD-making configuration code, such that the old-style reductionConf argument to Reco_tf will no longer work.

Some of the packages touched here may be completely obsolete and so more deletion may be possible, but for this MR the changes are restricted to DAOD building.

Thomas Strebler authored 1 year ago and Tadej Novak committed 1 year ago

Add FTag event scale factor CP algorithm config

Thomas Strebler authored 1 year ago and Tadej Novak committed 1 year ago

Remove deprecated runSelection and globalSF from JetJvtAnalysisConfig

Baptiste Ravina authored 1 year ago and Tadej Novak committed 1 year ago

CPAlgorithms: refactor the object cutflow

Scott Snyder authored 1 year ago and Tadej Novak committed 1 year ago

AthContainers: Update copy/copyForOutput to work on AuxVectorData.
    
Update the copy/copyForOutput methods of IAuxTypeVectorFactory / AuxTypeRegistry
to take AuxVectorData instances rather than bare pointers.
Remove the corresponding static methods from AuxTypeVector.

Refactoring to make possible dependencies between auxiliary variables,
as would be used to support packed links / jagged vectors.
Also alter them to operate on ranges, rather than a single element.

Revert "Merge branch 'bugfix_objectcutflow' into 'main'"

This reverts merge request !68554

Nils Erik Krumnack authored 1 year ago and Tadej Novak committed 1 year ago

introduce a simple per-algorithm memory monitor for EventLoop

This will **not** be able to detect memory leaks, but instead it will
just monitor how much memory an algorithm allocates during setup.
This allows to see if any of the algorithms has a particularly large
calibration table or similar.

It also contains occasional memory usage printouts during the job.  That
allows to see whether memory gets consumed in the job setup, or during
execute

This is mostly meant to allow debugging some current problems with the
Top jobs which quickly shoot up to 3.2GB memory consumption.

Baptiste Ravina authored 1 year ago and Tadej Novak committed 1 year ago

CPAlgorithms: refactor the object cutflow

The main usage here is that if OR is run before MET we don't want to use the OR selections with MET.  A similar issue is expected to come up with fakes/fakeable objects, i.e. some specific baseline cuts should not be applied (in fact they are inverted), but only for very specific users.

The mechanism chosen here is to mark the cuts when they are registered and then exclude by name when requesting the selection.

Joseph Earl Lambert authored 1 year ago and Tadej Novak committed 1 year ago

TextConfig: add block to add blocks from YAML file

Inorder to allow for users to add blocks from the YAML file without
editting the python code, a new block was added.

Baptiste Ravina authored 1 year ago and Tadej Novak committed 1 year ago

CPAlgorithms: expose "excludeNSWFromPrecisionLayers" to the muon config

Baptiste Ravina authored 1 year ago and Adam Edward Barton committed 1 year ago

CPAlgorithms: smarter systematic printouts

Thomas Strebler authored 1 year ago and Adam Edward Barton committed 1 year ago

Fix forceFullSimConfig default config for PhotonWorkingPointConfig

Filip Nechansky authored 1 year ago and Adam Edward Barton committed 1 year ago

TrigPrescalesAlg: Add option for prescale based on trigger formula

The trigger option in the prw tool can be a formula (trigA||trigB):
https://gitlab.cern.ch/ATLAS-EGamma/athena/-/blob/main/PhysicsAnalysis/AnalysisCommon/PileupReweighting/Root/PileupReweightingTool.cxx#L484
but current implementation of the alg is not intended for this use
and  would result in decoration:
`m_prescaleDecoration+triger_formula` which can lead to problems
because of `||`, spaces and such (not sure decorators allow it?).

This MR would change it that instead of passing list of triggers
one can pass them as `trigFormula` and the decoration would be only
`m_prescaleDecoration`, so it would be much more straighforward to use.

Taking advantage of existing implementation so the execute
part does not change, only the init.

Oliver Majersky authored 1 year ago and Johannes Elmsheuser committed 1 year ago

Rework interface for GRL and PRW block configuration in CP algos

Matthew Kenneth Maroun authored 1 year ago and Walter Lampl committed 1 year ago

Add Multiple File Force-Input Capability to FullCPAlgorithmsTest_X.py

Nils Erik Krumnack authored 1 year ago and Walter Lampl committed 1 year ago

introduce ExecDriver (plus other updates)

As the main feature this introduces ExecDriver.  The idea behind
ExecDriver is that it runs the actual job in a new executable
(replacing the current one).  This is meant to release all the memory
that gets allocated by our configuration helpers.  That is over 1GB
(mostly ROOT-python dictionaries), so this is a significant savings.

There are a couple more updates that work with this:
* on merging, if there is only one file I create a hard link instead of
  merging, which is a lot more efficient.
* BatchDriver now respects maxEvents (in a fashion), which makes it easier
  to do tests with LocalDriver and ExecDriver.

Waltteri Leinonen authored 1 year ago and Walter Lampl committed 1 year ago

Harmonize trigger naming within TriggerAnalysisConfig to TrigEventSelectorAlg, replace ',' with 'p' and '-' with '_'

Thomas Strebler authored 1 year ago and Walter Lampl committed 1 year ago

Added forceFullSimConfig for EGamma analysis configs for testing (ATLASG-2661)

Baptiste Ravina authored 1 year ago and Walter Lampl committed 1 year ago

CPAlgorithms: add event selectors for large-R jets

Thomas Strebler authored 1 year ago and Vakhtang Tsulaia committed 1 year ago

Updated fast sim property for ElectronPhotonFourMomentumCorrection