In this MR the following changes are made to DerivationFrameworkART
 * EXOT7 and SUSY9 are added to their trains
 * Tests are included for BPHY7,8 and 10 on the BLS and delayedBPhys streams
 * All events are processed for a given input file, unless there is a specific need to process fewer


Former-commit-id: c43737bbb6dae1e08583743f76fdf0b89011b60e

EXOT23, which runs on special RPVLL AODs, is introduced to ART in this MR. To do: the MC file is an unofficial privately produced file. We should replace this with a centrally produced one, when one is available.


Former-commit-id: 65884afab14200101690405d9ff9658e8683d69e

EXOT22 is a new format and in this MR is added to the appropriate train test.


Former-commit-id: b13888a0980002e86fe30685a1f963c417660956

In this MR the following changes are made to the DAOD building ART tests:
 * All available events now processed in the merge test
 * MC test file switched to ttbar (410501, r9778)
 * New train test added for TOPQ, running on MC only


Former-commit-id: 8337fe0f57ef180181e9f59c83e1a3b0de61863d

This MR makes the following adjustments:
 * drops EXOT14 completely
 * adds SUSY9 and adds an ART test for data and MC
 * Adds TOPQ5 to the ART train tests for data and MC
 * Adds BPHY8 to the ART train test for data
 * Reduces BPHY3 to 500 events in the ART tests
 * Amends the test input file used for the PHYSVAL data ART test

Where required the script-maker has been amended to reflect the above as well


Former-commit-id: 45309cc21d645eb9f2b801ff75b531a3a081037a

The same trains are set for MC as for data.


Former-commit-id: a9ef32acb13fb1d75d878c47f90abb253af85b01

This MR adds a line in all DAOD scripts to collect the output of the checkFile test to the tarball. The master script-generator is adjusted to reflect this.


Former-commit-id: 39457bbc9b804475b5cf39a5058e6157f2e7afd9

Former-commit-id: 76fe626bf4fdad5c1d62d22b5386b0fe8cd2a157

The input AOD file for the test_MC16BPHY8 ART test is adjusted to
a file from
mc16_13TeV.300999.Pythia8BEvtGen_A14_CTEQ6L1_Bplus_Jpsi_mu3p5mu3p5_Kplus.recon.AOD.e4397_s3126_r9703
This input file -- although it only contains 2000 events -- is currently the
best available file from a MC16 channel which BPHY8 is actually set up to
process while on the other side it was done with a reconstruction release
recent enough for the btagging xAODFix being run in 21.2 derivations.

The input file AOD.11705353._000001.pool.root.1 has been copied to
/eos/atlas/atlascerngroupdisk/data-art/grid-input/DerivationFrameworkART
so it will be picked up for inclusion in CVMFS.

It has been checked locally that the BPHY8 format works correctly with recent
21.2-AthDerivation and this new input file.

Please note that the file name has also been updated in makeARTScripts.py
to ensure that the adjustment will still be applied after this script is
run.


Former-commit-id: 8f0ba7475709a70220bea43059debe09f1882c57

Fabian Wilk authored 7 years ago and James Catmore committed 7 years ago

Former-commit-id: 93a7f90b486eb5bdcdfda82cd1aab374530d92da

This MR includes a number of additions to the DAOD ART tests
 * increased number of events in the test from 5000 to 8000 (data) and 10000 (MC)
 * adds checkFile.py report
 * adds a new format (HIGG4D6)
 * corrects the NTUP_PHYSVAL command in the PHYSVAL tests


Former-commit-id: c524b403b21f399e7c4aa0ba66d15cff6c32514e

ATLASDPD-976

The following containers were missing from DAOD_PHYSVAL:
 * PFlow: JetETMissNeutralParticleFlowObjects, JetETMissChargedParticleFlowObjects
 * Muons: MuonTruthParticles
 * Egamma: egammaTruthParticles
 * Taus: TauTracks

This MR adds them to the job options. The ART tests are also adjusted to include a test production of NTUP_PHYSVAL from the DAOD_PHYSVAL made in the same test.


Former-commit-id: b8295e093e75e753e4565cb1e9d449e42871e9c0

BPHY8 requires specific MC samples since it uses a look-up list of DSNs. This MR uses such a file in the test to avoid a python-level crash.


Former-commit-id: aba7270c8ab57ca934b54623651c3dc6ffea7cb8

In this MR the following changes are made to DerivationFrameworkART
 * Adding train tests (for data, 100 events without skimming)
 * Fixing the flavour tagging configuration for both data and MC
 * adding ```athenaopts="s"``` to the TRUTH format merging for debugging purposes


Former-commit-id: 63258a3641a07d54c9dd264b18bae403e2ee35a9

This adds the ```set -e``` to all of the ART jobs, such that if either of the tasks in an ART job fail (DAOD building or merging) then the job returns a failure code and is marked red by ART. Without this MR only the merge job is used to detect the success or failure. The script-making script is also updated to reflect this modification.


Former-commit-id: db33bd38f72183a791dd13156dadab941f200f09

This MR appends the pre-exec to the DAOD building jobs in the ART tests, required to pick up the latest flavour tagging calibration file tag. The ART-script making script is coorespondingly updated as well.


Former-commit-id: 3b47352eead93c50f7b8a9892599a7c1e2619354

AGENE-776

This MR introduces the TRUTH0,1 and 3 formats to the ART tests. A new input EVNT file needs to be introduced to CVMFS for this to run. This is being done today.


Former-commit-id: b6ecf280e6ba1ebe224d181f21e3ef6f7501e134

STDM1 and SUSY9 are obsolete and do not need regular testing. Dropping from the ART tests and also from DerivationFrameworkCore.


Former-commit-id: 2e5aae9dc8199bb76fad29fb6a6eff7e38861eb3

FTAG5 is not in current use and so should not be tested on a nightly basis.


Former-commit-id: b407b43a736810552da264dbc5e68ee4ed58eebf

Former-commit-id: 129c2df826ee4ee4f6a93fc4d7289850723398b6

Adding the non-TRUTH formats to the list of ART jobs, one script per format. No merging yet.


Former-commit-id: 635fb15d2d95f4cf6a9dab45693ef83310150f1d