Output file sequencer adaptation for AthenMT

Changes to AthenaOutputStream and OutputStreamSequencerSvc to allow output file sequences (automatic splitting of the output file into smaller chunks with sequence number added to the filename) to work in AthenaMT. Added protections (mutexes) to allow writing of different chunks concurrently. Small fix to ensure EventContext is present during EndProcessing incident. Output sequences in AthenaMP should work as before (including joboptions configuration)

added 2 commits

• d3000b87 - Teach the new file peeker about LFNs
• 3247bb51 - moved OutputSequencer into OutputStream, use multiple StreamTools

Compare with previous version

added 853 commits

• 3247bb51...4733d710 - 849 commits from branch atlas:master
• b398af9f - Adapted for multithreading
• b9fbc5e2 - moved OutputSequencer into OutputStream, use multiple StreamTools
• 8dfbb436 - Removed deadlock in incidents, still other failures
• c3f58552 - Proper locking in OutputStream for MT

Compare with previous version

added 1 commit

• 3ef515db - Attempt to make AthenaPoolCnvSvc write multiple files in parallel

Compare with previous version

Jenkins please retry a build

This merge request affects 3 packages:

• Control/AthenaServices
• Database/AthenaPOOL/AthenaPoolCnvSvc
• Database/PersistentDataModelAthenaPool

Adding @ssnyder ,@rbianchi ,@mnowak as watchers

added Core Database master labels

CI Result FAILURE

	Athena	AthSimulation
externals
cmake
make
required tests
optional tests

Full details available at NICOS MR-25035-2019-08-16-02-38
Athena: number of compilation errors 0, warnings 0
AthSimulation: number of compilation errors 0, warnings 0
For experts only: Jenkins output [CI-MERGE-REQUEST-CC7 2317]

added 1 commit

• b9edd359 - Make OutputStream referemce output match + small tweaks

Compare with previous version

Jenkins please retry a build

Hi @gemmeren, these changes should actually work (meaning parallel writing in MT, also for event ranges).
commitOutput() is still serialized, but I plan to fix that later too.
I keep WIP on this MR because I am about to leave so can't follow up on anything.
Cheers, Marcin

This merge request affects 4 packages:

• Control/AthenaServices
• Database/AthenaPOOL/AthenaPoolCnvSvc
• Database/AthenaPOOL/PoolSvc
• Database/PersistentDataModelAthenaPool

Adding @ssnyder ,@rbianchi ,@mnowak as watchers

CI Result SUCCESS

	Athena	AthSimulation
externals
cmake
make
required tests
optional tests

Full details available at NICOS MR-25035-2019-08-16-20-09
Athena: number of compilation errors 0, warnings 0
AthSimulation: number of compilation errors 0, warnings 0
For experts only: Jenkins output [CI-MERGE-REQUEST-CC7 2355]

added 1053 commits

• b9edd359...2051a621 - 1047 commits from branch atlas:master
• 5951dcab - Adapted for multithreading
• c10634b6 - moved OutputSequencer into OutputStream, use multiple StreamTools
• 07616b0f - Removed deadlock in incidents, still other failures
• 7ade6862 - Proper locking in OutputStream for MT
• b49ac468 - Attempt to make AthenaPoolCnvSvc write multiple files in parallel
• a7738aab - Make OutputStream referemce output match + small tweaks

Compare with previous version

added 1 commit

• e51dd25f - Resurrected OutputStreamSequencerSvc

Compare with previous version

added 1 commit

• 9277896f - added disconnectOutput to release PoolSvc context on the end of range

Compare with previous version

unmarked as a Work In Progress

changed title from WIP: Adapted for multithreading to Output file sequencer adaptation for AthenMT

changed the description

added 89 commits

• 9277896f...8bf75f61 - 88 commits from branch atlas:master
• 896d9f37 - Merge branch 'master' into sequencer.mt and fix conflict in TPCustomCnv

Compare with previous version

This merge request affects 4 packages:

• Control/AthenaServices
• Database/AthenaPOOL/AthenaPoolCnvSvc
• Database/AthenaPOOL/PoolSvc
• Database/PersistentDataModelAthenaPool

Adding @ssnyder ,@rbianchi ,@mnowak as watchers

added review-pending-level-1 label

added 1 commit

• d4643a13 - Final cosmetic fixes

Compare with previous version

This merge request affects 4 packages:

• Control/AthenaServices
• Database/AthenaPOOL/AthenaPoolCnvSvc
• Database/AthenaPOOL/PoolSvc
• Database/PersistentDataModelAthenaPool

Adding @ssnyder ,@rbianchi ,@mnowak as watchers

CI Result FAILURE

	Athena	AthSimulation
externals
cmake
make
required tests
optional tests

Full details available at NICOS MR-25035-2019-09-09-23-40
Athena: number of compilation errors 0, warnings 1
AthSimulation: number of compilation errors 0, warnings 0
For experts only: Jenkins output [CI-MERGE-REQUEST-CC7 3106]

CI Result FAILURE

	Athena	AthSimulation
externals
cmake
make
required tests
optional tests

Full details available at NICOS MR-25035-2019-09-10-00-00
Athena: number of compilation errors 0, warnings 1
AthSimulation: number of compilation errors 0, warnings 0
For experts only: Jenkins output [CI-MERGE-REQUEST-CC7 3107]

added 1 commit

• b3e5ac6e - Revert some changes to the AthenaOutputStream output reference

Compare with previous version

This merge request affects 4 packages:

• Control/AthenaServices
• Database/AthenaPOOL/AthenaPoolCnvSvc
• Database/AthenaPOOL/PoolSvc
• Database/PersistentDataModelAthenaPool

Adding @ssnyder ,@rbianchi ,@mnowak as watchers

CI Result SUCCESS

	Athena	AthSimulation
externals
cmake
make
required tests
optional tests

Full details available at NICOS MR-25035-2019-09-10-15-08
Athena: number of compilation errors 0, warnings 0
AthSimulation: number of compilation errors 0, warnings 0
For experts only: Jenkins output [CI-MERGE-REQUEST-CC7 3138]

Large MR. Sending to L2. Clara (L1)

added review-pending-level-2 label and removed review-pending-level-1 label

Otherwise looks fine to me, only those (maybe stupid) question....

Pavol [as L2 MR shifter]

added review-user-action-required label

resolved all threads

So looks like this could be approved.

Pavol [as L2 MR shifter]

added review-approved label and removed review-pending-level-2 review-user-action-required labels

merged

mentioned in commit 83df8841

added sweep:ignore label

mentioned in merge request !26639 (merged)