split track containers and persistency (versioning)

Introduces split-per-track-type persistency. Track, PID objects and ProtoParticles are all in split containers. Main reason to have more control over persistency.

Also fixes upstream proto/track persistency.

Tested with Hlt/Hlt2Conf/options/hlt2_2or3bodytopo_realtime.py and subsequent sprucing unpack step.

One of the things it does it cleans up and removes the use of merged containers that copy tracks/muonpid/richpid objects, but uses SharedObjectsContainer such that pointers to the individual per-track-type objects are used for merged containers.

Currently the tracks are copied to their persisted location, but this should be SharedObjectsContainers which should be persisted, this way the actual underlying object persistency is handled anonymously (needs !2281 (merged) and SharedObjectsContainers to be persisted)

It aims to fix #392 (closed). @sesen @graven @decianm

Also Ttrack reco should be cleaned up @isanders Maybe fixed in other MRs, see also #588

Also adds fixes for calohypo persistency in charged protoparticles (it started with this, this branch and got out of hand, as it needs these fixes to work).

In addition, uses work of @sbelin from Rec!3509 (closed), !2141 (closed) and DaVinci!897 (closed) (merged into the relevant branches)

Goes with LHCb!4127 (merged), Rec!3423 (merged), Alignment!394 (merged), DaVinci!926 (merged), MooreAnalysis!126 (merged), MooreOnline!268 (merged), lhcb-datapkg/PRConfig!327 (merged)

needs LHCb!4191 (merged) (cherry picked and committed to LHCb!4127 (merged))

added Event model Persistency RTA labels

mentioned in merge request Rec!3423 (merged)

mentioned in merge request LHCb!4127 (merged)

changed the description

added 1 commit

• 1749679c - fix upstream pp config

Compare with previous version

changed the description

added 1 commit

• 4edcfd63 - minor fixes

Compare with previous version

added 1 commit

• 38fc1476 - further upstream persistency config

Compare with previous version

added 1 commit

• d377e0b6 - minor fix

Compare with previous version

added 1 commit

• 12006a73 - fix upstream tracks persistency; remove duplicate pid object persistency

Compare with previous version

changed title from Draft: fix calohypo in proto / upstream persistency to Draft: upstream proto persistency fixes

changed the description

added 1 commit

• ed2c0daf - topo test using relevant reco seq, with downstream tracking too

Compare with previous version

@decianm you'd think its better to fix the rest of the track types (so all tracks are split) in a subsequent MR or this one?

added 1 commit

• a5e27613 - cleanup

Compare with previous version

@mveghel Have you tested this locally? Packing can't deal with SharedObjectContainers yet.

added 1 commit

• 433b7d77 - further cleanup

Compare with previous version

Before this MR is merged, due to the significant implications downstream, I suggest we have a meeting with all concerned parties in RTA and DPA to discuss how we handle the data we already have in the new schema

@graven @sesen @erodrigu @pkoppenb @cmarinbe @decianm

changed title from Draft: upstream proto persistency fixes to Draft: split track containers and upstream/ttrack proto persistency fixes

changed the description

changed the description

changed title from Draft: split track containers and upstream/ttrack proto persistency fixes to Draft: split track containers and upstream proto persistency fixes

added 1 commit

• eabcb1e3 - reco config with split track containers

Compare with previous version

added 1 commit

• 405c3125 - also produce empty persistable locs

Compare with previous version

added 1 commit

• 3daebc24 - cleanup

Compare with previous version

added 1 commit

• e7e25fc1 - versioning of persistence locations

Compare with previous version

I created this: LHCb#312 (closed) to keep track of the location versioning.

changed title from Draft: split track containers and upstream proto persistency fixes to Draft: split track containers and persistency (versioning)

changed the description

added 1 commit

• 32b16214 - added readers for old persistency versions

Compare with previous version

mentioned in merge request !2358 (merged)

mentioned in issue #526

added 1 commit

• 72513b3f - output always latest persistency version; config cleanup

Compare with previous version

resolved all threads

added 1 commit

• e3d76039 - minor cleanup

Compare with previous version

added 1 commit

• 7cfb4f9f - minor cleanup

Compare with previous version

added 147 commits

• 7cfb4f9f...f0628104 - 129 commits from branch master
• f0628104...69935265 - 8 earlier commits
• bb435f05 - cleanup
• 3b78e1fb - further cleanup
• 7c1c8aeb - reco config with split track containers
• 4785a7c2 - also produce empty persistable locs
• 258ff474 - cleanup
• aa38e6c4 - versioning of persistence locations
• ff3eb54a - added readers for old persistency versions
• 1de58512 - output always latest persistency version; config cleanup
• b951b413 - minor cleanup
• 28b8eb93 - minor cleanup

Compare with previous version

added 1 commit

• af65eddc - minor fix and cleanup

Compare with previous version

changed milestone to %RTA/2023.07.04

changed the description

marked this merge request as ready