split track containers and persistency (versioning)

Introduces split-per-track-type persistency. Track, PID objects and ProtoParticles are all in split containers. Main reason to have more control over persistency.

Also fixes upstream proto/track persistency.

Tested with Hlt/Hlt2Conf/options/hlt2_2or3bodytopo_realtime.py and subsequent sprucing unpack step.

One of the things it does it cleans up and removes the use of merged containers that copy tracks/muonpid/richpid objects, but uses SharedObjectsContainer such that pointers to the individual per-track-type objects are used for merged containers.

Currently the tracks are copied to their persisted location, but this should be SharedObjectsContainers which should be persisted, this way the actual underlying object persistency is handled anonymously (needs !2281 (merged) and SharedObjectsContainers to be persisted)

It aims to fix #392 (closed). @sesen @graven @decianm

Also Ttrack reco should be cleaned up @isanders Maybe fixed in other MRs, see also #588

Also adds fixes for calohypo persistency in charged protoparticles (it started with this, this branch and got out of hand, as it needs these fixes to work).

In addition, uses work of @sbelin from Rec!3509 (closed), !2141 (closed) and DaVinci!897 (closed) (merged into the relevant branches)

Goes with LHCb!4127 (merged), Rec!3423 (merged), Alignment!394 (merged), DaVinci!926 (merged), MooreAnalysis!126 (merged), MooreOnline!268 (merged), lhcb-datapkg/PRConfig!327 (merged)

needs LHCb!4191 (merged) (cherry picked and committed to LHCb!4127 (merged))

Hlt/RecoConf/python/RecoConf/hlt2_global_reco.py

@@ -156,11 +156,18 @@ def make_light_reconstruction(usePatPVFuture=False,
 
     # split/unmerged track conversion
     # TODO should be for all track types like this (or rather within the convert function)
-    upstream_tracks = hlt2_tracks.get(
+    best_upstream = hlt2_tracks.get(
         'BestUpstream', {"v1": TracksEmptyProducer(name="FakeUpstreamTracks")})
 
+    upstream_tracks = TrackContainerCopy(
+        name="CreateBestUpstreamContainer_{hash}",
+        inputLocations=[best_upstream['v1']],
+        outputs={
+            'outputLocation': persisted_location('Upstream')
+        }).outputLocation
+
     upstream_v3, upstream_rels = convert_tracks_to_v3_from_v1(
-        upstream_tracks["v1"],
+        upstream_tracks,
         track_types=["Upstream"],
     )
     tracks_v3.update(upstream_v3)
@@ -170,9 +177,8 @@ def make_light_reconstruction(usePatPVFuture=False,
     pvs = make_reco_pvs(velo_tracks, persisted_location('PVs'))
 
     # RICH
-    if skipRich:
-        rich_pids = None
-    else:
+    rich_pids = {'Merged': None, 'Upstream': None}
+    if not skipRich:
         rich_options = default_rich_reco_options()
         richRecConfs = make_all_rich_pids(
             best_tracks,
@@ -180,11 +186,14 @@ def make_light_reconstruction(usePatPVFuture=False,
             make_pids=make_rich_pids,
             track_types=rich_and_charged_proto_track_types)
         # specifically for optional upstream
-        richRecConfs["Upstream"] = make_rich_pids(
-            "Upstream", upstream_tracks["v1"], rich_options)
+        rich_pids['Upstream'] = make_rich_pids(
+            'Upstream',
+            upstream_tracks,
+            rich_options,
+            location=persisted_location('UpstreamRichPIDs'))['RichPIDs']
         # Merge them for now to be backwards compatible.
-        rich_pids = make_merged_rich_pids(richRecConfs,
-                                          rich_and_charged_proto_track_types)
+        rich_pids['Merged'] = make_merged_rich_pids(
+            richRecConfs, rich_and_charged_proto_track_types)
 
     # Calo, enable v3 track versions
     calo_pids = make_calo(
@@ -219,7 +228,7 @@ def make_light_reconstruction(usePatPVFuture=False,
     # default charged protoparticles
     charged_protos = make_charged_protoparticles_from(
         tracks=best_tracks,
-        rich_pids=rich_pids,
+        rich_pids=rich_pids['Merged'],
         calo_pids=calo_pids,
         muon_pids=muon_pids,
         track_types=rich_and_charged_proto_track_types,
@@ -228,9 +237,8 @@ def make_light_reconstruction(usePatPVFuture=False,
     # optional/split upstream protoparticles
     # TODO should be for all track types like this
     upstream_protos = make_charged_protoparticles_from(
-        tracks=upstream_tracks,
-        rich_pids=richRecConfs['Upstream']['RichPIDs']
-        if not skipRich else None,
+        tracks={"v1": upstream_tracks},
+        rich_pids=rich_pids['Upstream'],
         calo_pids=calo_pids,
         muon_pids=None,
         track_types=['Upstream'],
@@ -246,7 +254,7 @@ def make_light_reconstruction(usePatPVFuture=False,
     output = {
         "AllTrackHandles": hlt2_tracks,
         "Tracks": best_tracks["v1"],
-        "UpstreamTracks": upstream_tracks["v1"],
+        "Upstream": upstream_tracks,
         "VeloTracks": velo_tracks["v1"],
         "PVs": pvs["v3"],
         "PVs_v1": pvs["v1"],
@@ -277,7 +285,10 @@ def make_light_reconstruction(usePatPVFuture=False,
         output.update(caloDict)
 
     if not skipRich:
-        richDict = {"RichPIDs": rich_pids}
+        richDict = {
+            "RichPIDs": rich_pids['Merged'],
+            "UpstreamRichPIDs": rich_pids['Upstream'],
+        }
         output.update(richDict)
 
     if not skipMuon: