loopy unpacking and multi packing

Current implementation of persistency configures a packer and unpacker for each location. This MR changes this to:

• one packer per object type
• each unpacker for a given location also unpacks all linked locations if they are not unpacked already. This will allow to request a location to be unpacked individually.
• Packing and unpacking configurations are simplified. PersistRecoConf is removed as most of it's functionality is not needed anymore.

Part of set !3750 (merged) DaVinci!747 (merged) MooreAnalysis!92 (merged) Analysis!918 (merged) Moore!1638 (merged)

assigned to @sesen

added 1 commit

• 70a87259 - Fixed formatting

Compare with previous version

added 73 commits

• 70a87259...e647c6d8 - 14 commits from branch streamline-hltdaq
• 1666ac7b - Remove unused and obsolete HltVertexReports{Writer,Decoder},
• 46911306 - Streamline HltDAQ
• 31383a69 - Add encoding/decoding key to PackedBuffer
• 8d2142f5 - Add GitANNSvc, re-implement TCKANNSvc
• 07df675d - remove old HltANNSvc
• 3a98225d - Move anything RawBank related from EventPacker to HltDAQ...
• 19cd4cb1 - Increment version of DstData, HltDecReports and HltSelReports raw banks
• 34c574d8 - Add support for PackedObjectTypes, i.e. unique strings to a small set of integers
• b9e04016 - remove unused code
• fdc6c037 - Tweak C++ code
• 1cb3a367 - Add support for writing encoding table to git
• 28648ffd - remove mention of TCKANNSvc for readback
• c90fa848 - add writing of output_manifest to options
• 7a8f03f9 - Remove unused mapping of multiple strings to integers from ANNSvc interface
• 8bf667f5 - improve manifest handling
• ede2c64f - Do not 'distribute' the large InputBuffer into many buffers each in its own TES location.
• 1f611430 - make sure the locationID refers to the original TES location
• ec9e65e6 - remove unpackedToPackedLocationMap from PersistRecoConf
• 24a750d1 - GitANNSvc: Allow a 'stack' of git repos to be specified
• 7b8dc668 - remove mention of container (re)mapping
• d6992f05 - unify relation unpackers
• 7dcf6ddc - Fixed formatting
• db6fe824 - improve git interaction, and define proper source branch for merge request
• 4012e5e8 - make unknown output request a non-hard failure, just do not produce output
• 0edba130 - various tweaks
• 363db9aa - enforce DecisionID vs SelectionID distinction
• 4c7f8bcc - more robust treatment of git index
• 65667155 - fix HltSelReportsDecoder bug
• d604aa2f - allow for races
• 16bc68dd - more constraint checks on encoding dict content
• 99ae2260 - better error if no rawbanks with requested sourceID are found
• bece0264 - specify correct sourceID when decoding spruce output
• be338db9 - centralize persist reco persisted locations
• 1756886c - add packed buffer checker and support for DstData v2 decoding
• 8a894e3a - add posibility to overrule git repo using environment variable
• 07c9b6fe - Change data layout in metainfo repo
• 572f5ddd - prefix git repo json filenames with ann/, default to master instead of HEAD revision
• b0bdd29f - settle the DecisionID vs. SelectionID distinction once and for all
• fe95bf2e - add /cvmfs file-content-metada repo into stack of repos
• ffb24976 - do not load /Event/Rec/Summary automagically
• 7e5f2c2f - add argument to tweak persisted locations prefix
• 551e5e15 - add counters for input present to packers
• a40f0660 - Various improvement of the GitANNSvc and the git repo
• c1715584 - fix test
• 948adc01 - Recognize pass-through sprucing step, and require an input manifest in that case
• 55df4cd2 - cleanup
• c1185489 - replace OrderedDict of (repo,tag) with list of pairs
• 9dd0d561 - make it possible to run without cloning/init-ing a git repo, just using /cvmfs
• e9fdd41e - add {input,output}_manifest_file to LbExec options
• d19d4118 - Tweak git interaction to fix the (Allen) ci-test
• 99fadb99 - Move code used by both Moore and DaVinci here, improve handling of old .json.tck file
• 45b743a6 - cleanup
• 6efb34dc - Move encoding key related code to seperate file
• 0b4a4db2 - deal with potential data race when creating repo
• 4e45fd2f - reference update
• cd92ebdd - do not require master==HEAD prior to checking out master
• 471f2605 - update packing and unpacking to pack and unpack multiple locations
• 346fce60 - Fixed formatting
• b9606b5f - fix few things on reading, fix version for Loader

Compare with previous version

added 67 commits

• b9606b5f...30d74f06 - 65 commits from branch streamline-hltdaq
• b4a2dfc1 - update packing and unpacking to pack and unpack multiple locations
• d5b3d0c9 - fix few things on reading, fix version for Loader

Compare with previous version

added 61 commits

• d5b3d0c9...5ecf1349 - 59 commits from branch streamline-hltdaq
• c078f5ad - update packing and unpacking to pack and unpack multiple locations
• d6e6f019 - fix few things on reading, fix version for Loader

Compare with previous version

mentioned in merge request Moore!1638 (merged)

added 1 commit

• 51883542 - remove unused packages from application.py

Compare with previous version

added 67 commits

• 51883542...19618268 - 64 commits from branch streamline-hltdaq
• 2d7cde69 - update packing and unpacking to pack and unpack multiple locations
• cecc4634 - fix few things on reading, fix version for Loader
• 8edb3318 - remove unused packages from application.py

Compare with previous version

mentioned in merge request DaVinci!747 (merged)

mentioned in issue Moore#473 (closed)

added 69 commits

• 8edb3318...bfd6b56c - 66 commits from branch streamline-hltdaq
• 28a78a59 - update packing and unpacking to pack and unpack multiple locations
• 698ebdd9 - fix few things on reading, fix version for Loader
• fbf714c3 - remove unused packages from application.py

Compare with previous version

added 1 commit

• a49bc13d - Fixed formatting

Compare with previous version

added 1 commit

• 486e168f - remove unused import

Compare with previous version

added Persistency RTA enhancement labels

added 76 commits

• 486e168f...ae5a2a57 - 71 commits from branch streamline-hltdaq
• da41fd01 - update packing and unpacking to pack and unpack multiple locations
• 0bd75204 - fix few things on reading, fix version for Loader
• 69677a18 - remove unused packages from application.py
• 18ad82cd - Fixed formatting
• 3a7cabb9 - remove unused import

Compare with previous version

deleted the streamline-hltdaq branch. This merge request now targets the master branch

added 72 commits

• 3a7cabb9...62fb6d46 - 8 commits from branch master
• d7a107b1 - Remove unused and obsolete HltVertexReports{Writer,Decoder},
• 0719fb8b - Streamline HltDAQ
• 90e87c04 - Add encoding/decoding key to PackedBuffer
• 1a3466dc - Add GitANNSvc, re-implement TCKANNSvc
• 81b109cb - remove old HltANNSvc
• 3a44b3ff - Move anything RawBank related from EventPacker to HltDAQ...
• b848417c - Increment version of DstData, HltDecReports and HltSelReports raw banks
• 37e345df - Add support for PackedObjectTypes, i.e. unique strings to a small set of integers
• 3ea5c891 - remove unused code
• acf13739 - Tweak C++ code
• f430650a - Add support for writing encoding table to git
• 52bf1d91 - remove mention of TCKANNSvc for readback
• a473aab5 - add writing of output_manifest to options
• 30aaa987 - Remove unused mapping of multiple strings to integers from ANNSvc interface
• 5c665f57 - improve manifest handling
• 14178cc2 - Do not 'distribute' the large InputBuffer into many buffers each in its own TES location.
• 890cac8e - make sure the locationID refers to the original TES location
• 4c0a7de0 - remove unpackedToPackedLocationMap from PersistRecoConf
• c32a9731 - GitANNSvc: Allow a 'stack' of git repos to be specified
• 468fcb6a - remove mention of container (re)mapping
• 95b43aea - unify relation unpackers
• 230999d1 - Fixed formatting
• d5069b1b - improve git interaction, and define proper source branch for merge request
• bf8e0f04 - make unknown output request a non-hard failure, just do not produce output
• 16e87f3b - various tweaks
• ef0af29a - enforce DecisionID vs SelectionID distinction
• 45c5772b - more robust treatment of git index
• 18e24403 - fix HltSelReportsDecoder bug
• 506bcda5 - allow for races
• 9b39f9a7 - more constraint checks on encoding dict content
• 5d751996 - better error if no rawbanks with requested sourceID are found
• dd93f4bc - specify correct sourceID when decoding spruce output
• 439611d8 - centralize persist reco persisted locations
• e437eb38 - add packed buffer checker and support for DstData v2 decoding
• ed43c5b0 - add posibility to overrule git repo using environment variable
• 3b35b116 - Change data layout in metainfo repo
• fe3fd77f - prefix git repo json filenames with ann/, default to master instead of HEAD revision
• 18e39a99 - settle the DecisionID vs. SelectionID distinction once and for all
• ef7fa72b - add /cvmfs file-content-metada repo into stack of repos
• fdeebc13 - do not load /Event/Rec/Summary automagically
• 982e8252 - add argument to tweak persisted locations prefix
• d31c87b0 - add counters for input present to packers
• 96aa6b50 - Various improvement of the GitANNSvc and the git repo
• 215b60c7 - fix test
• 29ce1823 - Recognize pass-through sprucing step, and require an input manifest in that case
• 3ab21055 - cleanup
• 240ac2b8 - replace OrderedDict of (repo,tag) with list of pairs
• ec7bad37 - make it possible to run without cloning/init-ing a git repo, just using /cvmfs
• 91b34e89 - add {input,output}_manifest_file to LbExec options
• 3716f5d7 - Tweak git interaction to fix the (Allen) ci-test
• 9a9157ba - Move code used by both Moore and DaVinci here, improve handling of old .json.tck file
• 89b28db2 - cleanup
• c92a8962 - Move encoding key related code to seperate file
• a45dc226 - deal with potential data race when creating repo
• 5fd53cfd - reference update
• 285ce22b - do not require master==HEAD prior to checking out master
• 56f26132 - Add registry 'in front of' any key writes
• 3eee8a1b - add cmake function to create local filecontent-metadata repo
• 17cd05a3 - move code around to make Allen standalone happy
• f6032733 - update packing and unpacking to pack and unpack multiple locations
• ea029cd5 - fix few things on reading, fix version for Loader
• 9514c2f4 - remove unused packages from application.py
• 0af2bc4f - Fixed formatting
• 27dde554 - remove unused import

Compare with previous version

added 65 commits

• 27dde554...58c62b79 - 60 commits from branch master
• e43576e4 - update packing and unpacking to pack and unpack multiple locations
• 90a243d4 - fix few things on reading, fix version for Loader
• d285ee63 - remove unused packages from application.py
• 4efa1b63 - Fixed formatting
• 2f7c327d - remove unused import

Compare with previous version

added 1 commit

• ed66fbc7 - remove PersistRecoConf

Compare with previous version

added 8 commits

• ed66fbc7...d6ef5908 - 2 commits from branch master
• 6b51ce40 - update packing and unpacking to pack and unpack multiple locations
• a71cc3a3 - fix few things on reading, fix version for Loader
• 8b423007 - remove unused packages from application.py
• d8b071c2 - Fixed formatting
• 74fc89d5 - remove unused import
• 05c038ff - remove PersistRecoConf

Compare with previous version

added lhcb-head-2 label

added 1 commit

• 8d3e072f - Fixed formatting

Compare with previous version

• [2022-09-15 00:07] Validation started with lhcb-head-2#656
• [2022-09-15 09:10] Validation started with lhcb-master-mr#5655
• [2022-09-16 00:03] Validation started with lhcb-head-2#657
• [2022-09-16 17:13] Validation started with lhcb-master-mr#5674
• [2022-09-17 00:06] Validation started with lhcb-head-2#658
• [2022-09-18 00:03] Validation started with lhcb-head-2#659
• [2022-09-19 00:04] Validation started with lhcb-head-2#660

mentioned in merge request MooreAnalysis!92 (merged)

marked this merge request as ready

Hello @sesen, can this be tested or some further development is expected?

added 11 commits

• 8d3e072f...342d0549 - 4 commits from branch master
• 0e67d318 - update packing and unpacking to pack and unpack multiple locations
• 9863018f - fix few things on reading, fix version for Loader
• 20b3393b - remove unused packages from application.py
• 32135293 - Fixed formatting
• ff0de6d7 - remove unused import
• 1ee6e261 - remove PersistRecoConf
• 894603d5 - Fixed formatting

Compare with previous version

added 1 commit

• e9077139 - unify unpackers including protoparticleunpacker, remove unused algorithms from cmakelist

Compare with previous version

added 10 commits

• e9077139...de1ea843 - 2 commits from branch master
• 8188b5e0 - update packing and unpacking to pack and unpack multiple locations
• a91b5011 - fix few things on reading, fix version for Loader
• 65603468 - remove unused packages from application.py
• 958a78cd - Fixed formatting
• 89191eeb - remove unused import
• 9170865b - remove PersistRecoConf
• 79fd2bac - Fixed formatting
• 8455fa6c - unify unpackers including protoparticleunpacker, remove unused algorithms from cmakelist

Compare with previous version

added 1 commit

• e5e25a29 - add back unpacker/mappers needed for stripping tests, remove special ProtoParticleUnpacker

Compare with previous version

added 2 commits

• d468efea - Fixed formatting
• b603cd78 - move Loaders to src directory

Compare with previous version

added 1 commit

• d3aca7f9 - commit forgotten Loaders.h

Compare with previous version

added 1 commit

• f4b9440f - Fixed formatting

Compare with previous version

resolved all threads

added 1 commit

• 706bde79 - make relation unpacker warning more explicit

Compare with previous version

added 22 commits

• 706bde79...edf2edbf - 8 commits from branch master
• de980597 - update packing and unpacking to pack and unpack multiple locations
• fc531473 - fix few things on reading, fix version for Loader
• d1efe197 - remove unused packages from application.py
• e54d1e29 - Fixed formatting
• 0da7ed55 - remove unused import
• 11ee7bc0 - remove PersistRecoConf
• bd4012cf - Fixed formatting
• b9e2cd20 - unify unpackers including protoparticleunpacker, remove unused algorithms from cmakelist
• d1d9eaf2 - add back unpacker/mappers needed for stripping tests, remove special ProtoParticleUnpacker
• 242d85ec - Fixed formatting
• 0cdfca51 - move Loaders to src directory
• 6c600010 - commit forgotten Loaders.h
• 920b577d - Fixed formatting
• 702fcf20 - make relation unpacker warning more explicit

Compare with previous version

added 1 commit

• d69cef04 - Fixed formatting

Compare with previous version

mentioned in issue Moore#478 (closed)

• [2022-09-20 00:09] Validation started with lhcb-head-2#661
• [2022-09-21 00:04] Validation started with lhcb-head-2#662
• [2022-09-22 00:12] Validation started with lhcb-head-2#663
• [2022-09-23 00:07] Validation started with lhcb-head-2#664
• [2022-09-24 00:07] Validation started with lhcb-head-2#665
• [2022-09-25 00:04] Validation started with lhcb-head-2#666
• [2022-09-25 22:51] Validation started with lhcb-master-mr#5752

added 17 commits

• d69cef04...63142122 - 2 commits from branch master
• 92142f7c - update packing and unpacking to pack and unpack multiple locations
• d0eb4ce3 - fix few things on reading, fix version for Loader
• ef80db20 - remove unused packages from application.py
• f7dedc3f - Fixed formatting
• 15dccbf5 - remove unused import
• 0d3d7c5d - remove PersistRecoConf
• f75eedb6 - Fixed formatting
• 9c40c4c8 - unify unpackers including protoparticleunpacker, remove unused algorithms from cmakelist
• 174408b0 - add back unpacker/mappers needed for stripping tests, remove special ProtoParticleUnpacker
• a5c4a494 - Fixed formatting
• 81e827d4 - move Loaders to src directory
• 956a0912 - commit forgotten Loaders.h
• b0629a72 - Fixed formatting
• 6bbf4f0b - make relation unpacker warning more explicit
• d85ce489 - Fixed formatting

Compare with previous version

added 1 commit

• cd567d00 - use std::invoke

Compare with previous version

added 1 commit

• e49ef539 - Fixed formatting

Compare with previous version

resolved all threads

assigned to @rmatev

unassigned @sesen

Is this ready for a test?