Check of unique rawbank sourceID in SelectiveCombineRawBankViewsToPersist

Added rawbank sourceID check to SelectiveCombineRawBankViewsToPersist and CombineRawBankViewsToPersist Check is implemented as parsing unordered_multimap with saved sourceID of all banks which were already combined into a new rawevent.

Currently, test_excl_spruce_2023_1_data.qmt will fail as data taken during that period consist of bug where ODIN banks were saved twice. For that reason the newly introduced parameter, AllowSameSourceID, is set to True.

Goes together with Moore!2796 (merged)

Resolves Moore#614 (closed)

added Persistency label

requested review from @graven, @sesen, and @nskidmor

assigned to @msaur

@graven @nskidmor @sesen In this draft I am trying to implement checking of unique rawbank sourceID when combining into a new rawevent.

Currently I am using an unordered_multimap to collect all sourceIDs when bank is added to a new rawevent and checking those IDs when a new bank is added (expecting that the first bank is always unique).

Locally tested by requiring ODIN bank as a selective bank on top of the stream requirements. With this code only one ODIN bank is always saved.

Any ideas how to make it faster and more elegant are very welcome.

added RTA label

added 23 commits

• af060fec...9b4d32a0 - 22 commits from branch master
• 5b4f0d92 - check of unique rawbank sourceID

Compare with previous version

mentioned in merge request Moore!2436 (merged)

added 37 commits

• 5b4f0d92...06cd9515 - 35 commits from branch master
• 359174be - check of unique rawbank sourceID
• e9043e4b - CombineRawBankViewsToRawEvent modified

Compare with previous version

added 2 commits

• e9d8619c - preparing for rewriting SelectivePacker to allow ordering
• 82baa969 - added SourceID check to SelectiveCombiner

Compare with previous version

added 1 commit

• 2ba1ac16 - throwing exception instead of removing same-sourceID bank

Compare with previous version

added 124 commits

• 2ba1ac16...1ecddf03 - 119 commits from branch master
• 7db2ca41 - check of unique rawbank sourceID
• 1647bde7 - CombineRawBankViewsToRawEvent modified
• 9eae5549 - preparing for rewriting SelectivePacker to allow ordering
• adf5c8a0 - added SourceID check to SelectiveCombiner
• 7d41989d - throwing exception instead of removing same-sourceID bank

Compare with previous version

@graven @nskidmor @sesen Would you have any additional comments or could I mark this MR as ready and let it be tested and pass it to the queue for merging?

marked this merge request as ready

@mzdybal Could we run a ci-test to see if everything is fine? Thanks.

added ci-test-triggered label

• [2023-10-25 08:55] Validation started with lhcb-master-mr#9541
• [2023-11-07 18:22] Validation started with lhcb-master-mr#9630
• [2023-11-09 14:01] Validation started with lhcb-master-mr#9643
• [2023-11-10 08:27] Validation started with lhcb-master-mr#9646
• [2023-11-11 13:39] Validation started with lhcb-master-mr#9649
• [2023-12-14 08:36] Validation started with lhcb-master-mr#9913
• [2023-12-15 09:01] Validation started with lhcb-master-mr#9934
• [2023-12-30 07:45] Validation started with lhcb-master-mr#10017
• [2023-12-31 11:41] Validation started with lhcb-master-mr#10018
• [2024-01-01 07:45] Validation started with lhcb-master-mr#10019
• [2024-01-01 08:22] Validation started with lhcb-master-mr#10020
• [2024-01-01 09:22] Validation started with lhcb-master-mr#10021
• [2024-01-02 04:05] Validation started with lhcb-master-mr#10022
• [2024-01-08 07:57] Validation started with lhcb-master-mr#10035
• [2024-01-23 10:02] Validation started with lhcb-master-mr#10211

assigned to @mzdybal

mentioned in issue Moore#665 (closed)

Assigning to new shifter @jagoodin .

assigned to @jagoodin and unassigned @mzdybal

added 38 commits

• 7d41989d...b831fea2 - 32 commits from branch master
• ec109d61 - check of unique rawbank sourceID
• 25dd3714 - CombineRawBankViewsToRawEvent modified
• bb6927ab - preparing for rewriting SelectivePacker to allow ordering
• c352c545 - added SourceID check to SelectiveCombiner
• 61a8d186 - throwing exception instead of removing same-sourceID bank
• 06571cd2 - priting type of bank when having a same sourceID

Compare with previous version

added 1 commit

• df3f3cbe - additionally adding printing bank type for sourceID check to CombineRawBankViewsToRawEvent

Compare with previous version

mentioned in merge request Moore!2736 (closed)

added 12 commits

• df3f3cbe...578aa36c - 4 commits from branch master
• a0ed4a42 - check of unique rawbank sourceID
• 2405815a - CombineRawBankViewsToRawEvent modified
• 7338cebf - preparing for rewriting SelectivePacker to allow ordering
• 2e00df63 - added SourceID check to SelectiveCombiner
• 1814c2d0 - throwing exception instead of removing same-sourceID bank
• 696b73b5 - priting type of bank when having a same sourceID
• 6d0e5e0d - additionally adding printing bank type for sourceID check to CombineRawBankViewsToRawEvent
• 86ab8d06 - changing sourceID message from DEBUG to INFO

Compare with previous version

added 1 commit

• 3c6c3cb7 - implementing comment about improving error message

Compare with previous version

added 1 commit

• bec18e7c - improving debug messages

Compare with previous version

added 1 commit

• 57d7867e - Fixing missing filter for selective banks

Compare with previous version

@msaur I noticed we check for same source id for required and selective banks separately. This means we could still end up with multiple banks with same source ID. Since the nanofication MR, we only have selective banks. So we either remove the required raw banks option or check for overlap in required and selective banks.

mentioned in issue Moore#669 (closed)

assigned to @mramospe and unassigned @jagoodin

added 1 commit

• d74a82f6 - removing required_banks, part I

Compare with previous version

added 1 commit

• bfeb41dd - printing location of empty bank

Compare with previous version

added 1 commit

• ab708d0f - modified CombineRawBankViewsToRawEvent to have same logic/code as SelectiveCombiner

Compare with previous version

added 1 commit

• abfdbb68 - removing empty banks in CombineRawBankViewsToRawEvent

Compare with previous version

mentioned in issue Moore#671 (closed)

assigned to @bjashal and unassigned @mramospe

added 33 commits

• abfdbb68...e97440b5 - 17 commits from branch master
• e97440b5...6c05dc51 - 6 earlier commits
• f032a71c - additionally adding printing bank type for sourceID check to CombineRawBankViewsToRawEvent
• 8136152d - changing sourceID message from DEBUG to INFO
• 99b47808 - implementing comment about improving error message
• 0a778fb5 - improving debug messages
• f2634f47 - Fixing missing filter for selective banks
• 9c0a3c36 - removing required_banks, part I
• dc004693 - printing location of empty bank
• 2e576988 - modified CombineRawBankViewsToRawEvent to have same logic/code as SelectiveCombiner
• c9ddc4fb - removing empty banks in CombineRawBankViewsToRawEvent
• 0d20d197 - priting locaiton of empty banks when MSG::DEBUG

Compare with previous version

added 1 commit

• 5abe920c - comparing content of rawbanks with a same sourceID

Compare with previous version

unassigned @bjashal

added 1 commit

• 1bdf043f - comparing content of rawbanks with a same sourceID, moving variables within a proper if condition

Compare with previous version