[RTA/DPA BW tests] Add NoBias, Calib & Lumi streams to bandwidth tests

Goes with lhcb-datapkg/PRConfig!420 (merged) and LHCb!4565 (merged) (the latter is to be tested with !3506 (merged) DaVinci!1099 (merged))

Built on top of !3387 (merged), so that should be merged first.

Full bandwidth pages on real data can be found at https://rjhunter-bandwidth.web.cern.ch/

The LHCb MR prevents lots of ERROR messages popping up in GaudiPython jobs to work out the bandwidth when we have calibration lines that have no DstData rawbank, or an empty bank. This should be followed up #777.

• Dont explicitly switch off these 3 streams in the HLT2 stage of the bandwidth tests,
• Adds new options for sprucing them, taken from SprucingConfig.
• Add a VPRetinaCluster bind to the tests that was missing after it was moved out of settings/hlt2_pp_2024.py during veloSP data-taking.

assigned to @rjhunter

mentioned in merge request lhcb-datapkg/PRConfig!420 (merged)

added RTA label

I unfortunately get lots of these errors:

HltPackedBuffer...WARNING  DstData raw bank  has a zero encoding key, and it is not explicitly specified for decoding -- make sure that this really what you want
UnpackDstDataBank   ERROR GitANNSvc::fetch : unable to obtain json for key 00000000 from repositories [ ( '/tmp/dd4hep-stack/Moore/build.x86_64_v2-el9-gcc13-opt/file-content-metadata//.git' , 'master' ) , ( '/cvmfs/lhcb-condb.cern.ch/git-conddb/file-content-metadata.git' , 'master' ) ]
UnpackDstDataBank   ERROR Maximum number of errors ( 'ErrorMax':1) reached.
UnpackAfterPhys...  ERROR Maximum number of errors ( 'ErrorMax':1) reached.
EventLoopMgr      WARNING Execution of algorithm UnpackAfterPhysFilter failed
EventLoopMgr        ERROR Error processing event loop.
EventLoopMgr        ERROR Terminating event processing loop due to errors
EventLoopMgr        ERROR Terminating event processing loop due to errors

looking through DstData for the new streams. I gather that some of the lines don't actually have a DstData raw bank, so I think this is expected, but I should try to suppress these errors...

added 1 commit

• 7db14868 - Extend spruce_latest qmt to do the 3 new streams

Compare with previous version

added 10 commits

• 7db14868...46743178 - 6 commits from branch 2024-patches
• 6c2685d2 - Add remaining streams, and add VPRetinaCluster bind explicitly
• 4b660a6a - add back in sprucing configuration for nobias, lumi and hlt2calib streams
• 62146ac5 - Add options files for sprucing nobias, hlt2calib and lumi streams
• c98bff83 - Extend spruce_latest qmt to do the 3 new streams

Compare with previous version

changed the description

resolved all threads

changed the description

This can wait for !3387 (merged) as we'll have conflicts, but that one is already in testing.

mentioned in merge request !3387 (merged)

added 18 commits

• c98bff83...53638ec6 - 14 commits from branch 2024-patches
• ac053fec - Add remaining streams, and add VPRetinaCluster bind explicitly
• 41fef558 - add back in sprucing configuration for nobias, lumi and hlt2calib streams
• 3851f841 - Add options files for sprucing nobias, hlt2calib and lumi streams
• ebc176a0 - Extend spruce_latest qmt to do the 3 new streams

Compare with previous version

added 15 commits

• ebc176a0...04d19061 - 5 earlier commits
• 8baa0220 - Fix logic in sprucing_physicsstreams_config_check
• 8b0a1ac3 - Replace doca by sdoca
• 5cb6dade - Update References for: !3196 (merged) based on lhcb-2024-patches-mr/448
• 3ce65ced - Update References for: Allen!1615 (merged) based on lhcb-2024-patches-mr/460
• f78f52ac - Speed up TT lines
• 924611a2 - Enabling optimized control flow in hlt2_pp_2024
• cff772d0 - Move BJpsiK PID line from Turbo to Turbo+RAW in sprucing
• be9d0f7a - Add remaining streams, and add VPRetinaCluster bind explicitly
• 9e34aade - add back in sprucing configuration for nobias, lumi and hlt2calib streams
• adbe9100 - Add options files for sprucing nobias, hlt2calib and lumi streams

Compare with previous version

changed the description

Hello @lugrazet @shunan, I was asked to add these streams to the bandwidth tests. Any comments from you both? FYI it's built on top of !3387 (merged) (there was a small conflict otherwise), so the diff looks a little larger than it will do once that one is merged.

I will put the web pages somewhere soon, but there's already some headline results listed here

requested review from @lugrazet and @shunan

added PR/Moore_hlt2_and_spruce_bandwidth label

mentioned in merge request !3496 (merged)

changed the description

added 27 commits

• adbe9100...1f19210b - 24 commits from branch 2024-patches
• becedf2d - Add remaining streams, and add VPRetinaCluster bind explicitly
• acab060c - add back in sprucing configuration for nobias, lumi and hlt2calib streams
• 773c8ce8 - Add options files for sprucing nobias, hlt2calib and lumi streams

Compare with previous version