Re-enable passthrough of events too large to process in Allen

Re-introduces the functionality to pass through an event that is too large for Allen to process. As the sequence is not run for such events, the persistent store is updated in HostBuffersManager::writeSingleEventPassthrough with the minimal set of fields necessary for OutputHandler to process the event and append DecReport, SelReport and RoutingBits banks.

The DecReport uses a dedicated encoding key which encodes the following configuration:

{"Hlt1SelectionID": {"1": "Hlt1PassthroughLargeEventDecision"}, "InfoID": {}, "version": "0"}

The task ID an TCK are taken from the configuration.

RoutingBits that match Hlt1PassthroughLargeEvent (added as rb 26 in the default configuration) are set for these events.

The writing of SelReports has been refactored to make it usable outside of the sequence.

The SelReport is generated using the refactored writer and has been compared against an MDF produced using the passthrough.json sequence. In both cases, the configuration includes only a single line, which passes and has no associated objects to save so the output should be valid.

I tested the pathway by running Allen over a single event with -m 0 to ensure that the GPU memory would be insufficient to process the event. The Following banks were appended to the MDF:

0000b420: cbcb 1800 1103 0001 8428 68e7 0000 0000  .........(h.....
0000b430: 0100 0000 1101 0100 cbcb 1400 3500 0001  ............5...
0000b440: 0000 0000 0000 0000 0000 0000 cbcb 5800  ..............X.
0000b450: 360b 0020 1400 0000 8d46 0000 0c00 0000  6.. .....F......
0000b460: 0e00 0000 1000 0000 1300 0000 1400 0000  ................
0000b470: 0000 0000 0000 0000 0000 0000 0100 0200  ................
0000b480: 0100 0100 0100 0200 0000 0000 0100 0200  ................
0000b490: 0000 0000 0100 0300 0100 0000 0000 803f  ...............?
0000b4a0: 0000 0000 0a                             .....

Two monitoring counters are also added to HostBuffersManager to count the number of slice splits and the number of times an event is passed through. Specifically, these count the number of calls of the returnBufferUnfilled and writeSingleEventPassthrough functions, respectively.

Two CI tests are added to test the slice splitting and passthrough pathway. The first runs Allen over two events with zero GPU memory. The second attempts to decode the DecReports, SelReports and RoutingBits banks from these events.

The MR also includes a couple of small fixes to PersistentStore::inject, which is only used for this pathway - reserve was being called on a by-value copy of the AllenArgument so the pointer was not saved, and a specialisation for vector<bool> was missing.

Closes #494 (closed)

Merge with Moore!3212 (merged) lhcb-conddb/file-content-metadata!88 (merged) MooreOnline!374 (merged)

FYI @gligorov @raaij @edallocc @gcavalle

added 1 commit

• 0ad53c2b - Fixed formatting

Compare with previous version

added RTA label

changed the description

added hlt1-throughput-decreased label

added 17 commits

• 0ad53c2b...f7f303e4 - 16 commits from branch master
• 8fc09c7e - Merge remote-tracking branch 'origin/master' into dcraik_passthroughLargeEvents

Compare with previous version

added 1 commit

• 34b672c3 - Add test of the passthrough mechanism

Compare with previous version

What is the plan downstream of HLT1 for these events? Would they be processed by HLT2 or treated as passthrough also? Do we need a dedicated routing bit for them?

added 1 commit

• 30065f22 - Add counters for slice splits and passthroughs

Compare with previous version

changed the description

added 1 commit

• 763fa0de - Fixed formatting

Compare with previous version

added 1 commit

• 1a7b7940 - Attempt to fix standalone build

Compare with previous version

mentioned in merge request lhcb-conddb/file-content-metadata!88 (merged)

removed hlt1-throughput-decreased label

Thanks for working on this! I must admit I'm not a big fan of providing the empty SelReports as a binary blob; I think that way too fragile. @thboettc do you think it's possible to refactor (hopefully slightly) the SelReport creation code such that some of its functions can be reused to create an effectively empty SelReports bank with one decision set and empty sub-banks? Perhaps using DecReports as input?

The code that creates the DecReports can be similarly refactored to have a function that creates an empty set of reports and another that adds a report for a given line.

I'd also prefer to not use a separate encoding key for events where this mechanism fires. Can't the extra line be added to the "general" encoding key? To avoid issues with assumptions on available lines and the way gather_selections works, I'd propose to add a dummy line with the correct name that never fires, but whose decisions can be reused for this purpose.

@graven any thoughts?

changed milestone to %Developments targeting restart after May MD

added impacthigh label

added priorityhigh label

removed impacthigh label

mentioned in issue Moore#735 (closed)