Create Hlt1 line name to number mapping within Allen

Currently, the list of selection line names is saved for the gather selections algorithm as part of the json configuration written in Allen. However, the mapping between selection names and integers used when writing the DecReports is not saved. Instead, the json file is parsed with this function in Moore when running HLT2 / Sprucing and a map between line names and integers is created here. This map is then used to decode the DecReports.

Instead, the mapping between selection names and integers should be written as part of the Allen configuration into the json file and only read in Moore. This will avoid a wrong mapping between names and integers on different systems / with different configurations.

FYI @nskidmor @graven @rmatev @raaij @dcampora

Is there any sort of consistency that has to be maintained in these line-to-integer mappings? Eg. should it be consistent between configuration generations, should a line always get the same integer, etc.; or is it enough to just generate a line-to-integer mapping and store it in the json?

As long as the line-to-integer mappings are propagated from the Allen job to the Hlt2 job it doesnt really matter.

For instance with the AnnSvc PackedObjectLocations we do not expect the string-to-integer mappings to be consistent between different Moore configurations and a new json is created every job.

Thanks, a follow up question. @dovombru when you say "the mapping between selection names and integers used when writing the DecReports is not saved.", do you refer to this integer:

https://gitlab.cern.ch/lhcb/Allen/-/blob/master/device/selections/Hlt1/src/DecReporter.cu#L74

Yes, exactly. Currently, the selections are written out as the configuration of GatherSelections and that is used for decoding the DecReports. How can we be sure that they come in the same order as in selections?

This is guaranteed as it stands right now. The names of active lines is not a set but a list, it is guaranteed that the order will be the same in the dec reporter.

This is what is used in the tests that create a minimal TCK from the json file.

To be very concrete, the one that establishes the order is the configuration:

https://gitlab.cern.ch/lhcb/Allen/-/blob/master/configuration/python/AllenConf/persistency.py#L40

Yes, you are referring to the call of get_allen_hlt1_decision_ids I assume. That's the same function that is used in HLT2 when decoding the DecReports. The worry was that the order obtained here could depend on the system. But I understand now that we should be safe already.

The only thing that I see as open to interpretation is what is the first index. Judging from https://gitlab.cern.ch/lhcb/Allen/-/blob/master/device/selections/Hlt1/src/DecReporter.cu#L74 the first index is 1.

Can we please not have the convention that it is a list, and the number is the index in the list? The decoder wants a one-to-one invertible map between numbers and strings, and yes, I know that a unique list can be used as such and yes, it can be transformed into such a map, and yes, it's safe, and yes it works, but it's just easier if we just make it the same datastructure i.e. a map just as is done elsewhere, and then stick to it.

If you insist on making it a map, then we should support the map datatype as an Allen property (not supported currently). I would like to add proper support for properties since a while, and my last attempt was unsuccessful - if done properly, we could kill two birds with one stone and also solve #179 (closed).

That was a fun afternoon: !799 (merged)

Once that is merged, we can make a map property.

mentioned in merge request !799 (merged)

mentioned in merge request Moore!1424 (merged)

The mapping of line names to number is used in GatherSelections, implicitly in the DecReporter, and through the host_names_of_lines input in the RateValidator. It will likely be needed in more places and the routing bits is an obvious candidate (!689 (merged)).

The mapping is known at initialization (init()) time, but only in GatherSelections. For the routing bits algorithm, it would be very useful to also have it at the time of init(). A property could be added to all algorithms that need this, but seeing that it's already used in a number of places, I'd suggest putting it into the RuntimeOptions instead.

P.S. We don't have the equivalent of Gaudi services in Allen, otherwise that would have been a better solution.

I agree the mapping between the lines and their integer should live someplace central (such as a Service - as you say we still don't have that, and this is not the best moment to start working on that). RuntimeOptions is a good holder for that information.

In addition, another requirement is that this mapping be configured from the json. We could have a field in the RuntimeOptions that holds information configured from the json. This would effectively act as a centralized place to put the configuration of the application. It would also enable to move some of the Allen configuration to python as opposed to the runtime, if we decide to go that way.

Could you sketch -- very briefly -- what would eventually be needed to have services in Allen in an issue, so this is not forgotten?

Sketching this is something I would not want to do lightly, as even a simple sketch already implies a set of requisites which I would rather first discuss with several sources.

Concerning this issue, creating a map in GatherSelections would be easy to do, and creating a RuntimeOptions configuration that works across eg. Gaudi-Allen is a bit more involved. Perhaps we could do it in two separate MRs.

OK fair enough.

mentioned in merge request !689 (merged)

mentioned in issue LHCb#202

@dovombru to follow up on this issue, is this implemented now? How are the mappings passed from the HLT1 to HLT2 step?

cc @jedavies

Hi @nskidmor, the functionality to use maps as properties within Allen was added with !799 (merged). This is not yet used to configure the mapping of integer to string for the line configuration (something we have to follow up on). However, from the HLT2 perspective there should not be any change in the sense that the mapping is read from the json configuration file.

@dovombru @raaij can this be closed?

This was done as part of !911 (merged)

closed