replace IOVReset by LimitedIOVPaths

as discussed for example here, using LimitedIOVPaths would be preferable to IOVReset, which is currently used in

MooreOnline/AllenOnline/options/AllenConfig.py
MooreOnline/MooreOnlineConf/options/beamspot.py
MooreOnline/MooreOnlineConf/options/reco.py

and potentially in local monitoring stacks.

This requires to know the list of conditions to be added to LimitedIOVPaths, ideally generated automatically.

cc @msaur @cagapopo @rmatev

changed milestone to %RTA 2025 Data taking

added RTA WP4 RTA WP5 labels

mentioned in issue Moore#914

added rta-operations label

@jonrob @clara @raaij adding you here. To quote @jonrob "At some point we had a discussion (with @raaij perhaps others, I fail to find it now) that the list of conditions that should be added to LimitedIOVPaths is something that could/should be generated automatically online by the run control, as in principle they are the ultimate source of truth here, as this is what actually creates the new condition links each run. It could then be passed as a list to the tasks via the OnlineEnvBase infrastructure. I don't know if this ever went anywhere though".

Is there any news about getting this list from the Online environment?

Just to note, whilst having online generate the known list is probably the most elegant solution, the list is a) relatively short b) known and c) it does not matter if for a specific task it includes in this list conditions not accessed, as these are just ignored. So until such a list is available (if it ever is) from online it is not a big job to just have a place in PyConf that defines the complete list that tasks can just use. You can see the beginnings of that list here

https://gitlab.cern.ch/lhcb/Panoptes/-/blob/master/Rich/Panoptes/options/RichRefIndexCalib-Online.py?ref_type=heads#L63

when I made that it was I believe complete for what I needed for a tracking + RICH sequence (note though for the RICH I intentionally do not have the scale factor conditions in the above as these are being made by that task so cannot be included, but probably should be in a more general list).

My point is, waiting for such an Online list should not delay migrating tasks that need to use LimitedIOVPaths instead of IOVReset.

yes, I fully agree that it should be straightforward to have a predefined list

I was discussing a bit with @clara this morning and I think it would be easy to have the control system to write the list of produced LimitedIOVPaths in a file into the conditions database. Not as a condition, of course, but as metadata (a bit like the .schema.json file). A Python configuration function can extract the information and use it to configure Hlt/Alignment/Monitoring. We are planning to use this pattern (metadata in the conddb) in simulation for the virtual conddb tags we are finalizing.

@clemenci @clara do you have an estimated timeline for making this automatic procedure in place? From the investigations of @tmombach @zejia and @freiss it is clear that with the latest version of master RecoPhysMon will fail in production - so we need to switch to using LimitedIOVPaths before the start of 2025 data-taking. Depending on the timeline of the feature you're proposing, we can switch to using that directly, or if some more time is needed, we will need a temporary solution (most likely using a hardcoded list of conditions ).

@cagapopo For me you should probably start now to prepare a solution using a simple fixed list. Switching to something else a bit later on, whenever that is, should be trivial.

A hotfix !567 (closed) is available here, given that the proposal in this issue is a long-term project.

sorry @zejia can you provide the exact link to the hotfix you're talking about?

ah ok I see it now, adding it here for reference !567 (closed)

@cagapopo, I understand from @clara that the automatic generation of the list is not difficult (once we define where to write it and in which format).

What I believe is more complicated is to modify all the options that are setting the property LimitedIOVPaths so that they get the data from the list. I can provide a function so that you can do something along the lines of

dd4hep.LimitedIOVPaths = get_limited_iov_paths(dd4hep.ConditionsLocation, dd4hep.ConditionsLocation)

but somebody else should know where it should be invoked...

An alternative would be to modify DD4HepSvc so that it's done automatically and all you have to do is to drop the explicit setting of dd4hep.LimitedIOVPaths (I'd like to keep the property for backward compatibility and as a mean to override the list in the database).

changed the description

mentioned in issue #90

mentioned in merge request !567 (closed)