Project dependencies : use-cases

added DPA-WP3 label

Many thanks @pkoppenb for writing this up! My brief comments

I believe that we will always have some code which is useful offline but not necessary online. Given this a package like Analysis which DaVinci depends on but Moore does not depend on will always be needed. Which component goes where (Rec or Analysis) should be discussed case-by-case based on concrete requirements.
I believe strongly that having all configuration in one place will help to maintain its coherence. I think the people who developed the current Moore configuration spent a lot of time coming up with a coherent way to configure our applications and I think we should profit from this investment of time and thinking by enabling them to maintain the code in a coherent state for the next decade.
I believe that most of the problems identified with respect to versioning DaVinci is introduced by Rec and is therefore independent of DaVinci depending or not depending on Moore. I would appreciate reading a concrete counterexample which addresses this point if one exists.
Sprucing and HLT2 are different configurations of the same application and we will indeed need to support reading data where HLT2 used one Rec version while Sprucing used another. I believe this is mostly an issue of versioning the relevant persistencies. I would suggest thinking through the "rerun HLT2 or Sprucing in simulation removing PID requirements" which will I believe expose many of the real issues involved.

Just to make it explicit, I agree with the points raised here. To me option 2 is the preferred one.

Thanks, appreciated. So then indeed let's make it explicit: option 2 is clearly the carefully considered preference of RTA as a project.

added RTA label

mentioned in merge request DaVinci!711 (merged)

marked this issue as related to DaVinci#18 (closed)

mentioned in merge request !2924 (closed)

Status: waiting for DaVinci!711 (merged). @dfazzini is away, so this is not yet tested.

All the people who have an opinion on this topic should comment here. Would be useful to converge on this at the same time as on lbexec, LHCb#198 (closed), which will be discussed August 12.

Indeed a good timeline. So far we had very little feedback here on what is possible or not with one or the other option, meaning have DaVinci depend on Moore, what (dis)advantages, what implications given foreseen developments for offline, implications for maintenance. I have been dragging to post some feedback from me but will try and do asap ...

Hello @cattanem, @clemenci, you seemed to have some thoughts concerning a possible dependency of DaVinci on Moore, as per our quick chat at a recent PAC meeting. Could you post them here as we need varied input very much? Appreciated, thank you.

Use cases:

Be able to prototype selections with standard particles and directly produce a tuple with selections variables.
Add FunTuple to a trigger or sprucing line to produce a tuple with selections variables.

Option 2 addresses these two use case as it allows to reuse function used in the trigger or the sprucing in DaVinci.

(I actually made DaVinci depend on Moore and used it for use case 2, this worked well).

Did it work just as well simply with lb-run --runtime-project Moore/latest DaVinci/latest ... and no dependency of DaVinci on Moore? ["latest" is a placeholder/example.]

If not, why? And if yes, what's the drawback for your usecases?

Having DaVinci depend on Moore basically guarantees to have a consistent stack which can also be tested in the nightlies without extra developments. Using run time dependencies makes this considerably harder, especially since Moore and DaVinci depend on the same projects. As an example I tried different orders of the runtime dependence:

[lxplus710] /afs/cern.ch/user/s/sstahl > lb-run --runtime-project Moore/latest DaVinci/latest python
WARNING:lb-run:Selected v61r0 for "latest"
WARNING:lb-run:Decided best platform to use is x86_64_v3-centos7-gcc11-opt+g
WARNING:lb-run:Decided best container to use is None
Python 3.9.6 (default, Sep  6 2021, 15:36:22) 
[GCC 11.1.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from RecoConf.hlt2_global_reco import reconstruction
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/cvmfs/lhcb.cern.ch/lib/lhcb/MOORE/MOORE_v53r6/InstallArea/x86_64_v3-centos7-gcc11-opt+g/python/RecoConf/hlt2_global_reco.py", line 16, in <module>
    from .hlt1_tracking import make_reco_pvs
  File "/cvmfs/lhcb.cern.ch/lib/lhcb/MOORE/MOORE_v53r6/InstallArea/x86_64_v3-centos7-gcc11-opt+g/python/RecoConf/hlt1_tracking.py", line 43, in <module>
    from PyConf.Tools import (
ImportError: cannot import name 'MeasurementProviderT_MeasurementProviderTypes__VP_' from 'PyConf.Tools' (unknown location)

[lxplus710] /afs/cern.ch/user/s/sstahl > lb-run --runtime-project DaVinci/latest Moore/latest python
WARNING:lb-run:Selected v53r6 for "latest"
WARNING:lb-run:Decided best platform to use is x86_64_v3-centos7-gcc11-opt+g
WARNING:lb-run:Decided best container to use is None
Python 3.9.6 (default, Sep  6 2021, 15:36:22) 
[GCC 11.1.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from RecoConf.hlt2_global_reco import reconstruction
>>>

So one direction worked, the other not. It seems tedious to me to having to document which combinations work.

Interesting, there seems to be some non-consistency between the 2 Moore and DaVinci versions picked up.

Of course in practice I don't see why anyone would ever want to do the command 1 above since, in the discussion we're having, it would be Moore wanting to use FunTuple from Davinci, hence lb-run --runtime-project DaVinci/latest Moore/latest python, but nobody running a DaVinci job would ever really want/need to link to Moore in DaVinci since all PyConf and config stuff should by design be shared already (not yet the situation, unfortunately). In short, only one option - the one that worked in this example - is really the relevant thing to do.

PyConf and config stuff should by design be shared already (not yet the situation, unfortunately).

PyConf and the functions in Moore are designed to be shared as there is no global state until you call run_moore.

Right, that part is indeed shared since we requested ages ago to have PyConf moved to LHCb to configure DaVinci based on that. I was not totally clear, see the MR DaVinci!700 (merged) from Davide - at the moment DaVinci configures itself from several modules that contain duplication, other bits could be removed if they were not in Moore but rather in LHCb or Rec (these matters I believe are what triggered this big thread). We will get there whatever is decided in here.

Right, that part is indeed shared since we requested ages ago to have PyConf moved to LHCb to configure DaVinci based on that.

That's not what I meant but I won't comment on that point further.

changed the description

I'm finally about to put down some comments from my side but let me first interleave replies to the above:

I believe that we will always have some code which is useful offline but not necessary online. Given this a package like Analysis which DaVinci depends on but Moore does not depend on will always be needed. Which component goes where (Rec or Analysis) should be discussed case-by-case based on concrete requirements.

I think we all agree on this point of some code being only for offline. But see below for what a reshuffle or code implies if we go down fully the food chain ...

I believe strongly that having all configuration in one place will help to maintain its coherence. I think the people who developed the current Moore configuration spent a lot of time coming up with a coherent way to configure our applications and I think we should profit from this investment of time and thinking by enabling them to maintain the code in a coherent state for the next decade.

I also think we all agree on this. As DPA said many times - certainly I did - the few configuration files we presently have in DaVinci have always been largely temporary so that we could start working 2 years ago. The fact that we have recently created a MR to clean-up and merge with what Moore has is an implicit agreement with the idea. In fact I have alsways been convinced that this clean up and rearrangement should be done no matter what, meaning whether we make DaVinci dependent on Moore or not, and that because it gets rid of much duplication that is a maintenance nightmare.

I believe that most of the problems identified with respect to versioning DaVinci is introduced by Rec and is therefore independent of DaVinci depending or not depending on Moore. I would appreciate reading a concrete counterexample which addresses this point if one exists.

True that the strong links come from Rec ... In fact you could also say LHCb and even Gaudi, see some more thoughts from me below. In the end what matters is that we minimise the maintenance burden and the couplings, because they both exist and will exist even more in the near term.

Sprucing and HLT2 are different configurations of the same application and we will indeed need to support reading data where HLT2 used one Rec version while Sprucing used another. I believe this is mostly an issue of versioning the relevant persistencies. I would suggest thinking through the "rerun HLT2 or Sprucing in simulation removing PID requirements" which will I believe expose many of the real issues involved.

Fair enough, it certainly is an issue we touched upon several times already but we never actually got to the bottom of it. This being said, Sprucing is and will always be a Moore job, hence a separate topic from DaVinci.

Use cases:

Be able to prototype selections with standard particles and directly produce a tuple with selections variables.

You mean do what RTA has been doing with MooreAnalysis, in essence. DaVinci depending on Moore allows you to do this in 1 step rather than 2 (processing and then tupling). That's the difference. Now, in 2023, when you will have MC samples and some MinBias samples to work with, the processing will have been done by definition, so why would you go and redo it anyway? I don't see the need, really. Also, will you use some old reconstruction + persistency or the latest version? If that's the latest then I go back to my comment that the processing will have been done in the MC samples, hence not needed anymore. Blindly wanted to run a single job (for the wrong reasons) is not a good enough argument to implement strong coupling between applications IMO.

Add FunTuple to a trigger or sprucing line to produce a tuple with selections variables.

Ditto.

Let me now dump a few comments I've had in the back of my mind for a while:

MC INFORMATION

There are also matters related to MC matching to take into account. As you are suggesting to move FunTuple upstream in the stack, you are effectively suggesting that all needed tools and algorithms that are used by, or work in connection with MC info extraction for, FunTuple, be moved upstream likewise. Is that something you considered and are happy with? So far the RTA message was "we do not want such MC stuff". Unless you're fine?

MAINTENANCE, BRANCHING AND RELEASES

In your (RTA) argumentation I have seen so far no mention of what your proposal implies in terms of maintenance, branching and releases. We know from previous experience that branching is necessary and ieivitable. But we ought to minimise it, for obvious reasons. I miss any discussion from you of how you see these 3 matters with your proposal.

Just to give a concrete example: already now DPA/offline is keen to use the new LCG stack 102 with extra goodies. And we will want to move probably faster than RTA with Gaudi enhancements, for example we have already started a discussion with the ROOT team to test the new RNTuple for the creation of user ntuples in Analysis Productions (separately to production - will avoid the discussion of this topic here). That will have to be incorporated in the stack at some point, one way or another. Have you thought of use-cases such as these and how your proposal is better or worse to deal with such matters?

Aside these cool developments, we have to consider the "daily life" we all know from the past, of having to tag and release stacks for analysts to incorporate bug fixes, improvements and enhancements, new tools, etc. Some of these will be upstream in the stack, hence either route we take will see the same work (e.g. if the update is in LHCb or Rec). But what about things that would be only in Analysis? Are you willing to commit somebody - likely Christoph or Sasha given their strong push of the idea? - of order 10% of their time (1 person) for the next few years to deal with all the stack releases we will need and which will required tags upstream? Unless you are assuming that DPA will in the end be doing all tags and releases of the stacks according to needs? This has never been on the table and it is not a detail as far as I'm concerned. For completeness less me stress that we will have many stack releases in the near future one way or another. That we all know. Today already we have been waiting for more than 1 week for a release of Moore, which we asked for. It never came. The future seems tougher and I wonder how smooth the experience will be - not a fun discussion, but a needed one.

A "final" point in this round of initial comments from me: with your proposal, and given the packages presently in Analysis together with Phys/FunTuple, the move of the latter to Rec (or LHCb) would imply that other packages in Analysis will also have to move, as said above. We would then be left with a tiny Analysis project, at which point I see no real need to maintain an extra project with, say for the sake of argument, only 2-3 packages in it. It would then seem natural to get rid of Analysis ... at which point DaVinci also has no reason d'etre since the tupling functionality + MC stuff needed by it would all be upstream. In short, you are effectively suggesting, pragmatically speaking, to get rid of Analysis and DaVinci. Is that what you have in mind with your "option 2"? Because that is my interpretation trying to think it through.

To wrap up, I want to make sure we think this though and do not decide based on "I would like to do my job all in one".

I will stop here and this is more than enough for a first thread from me.

I'm glad we seem to agree on most of points 1 -- 4.

I think the topic of FunTuple should not be mixed with DaVinci depending on Moore. If we can agree that DaVinci depends on Moore we solve most of the issues and I am happy to then commit RTA to a good-faith negotiation on the any associated maintenance burden. FunTuple can be rediscussed at a later point as and when convenient for both of us, but moving it would not be a priority for RTA if DaVinci depends on Moore.

I agree that LCG dependencies are delicate however see point (3): once you depend on Rec you have to deal with this difficulty anyway. So it does not seem relevant to the decision we are trying to make here, especially because we estimate almost all the maintenance burden comes from Rec rather than Moore. Nevertheless I am again happy to commit RTA to a good-faith negotiation of the associated maintenance burden and setting of operational priorities. In the end for what concerns production the projects provide possible solutions: the collaboration makes decisions through the OPG and PPG. So as long as we present the technical possibilities in good faith choosing the concrete outcome is not really our job anyway.

Finally on your point about wanting to run jobs for the wrong reasons I am strongly (even rather dogmatically) opposed to a software management culture which tries to define for analysts what the correct and wrong ways to run their own jobs are. We have a clear division between production jobs, which are defined in a highly bureaucratised set of projects for good reasons of operational stability, maintainability, and general quality -- and analyst jobs which are defined by analysts with as much freedom as possible. Now we have a community of brilliant, creative, and intellectually diverse analysts to support. It seems to me completely counterproductive to second-guess (rather than listen) what they might want to do. Almost everything which I have worked towards for the past years has been to increase the choices and flexibility in how we analyse our data once it leaves the pit, not restrict it. Of course we cannot centrally support an infinity of use-cases but what we are discussing here are a few simple workflows which however allow a great deal of creative expression by analysts. (Which I think this collaboration should be encouraging) So I am afraid here we don't agree on the relevance of the question.

Finally getting to comment, I don't really have anything to add with what is already said. For me the biggest worry is the branching issue, as discussed by @erodrigu - we know, and we even plan - to have at least N+1 stacks of the software (one for each year of data-taking, driven by the need for a stable and reproducible trigger+sprucing configuration, and a master branch, with all new developments for "next year"). As I understand it, analysts will always want to use the master version of DaVinci, that gives access to the latest ROOT and LCG goodies. In the bad old world this was easily achievable by tagging the DaVinci/master stack whenever needed, without worrying with the other applications. With a dependency on Moore, it will be necessary to actively maintain at least two branches of Moore, the version for this year's data-taking and the version for DPA use with DaVinci. It is doable, but RTA need to agree to give it the priority dictated by DPA needs. One problem with such branching is also to forward port to master stack all fixes done on Moore/202N-patches stack, it's nothing new but may be an issue if the people responsible are the same as those trying to make the current year's trigger or sprucing work.

I would like to see this issue addressed explicitly in the proposal, and responsibilities and workflows formalised, when the dependency is introduced, and not as an afterthought later.

@cattanem thanks, but I still don't understand how Moore introduces so much extra work here compared to the dependency on Rec. Can you please spell out your thinking a bit more on this point?

As I already said that RTA will discuss in good faith on this point post-agreement I would appreciate if we could leave alone comments about formalising things. It is in my opinion counterproductive to the collaborative culture we should be trying to foster.

IMHO when you release an application (Moore, DaVinci) you give an implicit guarantee that the application has been tested and people can use it as a basis for further work. So, even though a tag of Moore/master (and the corresponding reconstruction) needed for a DaVinci release might never be used for production, you still have to do all the validation. It's perhaps not a bad thing to do repeated validations of master, but it takes time, and people should be made aware of this.

Which is tied to the good faith argument. In the end, you can have all the collaborative spirit you want, but there are only 24 hours in the day and priorities will need to be set. The experience of the last several months is that, for good reasons, priorities were being set within one project and other projects just had to wait.

Sure, but again, and I am sorry to be boring, the moment you depend on Rec you have to deal with all that. Moore is essentially just a shell of specific configurations around this. I really think that all your points are good points but simply they are things we have to deal with anyway. If I am missing something I'd really appreciate a concrete counterexample.

To clarify one point on the process, let me say here more directly what I hinted at above. The OPG and PPG set priorities, we do our best to carry out the plan. That's exactly what's happening now for example with commissioning and DD4HEP: we have received a crystal-clear order to drop everything else and make DD4HEP happen by the 3rd week of September and we are working accordingly. The same will be true in the future. RTA has done as much as any project to increase the collaborative participation of LHCb members in software maintenance and we will continue to encourage all colleagues to contribute to such activities. It then frankly leaves a bad taste in the mouth when colleagues demand written assurances of good faith from us. If I give my word that we will not dump the consequences of any decision on other projects and that we can discuss specific repartition of work calmly in the next weeks/months then this should be enough.

"Sure, but again, and I am sorry to be boring, the moment you depend on Rec you have to deal with all that. Moore is essentially just a shell of specific configurations around this. I really think that all your points are good points but simply they are things we have to deal with anyway. If I am missing something I'd really appreciate a concrete counterexample."

If I may, the difference I think Marco is eluding to is Moore is a project that releases an application, whereas Rec is not. Rec can only be used indirectly, when used by some other project that itself defines an application. This is why there is a sutble, but I think important distinction between DV requiring a new Rec release, and DV instead requiring a new Moore release.

OK but in practice almost all the hard work for switching e.g. LCG versions as noted above will come in LHCb/Rec, and Moore basically just provides extra validation on top of the validation which already exists in Rec. So if anything I suspect in practice this will help you anticipate problems and therefore potentially even reduce the overall maintenance burden from patches etc. I get that releasing an application feels different, and probably some appropriate disclaimers are needed in the release notes, but the material reality that almost all the code which does any real work lives in LHCb/Rec matters more imho.

(I know and understand that there is a real and non-negligible maintenance burden with releasing every extra package, and I realise all too well that extra tests which must pass are a real development burden as well. I'm not trying to wave that stuff away. But if the argument comes down to "too much work to have to look at all Moore tests every time" then since most of them are exposing problems in LHCb/Rec I would say better to depend on Moore but explicitly decide not to look at them rather than not depending on Moore.)

mentioned in merge request Analysis!898 (merged)

Today's discussion at the RTA-DPA meeting concluded this can go ahead and will be revisited in mid 2023.

MR DaVinci!711 (merged) can go ahead, @dfazzini . Closing this when merged.

Thanks Patrick. Can you also ensure the decision is logged in our DPA master task mentioned in the header.

Indeed we now have a few actions to take and we follow up post-commissioning via #389 (closed).

mentioned in issue #389 (closed)

marked this issue as related to #389 (closed)

Hey @gligorov, as a corrolary of the decision, MooreAnalysis now becomes redundant and irrelevant. I hope that you can discuss this and its implications among the RTA team so that we fully focus the near- and longer-term future in FunTuple et al. towards the best possible solution for tupling. Indeed I would qualify the maintenance of MooreAnalysis as a burden at this stage.

Happy to talk more with anyone on this as my 2 sentences are pretty short, though clear. Thanks a lot.

no pb @erodrigu we will discuss. Bon weekend!

Thanks. You too!

mentioned in commit DaVinci@0b4252ad

closed with merge request DaVinci!711 (merged)

MR DaVinci!711 (merged) is merged, so DaVinci now depends on Moore. Still need DaVinci!700 (merged) for the task to be completed.

Project dependencies : use-cases

Designs

Child items 0

Activity

MC INFORMATION

MAINTENANCE, BRANCHING AND RELEASES

Project dependencies : use-cases

Relates to

Activity

MC INFORMATION

MAINTENANCE, BRANCHING AND RELEASES