Always write out output file even when no events are selected

Currently productions with low selection rates often have issues when files have a small number of events and end up selecting nothing.

It would be nicer if FunTuple could always produce it's TTrees even if they contain no events. This would allow LHCbDIRAC to treat the jobs as successful and continue to merge the output files instead of marking the job as failed for not producing output.

(This might already be the case for Run 3 DaVinci but I just had a bunch of painful Run 1 productions to deal with for this issue.)

added Tuples label

I'm actually unsure myself if that's the case already with FunTuple. @amathad, can you comment on that and if it would be much work for FunTuple to comply with Chris's suggestion?

The only downside I see is that analysts might worry to find empty ntuples in files, concluding there was a bug in the writing process ...

So we want to write a root file even if an event does not contain any candidates? Is that correct?

This would allow LHCbDIRAC to treat the jobs as successful and continue to merge the output files instead of marking the job as failed for not producing output.

Is it painful to ask LHCbDIRAC to skip the jobs marked as failed and continue merging? Guess the answer is yes.

For reference - related issue/MR at gaudi/Gaudi!1432 (merged) and LHCb#295 (closed).

Is it painful to ask LHCbDIRAC to skip the jobs marked as failed and continue merging? Guess the answer is yes.

It's conceptually problematic as you then can't separate these two cases:

• The job didn't produce an output file
• The job produced the expected output file but wrote it to the wrong place

The only downside I see is that analysts might worry to find empty ntuples in files, concluding there was a bug in the writing process ...

We have a merging step so this should basically never happen.

Thanks @cburr. I think we all agree this is the way to go. Let's try and get a stack with this sorted in the next round, and that will require that one Gaudi MR to be in, cc @clemenci :-).

I don't think the Gaudi MR will fix this for DaVinci but we can wait and see.

Me either ... One thing is being necessary, the other sufficient ;-). [I did not dwell into Gaudi internals but believe experts when they tell me that one change is necessary. We will likely have some bits to change in DaVinci.]

mentioned in merge request gaudi/Gaudi!1432 (merged)

added Configuration For next release enhancement labels

This can up again with lhcb-datapkg/AnalysisProductions!483 (comment 6624266) which is using DaVinci/v46r5.

mentioned in merge request !937 (merged)

Just a heads up that the propagation of offline luminosity information will greatly profit from this. Related MRs: !937 (merged), LHCb!4174 (merged), LHCbIntegrationTests!47 (merged), gaudi/Gaudi!1478 (merged).

Tagging @efranzos @egraveri @nskidmor @erodrigu @fferrari @rmatev @elniel

by "greatly profit" I think you mean that this is the only way to correctly account for the luminosity. If an ntuple job does not produce an output file when no events are selected, that would mean you miss the luminosity corresponding to the input file. The end result is that the luminosity will be underestimated.

(This might already be the case for Run 3 DaVinci but I just had a bunch of painful Run 1 productions to deal with for this issue.)

It's the first time I hear about this being a problem for Run 1/2. Perhaps it's good to keep this in mind in case people observe discrepancies

Run 1+2 only had production issues with this for MC. For real data the lumi tuple was always written side-stepping the issue.

I was wondering for lumi does either of this solution work? Or is the problem a bit more general?

@rmatev my point exactly.

I think the problem is more general and I would not couple tupling and lumi. In that sense the fact that lumi FSRs use the Sink functionality is most probably the way to go.

Why does it work for Sprucing though, @cburr we sorted this for Sprucing right? Output files are always created with the FSRs in even if no events pass

mentioned in merge request LHCb!4174 (merged)

mentioned in issue Moore#627 (closed)

mentioned in issue Moore#634 (closed)

Hello @cbozzi, @bcouturi, the issue here is not that DaVinci specific but more Gaudi related, since a solution likely requires updates in Gaudi. @clemenci is away. Can you let us know who else from the Computing project can help us undo the knot? I recall there was some issue created by @cburr, I believe, that relates to this. If anyone can post the link that would help get all matters/parties involved.

Many thanks.

assigned to @clemenci

Hello @clemenci, I'm assigning this to you since the issue is becoming urgent for nominal data taking in mid-ish September IMO. I suspect we may need to touch Gaudi again, not just things on the LHCb side, hence the assignment. Let's please discuss soon when you're back from holidays (welcome back, hope you had a good/proper break, BTW). Many thanks.

Hi @erodrigu, can you provide a test case that exposes the problem? Something along the lines of https://gitlab.cern.ch/gaudi/Gaudi/-/blob/master/GaudiExamples/tests/pytest/test_outputfile_no_events.py would be perfect, but a simple test in DaVinci with FunTuple would be OK too.

I need something to help me finding where to inject the fix and to check if the fix actually works.

Hi @clemenci, we have a lumi test for this purpose: $DAVINCIEXAMPLESROOT/tests/qmtest/tupling.qms/test_davinci_tupling_from_spruce_noEventsInSelection.qmt

Tagging @efranzos who wrote that test.

It would also be nice to have a reproducer for DecayTreeTuple as well in case fixing both at the same time is practical so it's also annoying for Run 1+2.

Good morning @clemenci, all. I was too late with my reply as I was replying elsewhere :D. People are "au taquet", it seems .

A simple plain Gaudi(Python) unit test would also seem like a good idea to me as we want to ensure that the FSR infrastructure works just fine even if we are not doing a standard tupling job - imagine for the sake of argument a merging job or a copy-stream type of job. Maybe I'm being maniac but better be as comprehensive as possible, once and for all.

Thank you very much.

Hello @clemenci, let me check with you since we're talking today a lot around lumi - are your investigations here going OK? Do you need anything else? We're trying to get things around lumi ready as much as possible in time for LHCb week.

Thanks for the reminder. I'm basically done with what I wanted to do this week. I start now a build of the full stack so that I can have it done on Monday.

First thing I found is that lhcb-master is broken: see Rec!3363 (comment 7113685)

Call me back when it is fixed.

Morning @clemenci. Rec ... DaVinci all green again :-).

The summary of my investigation is that the handling of ROOT files in Gaudi is a disaster.

The weird thing with NTuples is that the output ROOT file is always created, but it gets re-created when we try to write NTuple entries to it. On the second create we notify LumiFSRtoTTree, but not on the first one and that's why we only get the Lumi TTree is we actually select something.

Patching this weird behaviour might be possible, but the hack is going to be too awful.

Anyway I have an idea for a cleaner fix. For that I'd like to have a chat with @efranzos in the coming weeks, since it involves some change in the logic of both FSRSink and LumiFSRtoTTree.

Hi @clemenci, thanks so much for investigating this. What you describe is very weird and I wonder if it is actually intended to work this way, or it just 'happens' to work. Obviously the ideal scenario would be to have this cleaned up, but I understand that the moment you touch this logic you immediately affect all the rest of the LHCb software. Anyway, please, keep me in the loop when discussing your idea for a cleaner fix. Looking forward!

I think the current implementation is the result of some poorly designed services. I do not think it's meant to work like that, but it has been made to work despite the mess.

Can you keep me up to date on changes to FSRSink as this is Sprucing

Hello @clemenci, we talked the other day in your office about this matter. As it is important albeit not trivial, can you post an update here on the status of the solution? Are you needing any input from DPA? Advance thanks.

mentioned in issue #136 (closed)

mentioned in merge request lhcb-core/lhcbstacks!182 (merged)

mentioned in issue lhcb-simulation/lbmcsubmit#71 (closed)

mentioned in issue LHCbIntegrationTests#7