Enable ParticleCombiner together with FunTuple

Lack of example to do ParticleCombiner together with FunTuple.

cc @amathad @dfazzini

As I mentioned at our live chat you can put up any algorithm in a workflow, expecially if you create that whole workflow yourself. Of course DaVinci is prepared in a way to make it as easy as possible to the typical user ... For the sake of example look at this trivial snipped under https://gitlab.cern.ch/lhcb/DaVinci/-/blob/master/DaVinciExamples/python/DaVinciExamples/debugging/example-PrintDecayTree.py, which I added precisely to show that a DaVinci job can be anything that is allowed for a Gaudi job given the information on the input file and the code available via the dependent libraries.

Whether or not what you want to do is the best way for the job, we can certainly create a snippet. For a start you should then write down the ParticleCombiner workflow part (sequencers are deprecated) ...

assigned to @pkoppenb

This https://gitlab.cern.ch/lhcb/DaVinci/-/blob/master/DaVinciExamples/python/DaVinciExamples/tupling/example-tupling-DTF-run-mc.py example for instance calls make_detached_mumu() which is in https://gitlab.cern.ch/lhcb/DaVinci/-/blob/master/Phys/DaVinci/python/DaVinci/standard_particles.py . You could take these examples as a start and make your own particles. Let me know if that works.

@pkoppenb thanks a lot! Looks promising. Will have a try. Do I understand correctly one can not try with HLT2 output yet for the moment?

The problem with HLT2 output is only related to the MC particle and relations, as long as you use functors that don't need them, like THOR_MASS, the job should work.

@peilian This MR !598 (merged) uses Hlt2 particles.

Hi @peilian, do you now have an example that works for you of inserting a ParticleCombiner instance, based on the examples hinted at above? Then we can close this.

Hi @pkoppenb @erodrigu @dfazzini sorry for coming back late. I just found some time to try this out. The script is running but I got long error about the PV upacking UnpackParticlesAndVertices INFO Unknown objec: Container type 6+210 key 4281. See log in /afs/cern.ch/user/p/peilian/eoslhcb/public/FunTuple/log.out. Any idea about how to fix this? Thanks!

You could find my example and script here: /afs/cern.ch/user/p/peilian/eoslhcb/public/FunTuple, running with command DaVinci/run davinci run-mc --inputfiledb B2JpsiK b2jpsik_data.yaml --joboptfile b2jpsik_opts.yaml --user_algorithms ntuple_b2jpsik:main.

In addition, while import ParticleCombiner from DaVinci.standard_particle, I realized that this is still using LoKi. Maybe ThOr based Combiner needs to be added?

That's a good point. The https://gitlab.cern.ch/lhcb/DaVinci/-/blob/master/Phys/DaVinci/python/DaVinci/standard_particles.py and https://gitlab.cern.ch/lhcb/DaVinci/-/blob/master/Phys/DaVinci/python/DaVinci/algorithms_pyconf.py modules are quite old and should be revised. Do we need all these dvalgorithm methods? @mamartin @erodrigu

Hi. Not, really, I don't think we will need most of them in DaVinci given what the standard user will be doing in ntupling jobs. I duplicated those files from Moore a while back because we needed them to be able to work on the new DaVinci and, most importantly, test it with what was available from online. But I always said it was a temporary measure. To be discussed more extensively next year but IMO many of these helpers, which are now largely duplicated from Moore to DaVinci, should be moved out of Moore if not Moore specific and made available to both online and offline. Analysis is probably a good place. This is no different from what we asked to be done for PyConf, now in LHCb. Some bits may still stay in Moore and that's OK. But I would really push towards no duplication offline versus online, and that should include helpers as well as framework building blocks.

In other words, lets try and sort the "issue" at source and avoid hacky and/or temporary solutions as we approach commissioning times next year.

My 2 cents anyway.

(This "redesign" should clearly be attached to WP3 planning on relevant tasks, not this chatty "issue" :-).)

added DPA-WP3 label

Hello @peilian, is this "issue" still worth keeping open? I had the impression that it was all about an example in the end. Otherwise can you update here what is missing? Advance thanks.

Hi @erodrigu sorry for being silent for a while. I will try to take another look and close it if my test runs. Last time I had problem to run it.

OK, sounds good. Note that if you did not find an example of what you need in DaVinciExamples then we could/should add one based on your options file to improve on "example coverage".

Hi @erodrigu @amathad @dfazzini @pkoppenb I finally come back to this issue. I have prepared the option file with upfront ParticleCombinerWithPVs algorithm to make additional particles, however I have problem to unpack the reconstructed info with errors below. The *.tck.json (/afs/cern.ch/user/p/peilian/eoslhcb/public/FunTuple/options/Hlt2DiMuonLines_noUT.tck.json) file has path /Event/Rec/Vertex/Primary or /Event/HLT2/pRec/Vertex/Primary but not /Event/pRec/Vertex/Primary. How could I let the unpacker know the correct path? I am currently using the upfront_reconstruction from here Thanks in advance.

You could find the option file: /afs/cern.ch/user/p/peilian/eoslhcb/public/FunTuple/options/funtuple_dimuon_bs2jpsiphi.py and other inputs in the same folder.

HLTControlFlowMgr                     FATAL Failed event detected on s: 0  e: 0
UnpackRecVertices                     ERROR Cannot retrieve /Event/pRec/Vertex/Primary from transient store.
UnpackRecVertices                     ERROR Maximum number of errors ( 'ErrorMax':1) reached.
HLTControlFlowMgr                     FATAL Event failed in Node UnpackRecVertex/UnpackRecVertices
UnpackBestTracks                      ERROR Cannot retrieve /Event/pRec/Track/Best from transient store.
UnpackBestTracks                      ERROR Maximum number of errors ( 'ErrorMax':1) reached.
HLTControlFlowMgr                     FATAL Event failed in Node UnpackTrack/UnpackBestTracks
UnpackMuonTracks                      ERROR Cannot retrieve /Event/pRec/Track/Muon from transient store.
UnpackMuonTracks                      ERROR Maximum number of errors ( 'ErrorMax':1) reached.
HLTControlFlowMgr                     FATAL Event failed in Node UnpackTrack/UnpackMuonTracks
UnpackNeutralProtos#1                 ERROR Cannot retrieve /Event/pRec/ProtoP/Neutrals from transient store.
UnpackNeutralProtos#1                 ERROR Maximum number of errors ( 'ErrorMax':1) reached.
HLTControlFlowMgr                     FATAL Event failed in Node UnpackProtoParticle/UnpackNeutralProtos#1
UnpackChargedProtos#1                 ERROR Cannot retrieve /Event/pRec/ProtoP/Charged from transient store.
UnpackChargedProtos#1                 ERROR Maximum number of errors ( 'ErrorMax':1) reached.
HLTControlFlowMgr                     FATAL Event failed in Node UnpackProtoParticle/UnpackChargedProtos#1
UnpackChargedProtos#1                 ERROR Cannot retrieve /Event/pRec/ProtoP/Charged from transient store.
UnpackChargedProtos#1                 ERROR Maximum number of errors ( 'ErrorMax':1) reached.
HLTControlFlowMgr                     FATAL Event failed in Node FunTupleBase_Particles/Phi_Tuple
UnpackChargedProtos#1                 ERROR Cannot retrieve /Event/pRec/ProtoP/Charged from transient store.
UnpackChargedProtos#1                 ERROR Maximum number of errors ( 'ErrorMax':1) reached.
HLTControlFlowMgr                     FATAL Event failed in Node FunTupleBase_Particles/Tuple

Full log: funtuple_test.log

This was no longer in my to-do list. Will add. We used to have such an example in the early days.

@ldufour proposes an example doing spectroscopy with extra tracks (at https://indico.cern.ch/event/1119747/). Need to check if we have a file that actually contains such information.

marked this issue as related to Rec#353 (closed)

Is this still open? Adapting to DaVinci v61r0 I get a similar but different error

Unknown requested input by DataPacking::Buffer::Unpack<LHCb::RecVertexPacker>/RecVertexUnpacker : '/Event/Spruce/pHLT2/Rec/Vertex/Primary'.

where I did the following changes

diff /afs/cern.ch/user/p/peilian/eoslhcb/public/FunTuple/options/funtuple_dimuon_bs2jpsiphi.py  options/funtuple_dimuon_bs2jpsiphi.py
8c8
< from DaVinci.reco_objects import make_pvs_for, make_pvs
---
> from DaVinci.reco_objects import make_pvs_v2, make_pvs
17c17,18
< from DaVinci.reco_objects import upfront_reconstruction_from_file as upfront_reconstruction
---
> from DaVinci.reco_objects import upfront_reconstruction as upfront_reconstruction
> from DaVinci import options
74c75
< v2_pvs = make_pvs_for(process="HLT2")
---
> v2_pvs = make_pvs_v2(process="HLT2")
83c84
< my_filter = add_filter("HDRFilter_Jpsi2MuMu", "HLT_PASS('Hlt2_JpsiToMuMuDetachedLineDecision')")
---
> my_filter = add_filter(options, "HDRFilter_Jpsi2MuMu", "HLT_PASS('Hlt2_JpsiToMuMuDetachedLineDecision')")

Hi @pkoppenb , looking at your error message it seems to me that it could be a mismatch in the process value between the DV options.process (maybe set to 'Spruce') and your option file where you are passing process="HLT2". Can this be the case?
If all the files are in a public location I can try to run the job.

It says "Hlt2" in the yaml file. can you read /afs/cern.ch/user/p/pkoppenb/public/options ?

I had a look at the code. The problem seems to be related to the fact that all the functions in reco_objects (as upfront_reconstruction and make_pvs) assume process='Spruce' as default. So you need to explicitly configure the process value when you invoke them, e.g.: pvs=pvs(process=process) L50.

mentioned in merge request !773 (merged)

mentioned in commit 353988ff

closed with merge request !773 (merged)