DaVinci issueshttps://gitlab.cern.ch/lhcb/DaVinci/-/issues2024-03-18T15:03:37+01:00https://gitlab.cern.ch/lhcb/DaVinci/-/issues/184Follow-up from "Follow up on particle PV pointer, offline PV unbiasing example"2024-03-18T15:03:37+01:00Andre GuntherFollow-up from "Follow up on particle PV pointer, offline PV unbiasing example"The following discussion from !1013 should be addressed:
- [ ] @wouter started a [discussion](https://gitlab.cern.ch/lhcb/DaVinci/-/merge_requests/1013#note_7721519): (+17 comments)
> I still found something strange concerning the...The following discussion from !1013 should be addressed:
- [ ] @wouter started a [discussion](https://gitlab.cern.ch/lhcb/DaVinci/-/merge_requests/1013#note_7721519): (+17 comments)
> I still found something strange concerning the "NDOF" field: It is stored in the TTree as an integer (which it should be), but if I scan it with TTree::Scan or draw with TTree::Draw it shows up as a double with nonsense values. Any ideas what could go wrong? Does FunTuple need special instructions for an integer value?https://gitlab.cern.ch/lhcb/DaVinci/-/issues/183Job is successful despite the error but no output file2024-03-26T09:45:12+01:00Federica FabianoJob is successful despite the error but no output fileTwo subjobs (Job IDs: 857672513 and 857672509) claim to be "completed", but:
- the application status is "No Output Data Files To Upload", indeed they have no output file;
- they both report the same error in the log file: "R__unzipLZMA:...Two subjobs (Job IDs: 857672513 and 857672509) claim to be "completed", but:
- the application status is "No Output Data Files To Upload", indeed they have no output file;
- they both report the same error in the log file: "R__unzipLZMA: error 9 in lzma_code".
Their site is the same: NCBJ.
I attach the full log files of the two subjobs, while the LFN of the two input files are:
- for 857672509, LFN:/lhcb/LHCb/Collision17/IFT.DST/00105545/0000/00105545_00002070_1.ift.dst
- for 857672513, LFN:/lhcb/LHCb/Collision17/IFT.DST/00105545/0000/00105545_00006999_1.ift.dst
[Script1_Ganga_GaudiExec_857672513.log](/uploads/2a64b974b1e1217c3ff80ceeba1e9fb5/Script1_Ganga_GaudiExec_857672513.log)[Script1_Ganga_GaudiExec_857672509.log](/uploads/29491ea3a43a6fd37a70bb8f708b6398/Script1_Ganga_GaudiExec_857672509.log)Mark SmithMark Smithhttps://gitlab.cern.ch/lhcb/DaVinci/-/issues/182FunTuple DTF_BestPV_PV_X stores position of decay vertex instead of PV2024-03-14T15:59:28+01:00Wouter Hulsbergenwouterh@nikhef.nlFunTuple DTF_BestPV_PV_X stores position of decay vertex instead of PVThe quantity that FunTuple labels as "Bs_DTF_BestPV_PV_X" etc, is actually the endvertex of the B instead of its origin.
It is not so easy to fix this unless we add something to DTFAlg to access the head of the decay tree. (But see also...The quantity that FunTuple labels as "Bs_DTF_BestPV_PV_X" etc, is actually the endvertex of the B instead of its origin.
It is not so easy to fix this unless we add something to DTFAlg to access the head of the decay tree. (But see also [this issue](https://gitlab.cern.ch/lhcb/Rec/-/issues/540) for what may be an easier way forward.) Until that is fixed, it may be better to remove this altogether.
However, to get access to the PV associated to the particle there is now an easier set of functors. I added 'functorcollections.PrimaryVertexInfo' to add this to FunTuple:
https://gitlab.cern.ch/lhcb/DaVinci/-/blob/sevda-v2PV-packing/Phys/FunTuple/python/FunTuple/functorcollections.py#L368
We can eventually extend this with the value of 'F.OWNPV_IPCHI2' which is probably the one functor that is most wanted.Jiahui ZhuoJiahui Zhuohttps://gitlab.cern.ch/lhcb/DaVinci/-/issues/181Problematic file with dq_flags OK in PbPb2023 sprucing data2024-03-15T14:07:06+01:00Pedro NogarolliProblematic file with dq_flags OK in PbPb2023 sprucing dataIn the production of diphoton tuple using PbPb2023 sprucing data (https://gitlab.cern.ch/lhcb-datapkg/AnalysisProductions/-/issues/271), one file crashed - LFN:/lhcb/LHCb/Lead23/IFT.DST/00208341/0005/00208341_00055039_1.ift.dst. The logs...In the production of diphoton tuple using PbPb2023 sprucing data (https://gitlab.cern.ch/lhcb-datapkg/AnalysisProductions/-/issues/271), one file crashed - LFN:/lhcb/LHCb/Lead23/IFT.DST/00208341/0005/00208341_00055039_1.ift.dst. The logs of the crash can be found in issue https://gitlab.cern.ch/lhcb-datapkg/AnalysisProductions/-/snippets/3013.
Logs:
https://lhcb-dirac-logse.web.cern.ch/lhcb/LHCb/Lead23/LOG/00215126/0000/00005594/
Crash:
```HLTControlFlowMgr FATAL Event failed in Node VoidFilter/Hlt1HeavyIonPbPbDiphoton2023
Hlt1DecReportsDecoder_xxxxxxxx ERROR Maximum number of errors ( 'ErrorMax':1) reached.
Hlt1DecReportsDecoder_xxxxxxxx ERROR Hlt1DecReportsDecoder_e139c7d0 : No HltDecReports RawBank with sourceID 1 found
====================================================================================================
HLTControlFlowMgr FATAL Failed event detected on s: ? e: ?
```Michel De CianCarla Marin Benitocarla.marin.benito@cern.chMichel De Cianhttps://gitlab.cern.ch/lhcb/DaVinci/-/issues/179Follow-up from "Add example for use of own and unbiased PV with DTF"2024-02-29T21:02:15+01:00Sevda EsenFollow-up from "Add example for use of own and unbiased PV with DTF"Currently the proposed method to get an unbiased PV for a particle is to create a new particle using the 'ParticleUnbiasedPVAdder' algorithm. See the example here:
https://gitlab.cern.ch/lhcb/DaVinci/-/blob/sevda-v2PV-packing/DaVinciExa...Currently the proposed method to get an unbiased PV for a particle is to create a new particle using the 'ParticleUnbiasedPVAdder' algorithm. See the example here:
https://gitlab.cern.ch/lhcb/DaVinci/-/blob/sevda-v2PV-packing/DaVinciExamples/python/DaVinciExamples/tupling/option_davinci_tupling_DTF_pvs.py
FunTuple allows to create different branches for a particle using different 'DTF' refits. However, the particles with unbiased PVs are actually new particles, copies of the particles in the original list. Hence, one cannot add the 'DTF' information with the unbiased PVs to the same ntuple as eventual other DTF refits (or other branches).
The solution, chosen in the example, is to create two ntuples. These can then be linked for instance using 'TTree::AddFriend'. (I checked that this works on the output of the test.) The question is if there is a different solution which we can somehow write the information to the same TTree.
That is what started this issue, but the more I think about it, the less important it seems :-)
Adding @erodrigu and @amathad for comments.
(cc @sesen, @avilla, @sperazzi)
See original discussion in merge request: https://gitlab.cern.ch/lhcb/DaVinci/-/merge_requests/1035#note_7654412Eduardo RodriguesAbhijit MathadEduardo Rodrigueshttps://gitlab.cern.ch/lhcb/DaVinci/-/issues/178Creation of DaVinci test input files2024-03-25T22:35:37+01:00Nicole SkidmoreCreation of DaVinci test input filesWe need a comprehensive record of how the input files for DaVinci tests are created. This has caused me (and @sesen) much pain the past days and I will not be responsible for this again.
An issue is obviously not the best place for thi...We need a comprehensive record of how the input files for DaVinci tests are created. This has caused me (and @sesen) much pain the past days and I will not be responsible for this again.
An issue is obviously not the best place for this information but I put it here for now.
cc @erodrigu, @amathad
# To create a file serving the purpose of `/eos/lhcb/wg/dpa/wp3/tests/hlt2_passthrough_thor_lines.dst`
For tests including [`option_davinci_tupling_DTF_pvs.py`](https://gitlab.cern.ch/lhcb/DaVinci/-/blob/master/DaVinciExamples/python/DaVinciExamples/tupling/option_davinci_tupling_DTF_pvs.py)
## HLT2 step
* Creates HLT2 output which then goes through passthrough Sprucing for DaVinci test ...
* Created using up-to-date `master` stack on 25/02/2024 using platform `x86_64_v2-el9-gcc13+detdesc-opt`. (Note MC can only use detdesc platforms at this time)
* To run : inside `/Moore`
```./run gaudirun.py hlt2_bsjpsiphi_ppthorrealtime.py```
<details closed>
<summary>hlt2_bsjpsiphi_ppthorrealtime.py</summary>
```python
"""
Creates HLT2 output which then goes through passthrough Sprucing for DaVinci test ...
Created using up-to-date master stack on 25/02/2024 using platform `x86_64_v2-el9-gcc13+detdesc-opt`
(Note MC can only use detdesc platforms at this time)
To run : inside Moore
```./run gaudirun.py hlt2_bsjpsiphi_ppthorrealtime.py```
"""
from Moore import options, run_moore
from RecoConf.global_tools import stateProvider_with_simplified_geom, trackMasterExtrapolator_with_simplified_geom
from RecoConf.reconstruction_objects import reconstruction
from RecoConf.hlt2_global_reco import reconstruction as hlt2_reconstruction, make_fastest_reconstruction
from Hlt2Conf.lines import all_lines
import re
######## Following is for any regular pp thor job ########
# Taken from https://gitlab.cern.ch/lhcb/Moore/-/blob/master/Hlt/Hlt2Conf/options/hlt2_pp_thor.py
### explicitly remove lines that require hlt1_filter_code
def remove_lines(lines_dict, pattern_to_remove):
filtered = {
name: line
for name, line in lines_dict.items()
if re.search(pattern_to_remove, name) is None
}
return filtered
to_remove = [
'Hlt2IFT_SMOG2GECPassthrough', 'Hlt2IFT_SMOG2LumiPassthrough',
'Hlt2IFT_SMOG2MBPassthrough', 'Hlt2IFT_Femtoscopy_InclLambdaLL',
'Hlt2IFT_Femtoscopy_InclXiLLL', 'Hlt2IFT_Femtoscopy_InclOmegaLLL',
'Hlt2IFT_Femtoscopy_LambdaP', 'Hlt2IFT_Femtoscopy_LambdaP_lowK',
'Hlt2IFT_Femtoscopy_LambdaLambda', 'Hlt2IFT_Femtoscopy_LambdaLambda_lowK',
'Hlt2IFT_Femtoscopy_XiP', 'Hlt2IFT_Femtoscopy_XiP_lowK',
'Hlt2IFT_Femtoscopy_XiLambda', 'Hlt2IFT_Femtoscopy_XiLambda_lowK',
'Hlt2IFT_Femtoscopy_XiXi', 'Hlt2IFT_Femtoscopy_OmegaP',
'Hlt2IFT_Femtoscopy_OmegaP', 'Hlt2IFT_Femtoscopy_OmegaP_lowK',
'Hlt2IFT_Femtoscopy_OmegaLambda', 'Hlt2IFT_Femtoscopy_OmegaXi',
'Hlt2IFT_Femtoscopy_OmegaOmega', 'Hlt2QEE_DiElectronPrompt_PersistPhotons',
'Hlt2QEE_DiElectronPrompt_PersistPhotons_FULL',
'Hlt2QEE_DiElectronDisplaced_PersistPhotons',
'Hlt2QEE_DiElectronDisplaced_PersistPhotons_FULL',
'Hlt2QEE_DiElectronPrompt_PersistPhotonsSS',
'Hlt2QEE_DiElectronDisplaced_PersistPhotonsSS'
]
trunc_lines = all_lines
for remove in to_remove:
trunc_lines = remove_lines(trunc_lines, remove)
print("Removed lines: ", all_lines.keys() - trunc_lines.keys())
def make_lines():
return [builder() for builder in trunc_lines.values()]
public_tools = [
trackMasterExtrapolator_with_simplified_geom(),
stateProvider_with_simplified_geom()
]
######## Above is for any regular pp thor job ########
## Below is specific for the creation of DaVinci test input file ##
input_files = [
# Upgrade08 simulations of Bs2JpsiPhi avaliable in PRConfig at
# https://gitlab.cern.ch/lhcb-datapkg/PRConfig/-/blob/master/python/PRConfig/TestFileDB.py#L11029
'/eos/lhcb/grid/prod/lhcb/MC/Upgrade/XDIGI/00122720/0000/00122720_00000002_1.xdigi'
]
## Needed as these are 2020 MC samples
from RecoConf.decoders import default_ft_decoding_version
default_ft_decoding_version.global_bind(value=2)
from RecoConf.hlt1_muonid import make_muon_hits
make_muon_hits.global_bind(geometry_version=2)
options.input_files = input_files
options.input_type = 'ROOT'
options.input_raw_format = 0.5
options.evt_max = -1
options.simulation = True
options.data_type = 'Upgrade'
options.dddb_tag = 'dddb-20201211'
options.conddb_tag = 'sim-20201218-vc-mu100'
options.output_manifest_file = "hlt2_bsjpsiphi_ppthorrealtime.tck.json"
options.scheduler_legacy_mode = False
options.output_file = "hlt2_bsjpsiphi_ppthorrealtime.mdf"
options.output_type = "MDF"
with reconstruction.bind(from_file=False),\
hlt2_reconstruction.bind(make_reconstruction=make_fastest_reconstruction):
config = run_moore(options, make_lines, public_tools)
```
</details>
* Log file here `/eos/lhcb/wg/dpa/wp3/tests/logs/hlt2_bsjpsiphi_ppthorrealtime.log`
* Have 145 events passing `Hlt2B2CC_BsToJpsiPhi_Detached` Hlt2 line
```
LAZY_AND: Hlt2B2CC_BsToJpsiPhi_DetachedDecisionWithOutput #=1000 Sum=145
```
* MDF output and json also at `/eos/lhcb/wg/dpa/wp3/tests/logs/` for now
## Sprucing step
* Creates Turbo (passthrough) Spruce output for DaVinci tests. Runs on top of the HLT2 file created above
* Created using up-to-date `master` stack on 25/02/2024 using platform `x86_64_v2-el9-gcc13+detdesc-opt`.
* Use `lbexec` to run : inside `/Moore`
```
./run lbexec Hlt2Conf.Sprucing_tests:spruce_passthrough spruce.yaml
```
where the `yaml` is
<details closed>
<summary>spruce_bsjpsiphi_ppthorrealtime.yaml</summary>
```yaml
process: Spruce
input_raw_format : 0.5
input_type : 'RAW'
input_files : [
'hlt2_bsjpsiphi_ppthorrealtime.mdf'
]
input_manifest_file : "hlt2_bsjpsiphi_ppthorrealtime.tck.json"
simulation : True
data_type : 'Upgrade'
conddb_tag: "sim-20201218-vc-mu100"
dddb_tag: "dddb-20201211"
output_file : 'spruce_bsjpsiphi_passthrough.{stream}.dst'
output_type : 'ROOT'
output_manifest_file : "spruce_bsjpsiphi_passthrough.tck.json"
input_process: Hlt2
```
</details>
* Log file here `/eos/lhcb/wg/dpa/wp3/tests/logs/spruce_bsjpsiphi_ppthorrealtime.log`
* Note the output dst is called `spruce_bsjpsiphi_passthrough..dst` as `stream` is `None`
* Inspecting the dst I see (using rawbanks from `/Event/default/RawEvent`)
```
>>> countevents_decision("Hlt2B2CC_BsToJpsiPhi_Detached")
Hlt2B2CC_BsToJpsiPhi_Detached fired 145 times
```
* DST output and json also at `/eos/lhcb/wg/dpa/wp3/tests/logs/` for now
* This should be sufficient for the testsNicole SkidmoreNicole Skidmorehttps://gitlab.cern.ch/lhcb/DaVinci/-/issues/177Add a DV test related to neutral truth ID2024-03-13T11:50:26+01:00Abhijit MathadAdd a DV test related to neutral truth IDFollowing from this discussion [here](https://gitlab.cern.ch/lhcb/LHCb/-/merge_requests/4453#note_7637657). We need to add a DV test for neutral truth ID.Following from this discussion [here](https://gitlab.cern.ch/lhcb/LHCb/-/merge_requests/4453#note_7637657). We need to add a DV test for neutral truth ID.Yingrui HouJiahui ZhuoYingrui Houhttps://gitlab.cern.ch/lhcb/DaVinci/-/issues/175Some ideas to address the high memory usage in FunTuple2024-02-22T20:43:02+01:00Jiahui ZhuoSome ideas to address the high memory usage in FunTupleFollowing the discussion in [DPA-WP3](https://indico.cern.ch/event/1374470/), I am creating this issue.
High memory usage in FunTuple was first observed at https://gitlab.cern.ch/lhcb/Rec/-/issues/365. When a user creates a very large F...Following the discussion in [DPA-WP3](https://indico.cern.ch/event/1374470/), I am creating this issue.
High memory usage in FunTuple was first observed at https://gitlab.cern.ch/lhcb/Rec/-/issues/365. When a user creates a very large FunTuple, memory usage can rise to over 7 GB. This was initially suspected to be a memory leak at DTF.
After addressing the memory leak at DTF, the issue of high memory usage persisted in large FunTuple production. It was then identified as an issue with Functor JIT compilation, as detailed at https://gitlab.cern.ch/lhcb/LHCb/-/issues/299#note_6645917.
Although the idea behind FunTuple is to tuple only the variables used for analysis, in practice, users often tuple all possible values. This leads to the production of more than 10 tuples with over 500 branches in a single DaVinci job (5k in total). Since each branch is an independent functor in FunTuple, it is reasonable to expect significant resources to be used in JIT to compile all the required functors (on the order of 10^3).
Ideally, during JIT compilation, repeated functors should only be compiled once. This caching mechanism relies on the functor's hash code, computed through the `str` expression of the functor. If two functors have different hash codes, they both need to be compiled, otherwise, they are treated as the same functor. Thus, we do not need to compile on the order of 10^3 functors if most of them are repeated.
However, certain functors that contain hard-coded properties, such as `F.TES(<LOCATION>)` or `F.VALUE_FROM_DICT(<KEY>)`, will disrupt this cache. Different `<LOCATION>` and `<KEY>` values lead to different `str` expressions, so the composite functors must be compiled repeatedly.
One proposed solution to this problem is to create the JIT functor cache before submitting the job, uploading it to CVMFS or directly to the grid so that it can be used directly during the DaVinci job without recompiling functors. However, this approach is well-known to be difficult to implement because not all users have access to upload cache files into CVMFS, and packaging the compiled cache file into a grid job is not trivial.
A more direct solution, proposed by this issue, is to try to avoid the situation by reducing the number of uncachable functors in DaVinci jobs. The following ideas are proposed:
1. Avoid the use of functors that alter the cache hash, such as `F.VALUE_FROM_DICT`, and instead use FunTuple to handle more complex output types like `std::map<string, ...>`. For instance, in the case of TIS/TOS, this would significantly reduce the number of functors we need to compile by compiling a single functor for all selection lines, instead of one for each. The same logic can be applied to other functors, such as using `F.P` instead of three different functors (`F.PX`, `F.PY`, `F.PZ`).
2. Extend FunTuple to support extra input for configuring a functor. For example, we could create a new functor `F.APPLY_TES` that receives a `str` TES location as input and returns a `F.TES(CORRESPOND_LOCATION)`. This way, we could do the following:
```
{
"ANY_RESULT1": (F.ANYFUNCTOR @ F.APPLY_TES, LOCATION_1),
"ANY_RESULT2": (F.ANYFUNCTOR @ F.APPLY_TES, LOCATION_2)
}
```
In the FunTuple backend, the functor `Func = F.ANYFUNCTOR @ F.APPLY_TES` would be registered by the Functor Factory as a single functor.
For more complex situations:
**case 1**:
```
# traditionally
A = FUNC.bind(TES(<location>), FORWARDARGS)
# then in cpp we call
# A(LHCb::Particle)
# newly
B = FUNC.bind(APPLY_TES, APPLY(FORWARDARGS))
# then in cpp we call
# B(<location>)(LHCb::Particle)
```
**case 2**:
```
# traditionally
C = FUNC.bind(TES(<location2>), FUNC.bind(TES(<location1>), FORWARDARGS))
# then in cpp we call
# C(LHCb::Particle)
# newly
D = FUNC.bind(APPLY_TES, APPLY(FUNC1.bind(APPLY_TES, APPLY(FORWARDARGS))))
# then in cpp we call
# D(<location1>)(<location2>)(LHCb::Particle)
```
Where `APPLY(<AnyFunctor>)` is a Functor that receives anything as input and returns `<AnyFunctor>`
Using these types of functors that return a new functor, we should be able to use any extra inputs to configure very complex composite functors. We can use Python classes to hide these mechanics from users, such as automatically handle and replace `F.TES` with `F.APPLY_TES`
I am opening this issue to discuss these new ideas. Please note that such implementations can be designed to affect FunTuple exclusively, without affecting HLT lines, making this a topic for DPA discussion.
FYI: @amathad @erodrigu @ahennequ
Please notify more individuals who might be interested in this discussion.https://gitlab.cern.ch/lhcb/DaVinci/-/issues/174Tuple values of BPVLTIME do not always correspond to online values2024-02-05T19:23:46+01:00Laurent DufourTuple values of BPVLTIME do not always correspond to online valuesIn the 2023 data, we ran with a requirement of `BPVLTIME > 0.2ps` ( https://gitlab.cern.ch/lhcb/Moore/-/blob/master/Hlt/Hlt2Conf/python/Hlt2Conf/lines/charm/d_to_hhh.py#L136 ). However, offline, we see a small portion (2%) of entries wit...In the 2023 data, we ran with a requirement of `BPVLTIME > 0.2ps` ( https://gitlab.cern.ch/lhcb/Moore/-/blob/master/Hlt/Hlt2Conf/python/Hlt2Conf/lines/charm/d_to_hhh.py#L136 ). However, offline, we see a small portion (2%) of entries with a `nan` value for the BPVLTIME.
This also happens when there is only 1 PV present in the event, which excludes the IP <-> IPCHI2 association as the culprit.
The resolution of the reconstruction does look degraded, but it's clear this is also happening to signal.
The tuples are produced as part of the analysis production: https://lhcb-analysis-productions.web.cern.ch/productions/?wg=charm&analysis=charmprodasym
and part of the Ds -> K K pi selectionhttps://gitlab.cern.ch/lhcb/DaVinci/-/issues/173Towards 100% unit test coverage2024-02-01T16:41:34+01:00Eduardo RodriguesTowards 100% unit test coverageA lot of the Python code is effectively tested via all that is in DaVinciTests and DaVinciExamples. Those are actual jobs.
Speaking of unit tests per se, here is the current situation:
```
-- Docs: https://docs.pytest.org/en/stable/how-...A lot of the Python code is effectively tested via all that is in DaVinciTests and DaVinciExamples. Those are actual jobs.
Speaking of unit tests per se, here is the current situation:
```
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
---------- coverage: platform linux, python 3.9.12-final-0 -----------
Name Stmts Miss Cover Missing
-------------------------------------------------------------------------
Phys/DaVinci/python/DaVinci/LbExec.py 48 10 79% 115-133
Phys/DaVinci/python/DaVinci/algorithms.py 60 15 75% 46, 74-89, 111-116, 127, 161-162
Phys/DaVinci/python/DaVinci/config.py 65 28 57% 101-119, 157-162, 186-222, 227
-------------------------------------------------------------------------
TOTAL 173 53 69%
```
Ideally the unit tests would achieve 100% coverage.https://gitlab.cern.ch/lhcb/DaVinci/-/issues/172Clean-up DV tests for the usage of reconstrunction objects2024-03-27T16:00:58+01:00Abhijit MathadClean-up DV tests for the usage of reconstrunction objects@sesen pointed that the DV tests are using a mix of `make_*` and `get_` functions and it is hard to maintain.@sesen pointed that the DV tests are using a mix of `make_*` and `get_` functions and it is hard to maintain.https://gitlab.cern.ch/lhcb/DaVinci/-/issues/170Add a test/example of building composite particles in DaVinci2024-01-15T16:07:50+01:00Laurent DufourAdd a test/example of building composite particles in DaVinciWhile a common use-case in Run 2, there currently is not yet an example showing how people can construct new composites with DaVinci and in the Run-3 data.
This is probably an important example to have.While a common use-case in Run 2, there currently is not yet an example showing how people can construct new composites with DaVinci and in the Run-3 data.
This is probably an important example to have.https://gitlab.cern.ch/lhcb/DaVinci/-/issues/167Adapt MCTruthAndBkgCat.py to use relations tables given as input2023-12-21T10:46:28+01:00Abhijit MathadAdapt MCTruthAndBkgCat.py to use relations tables given as inputThe MR that has been inactive for quite some time need to be addressed. We plan to retire Analysis, therefore, I am creating this issue and closing the inactive MR. The changes in the MR included:
Changing this
```python
relat...The MR that has been inactive for quite some time need to be addressed. We plan to retire Analysis, therefore, I am creating this issue and closing the inactive MR. The changes in the MR included:
Changing this
```python
relations = [
get_pp2mcp_relations(prefix(rel, root_in_tes))
for rel in relations_locs
]
relations = []
```
to this:
```python
if not isinstance(relations_locs[0], str):
relations = relations_locs
else:
relations = [
get_pp2mcp_relations(prefix(rel, root_in_tes))
for rel in relations_locs
]
```
FYI: @pherrero and @sstahlSascha StahlPaula Herrero GasconSascha Stahlhttps://gitlab.cern.ch/lhcb/DaVinci/-/issues/157[HltEfficiencyChecker]: Better error messages when annotated decay descriptor...2023-12-14T17:35:03+01:00Ross John Hunter[HltEfficiencyChecker]: Better error messages when annotated decay descriptor isnt specified properly.*Note: HltEfficiencyChecker is being moved from MooreAnalysis to DaVinci in !1005. This issue only becomes relevant to DaVinci once that is merged*
Moved from MooreAnalysis#22. To be addressed soon after merge of !1005.
Reported by @mk...*Note: HltEfficiencyChecker is being moved from MooreAnalysis to DaVinci in !1005. This issue only becomes relevant to DaVinci once that is merged*
Moved from MooreAnalysis#22. To be addressed soon after merge of !1005.
Reported by @mkenzie - many thanks.
If you don't add the annotations you'll get
```
File "/home/epp/phskbk/Scratch/lhcb/LHCbStack/stack/Moore/Hlt/Hlt2Conf/options/bnoc/eff_Bs2KpiKpi.py", line 231, in <module>
run_moore_with_tuples(
File "/home/epp/phskbk/Scratch/lhcb/LHCbStack/stack/MooreAnalysis/HltEfficiencyChecker/python/HltEfficiencyChecker/config.py", line 180, in run_moore_with_tuples
top_cf_node = add_efficiency_tuples(
File "/home/epp/phskbk/Scratch/lhcb/LHCbStack/stack/MooreAnalysis/HltEfficiencyChecker/python/HltEfficiencyChecker/config.py", line 279, in add_efficiency_tuples
decay, branches, _ = parse_descriptor_template(descriptor_template)
File "/home/epp/phskbk/Scratch/lhcb/LHCbStack/stack/MooreAnalysis/HltEfficiencyChecker/python/HltEfficiencyChecker/config.py", line 328, in parse_descriptor_template
parent = particles[
IndexError: list index out of range
```
which isn't super helpful. Also, if you place the `${name}` annotations inside the brackets rather than outside, you also won't get a helpful message.
These error messages should be improved.Ross John HunterRoss John Hunterhttps://gitlab.cern.ch/lhcb/DaVinci/-/issues/156Create a CONTRIBUTING.md file2023-12-04T18:48:26+01:00Eduardo RodriguesCreate a CONTRIBUTING.md fileIndeed we can then point people to it for how to contribute code and follow basic guidelines. (Cf. one of comments on today's talk on "guidelines for MRs" at the C&SW parallel session of the LHCb week.)Indeed we can then point people to it for how to contribute code and follow basic guidelines. (Cf. one of comments on today's talk on "guidelines for MRs" at the C&SW parallel session of the LHCb week.)https://gitlab.cern.ch/lhcb/DaVinci/-/issues/155High failure rate and incorrect results from Run3 DTF with PV constraint (and...2024-03-14T10:07:56+01:00Laurent DufourHigh failure rate and incorrect results from Run3 DTF with PV constraint (and cannot check the status)We're using the DecayTreeFitter to infer the kinematics of a track from a V0 decay, as we did in Run 1/2. The decay is built from a VELO track and a long track, so there is plenty of freedom for the fit. We can also do our own kinematic ...We're using the DecayTreeFitter to infer the kinematics of a track from a V0 decay, as we did in Run 1/2. The decay is built from a VELO track and a long track, so there is plenty of freedom for the fit. We can also do our own kinematic constraint, for which the results look like they make sense. However, from the DTF we get a very poor resolution, and see many, many fit failures. The degree of failure seem to indicate something is going wrong...
Let me make the remark that I couldn't find a way to actually see the (1) CHI2 and (2) Status of the DTF. These are very important to have access to!
An example options file that immediately reproduces this would be:
```
from DaVinci import Options, make_config
from PyConf.reading import get_particles, get_pvs, get_pvs_v1
from FunTuple import FunctorCollection as FC
from FunTuple import FunTuple_Particles as Funtuple
from FunTuple.functorcollections import SelectionInfo
from DaVinci.algorithms import create_lines_filter
from PyConf.Algorithms import FlattenDecayTree, VeloIDOverlapRelationTable
from RecoConf.reconstruction_objects import reconstruction
from Hlt2Conf.standard_particles import (make_long_pions)
from Hlt2Conf.algorithms_thor import ParticleFilter
import Functors as F
def make_tuple(v2_pvs,line):
fields = {
"KS0": "KS0 -> [pi+]CC [mu-]CC",
"pip": "KS0 -> ^[pi+]CC [mu-]CC",
"pim": "KS0 -> [pi+]CC ^[mu-]CC",
}
input_data=get_particles("/Event/HLT2/Hlt2KshortVeloLong{}/Particles".format(line))
from DecayTreeFitter import DecayTreeFitter
DTF = DecayTreeFitter(
name = 'DTF_{hash}',
input_particles = input_data,
input_pvs = v2_pvs
)
DTF_collection_daughters = (FC( {"DTF_PX": DTF(F.PX),
"DTF_PY": DTF(F.PY),
"DTF_PZ": DTF(F.PZ),
"DTF_MASS": DTF.MASS,
"DTF_PT": DTF(F.PT)}))
hlt2_line = ['Hlt2KshortVeloLong{}'.format(line)]
selinfo = SelectionInfo(selection_type="Hlt2", trigger_lines=hlt2_line)
with reconstruction.bind(from_file=True, spruce=True):
long_pions=make_long_pions()
flattened_decay_tree = FlattenDecayTree( InputParticles=input_data )
basic_particles = ParticleFilter( flattened_decay_tree.OutputParticles, F.FILTER( F.ISBASICPARTICLE ), name="MyParticleFilterThing_{hash}" )
relation_table_match_to_long = VeloIDOverlapRelationTable(MatchFrom=basic_particles, MatchTo=long_pions)
matching_variables = FC( {'MATCHLONG_P[nMatchTracks]': F.REVERSE_RANGE @ F.MAP_INPUT_ARRAY(Functor=F.P, Relations=relation_table_match_to_long.OutputRelations),
"Velo2LongMatch_WEIGHT[nMatchTracks]": F.REVERSE_RANGE @ F.MAP_WEIGHT(Relations=relation_table_match_to_long.OutputRelations),
"MATCHLONG_ID[nMatchTracks]": F.REVERSE_RANGE @ F.MAP_INPUT_ARRAY(Functor= F.PARTICLE_ID, Relations=relation_table_match_to_long.OutputRelations),
"MATCHLONG_PID_P[nMatchTracks]": F.REVERSE_RANGE @ F.MAP_INPUT_ARRAY(Functor=F.PID_P, Relations=relation_table_match_to_long.OutputRelations),
"MATCHLONG_PT[nMatchTracks]": F.REVERSE_RANGE @ F.MAP_INPUT_ARRAY(Functor=F.PT, Relations=relation_table_match_to_long.OutputRelations)} )
variables = {
"KS0": DTF_collection_daughters,
"pip" : DTF_collection_daughters,
"pim" : DTF_collection_daughters + matching_variables,
}
tuple = Funtuple(
name="DecayTreeTuple{}".format(line),
tuple_name="DecayTree",
fields=fields,
variables=variables,
event_variables= selinfo,
inputs=input_data
)
my_filter = create_lines_filter(name="HDRFilter_Hlt2KshortVeloLong{}".format(line), lines=["Hlt2KshortVeloLong{}".format(line)])
return [my_filter,tuple]
def main(options: Options):
v2_pvs = get_pvs()
name="Hlt2KshortVeloLong"
lines=["","_15","_20","_high"]
tuples={}
for l in lines:
tuples[name+l]=make_tuple(v2_pvs,l)
return make_config(options, tuples)
from DaVinci import Options
options = Options(
data_type="Upgrade",
input_type="ROOT",
simulation=False,
write_fsr=False,
ntuple_file="DVTuples.root",
input_files=["/eos/lhcb/grid/prod/lhcb/LHCb/Collision23/TRACKEFF.DST/00198132/0000/00198132_00000516_1.trackeff.dst"],
geometry_version="run3/trunk",
conditions_version="master",
python_logging_level="3",
input_raw_format=0.5,
evt_max=100000,
print_freq="10000",
input_process="TurboPass",
input_stream= "trackeff",
)
with options.apply_binds():
main(options)
```
@pherreroAbhijit MathadAbhijit Mathadhttps://gitlab.cern.ch/lhcb/DaVinci/-/issues/154Error when trying to create variables with different DTF constraints using th...2023-11-22T11:43:22+01:00Michele AtzeniError when trying to create variables with different DTF constraints using the same particle substitutions@amathad @jzhuo When trying to create variables with different DTF constraints using the same particle substitutions, the following error is thrown:
```plaintext
PyConf.utilities.ConfigurationError: Not allowed to instantiate Substitute...@amathad @jzhuo When trying to create variables with different DTF constraints using the same particle substitutions, the following error is thrown:
```plaintext
PyConf.utilities.ConfigurationError: Not allowed to instantiate SubstitutePIDAlg_Particles multiple times with different names, PIDSubstitution_DTF_B0 versus ['PIDSubstitution_DTF_Jpsi'], but otherwise identical properties.
```
This error is obtained doing the following:
```plaintext
lb-run -c x86_64_v2-centos7-gcc12+detdesc-opt DaVinci/v63r14 bash --norc --noprofile
lbexec failing_example_DTF_PID_substitution:main $DAVINCIEXAMPLESROOT/example_data/test_passthrough_thor_lines.yaml
```
The script can be found in [here](https://gitlab.cern.ch/-/snippets/2904)Abhijit MathadJiahui ZhuoAbhijit Mathadhttps://gitlab.cern.ch/lhcb/DaVinci/-/issues/151Strange error on hitting the vector size limit in analysis production2023-10-24T17:17:07+02:00Abhijit MathadStrange error on hitting the vector size limit in analysis productionMoving from [issue](https://gitlab.cern.ch/lhcb-datapkg/AnalysisProductions/-/issues/56#note_7231482) to here.
Problematic functors:
```
"Tr_T_TRUEPRIMARYVERTEX_X": F.VALUE_OR(-1000) @ F.MAP_INPUT_ARRAY(Functor = F.VALUE_OR(-1000) @ M...Moving from [issue](https://gitlab.cern.ch/lhcb-datapkg/AnalysisProductions/-/issues/56#note_7231482) to here.
Problematic functors:
```
"Tr_T_TRUEPRIMARYVERTEX_X": F.VALUE_OR(-1000) @ F.MAP_INPUT_ARRAY(Functor = F.VALUE_OR(-1000) @ MCTRUTH_TAG(F.MC_PV_VX), Relations = tagAlg_rels),
"Tr_T_TRUEPRIMARYVERTEX_Y": F.VALUE_OR(-1000) @ F.MAP_INPUT_ARRAY(Functor = F.VALUE_OR(-1000) @ MCTRUTH_TAG(F.MC_PV_VY), Relations = tagAlg_rels),
```
@leuecker :
> Removing the first of the `F.VALUE_OR(-1000)` actually **fixes the issue**.
@amathad:
> It is very strange to me that `max_size` (that dictates array size) error is fixed by removing something completely unrelated the `VALUE_OR` :confused:
The files to reproduce the error is [here](https://gitlab.cern.ch/leuecker/mre_davinci_max_vec_size) (Thanks @leuecker for this)Abhijit MathadAbhijit Mathadhttps://gitlab.cern.ch/lhcb/DaVinci/-/issues/168Review content of functor collections2024-03-27T18:45:59+01:00Eduardo RodriguesReview content of functor collectionsWe should ensure that the functor collections (https://gitlab.cern.ch/lhcb/Analysis/-/blob/master/Phys/FunTuple/python/FunTuple/functorcollections.py) do not content redundant information. We should also define which variables are a must...We should ensure that the functor collections (https://gitlab.cern.ch/lhcb/Analysis/-/blob/master/Phys/FunTuple/python/FunTuple/functorcollections.py) do not content redundant information. We should also define which variables are a must and which can be optional; it is certainly the case that some collections already have a distinction, via optional argument.
Example: the `Kinematics` collection contains px, py and pt, where the latter is clearly redundant.Eduardo RodriguesAbhijit MathadAnfeng LiEduardo Rodrigueshttps://gitlab.cern.ch/lhcb/DaVinci/-/issues/145Automatically configuring for Analysis Productions2024-03-28T14:02:45+01:00Chris BurrAutomatically configuring for Analysis ProductionsWhen submitting Run 1 and Run 2 Analysis Productions most people take advantage of the `automatically_configure: true` option.
This removes the need for people to handle information relating to input data and makes people's options files...When submitting Run 1 and Run 2 Analysis Productions most people take advantage of the `automatically_configure: true` option.
This removes the need for people to handle information relating to input data and makes people's options files much more consise.
It works by generating a Python file like this one which is prepended to the list of options files provided by the user:
<details><summary>Click to see examples</summary>
```python
from Configurables import DaVinci
try:
DaVinci().Turbo = False
except AttributeError:
# Older DaVinci versions don't support Turbo at all
pass
DaVinci().RootInTES = '/Event/Bhadron'
DaVinci().InputType = 'MDST'
DaVinci().DataType = '2015'
DaVinci().Simulation = False
DaVinci().Lumi = True
from Configurables import CondDB
CondDB(LatestGlobalTagByDataType=DaVinci().DataType)
```
```python
from Configurables import DaVinci
try:
DaVinci().Turbo = False
except AttributeError:
# Older DaVinci versions don't support Turbo at all
pass
DaVinci().RootInTES = '/Event/AllStreams'
DaVinci().InputType = 'MDST'
DaVinci().DataType = '2018'
DaVinci().Simulation = True
DaVinci().Lumi = False
DaVinci().DDDBtag = 'dddb-20170721-3'
DaVinci().CondDBtag = 'sim-20190430-vc-mu100'
```
</details>
For Run 3 this option hasn't yet been implemented as it wasn't clear what should be done however we're now approching a state where it might start to be reasonable.
The mechanism for doing this in Run 3 was designed into `lbexec` and the `extra_options` parameter that people have been using when submitting productions. For example:
```yaml
extra_options:
input_type: ROOT
input_process: "TurboPass"
input_stream: "bandq"
input_raw_format: 0.5
simulation: False
data_type: "Upgrade"
geometry_version: trunk
conditions_version: master
```
I'm opening this issue to start the discussion around what we need to implement to be able to make this work robustly for 2024.
In general I think we should favour setting the defaults in the physics application itself if we can (possibly via the new FSRs).Davide FazziniDavide Fazzini