MooreAnalysis issueshttps://gitlab.cern.ch/lhcb/MooreAnalysis/-/issues2020-06-09T17:07:05+02:00https://gitlab.cern.ch/lhcb/MooreAnalysis/-/issues/1Add ability to deal with HLT2 lines to HltEfficiencyChecker2020-06-09T17:07:05+02:00Ross John HunterAdd ability to deal with HLT2 lines to HltEfficiencyCheckerRoss John HunterRoss John Hunterhttps://gitlab.cern.ch/lhcb/MooreAnalysis/-/issues/2Follow-up from "Add HltEfficiencyChecker tool to mooreanalysis" - behaviour o...2022-05-20T11:13:03+02:00Ross John HunterFollow-up from "Add HltEfficiencyChecker tool to mooreanalysis" - behaviour of the routing bits writer/ Z -> mu mu old seg faults.The following discussion from !1 should be addressed:
- [ ] @rjhunter started a [discussion](https://gitlab.cern.ch/lhcb/mooreanalysis/merge_requests/1#note_3109874): (+18 comments)
> @olupton and I made some progress on assessing...The following discussion from !1 should be addressed:
- [ ] @rjhunter started a [discussion](https://gitlab.cern.ch/lhcb/mooreanalysis/merge_requests/1#note_3109874): (+18 comments)
> @olupton and I made some progress on assessing where the $`Z \rightarrow \mu\mu`$ issue is coming from.
>
> First of all, the control flow of this tool is like so:
> ```python
> trigger_node = CompositeNode(
> 'trigger_node',
> combineLogic=NodeLogic.NONLAZY_AND,
> children=[
> top_cf_node, et,
> #top_cf_node,
> mc_unpackers()["MCParticles"],
> mc_unpackers()["MCVertices"], mcdtt
> ],
> forceOrder=True)
> ```
>
> If `et` (an instance of an `EventTuple` algorithm) is included, all decay modes work except $`Z \rightarrow \mu\mu`$, which seg faults as `TupleToolTrigger` is trying to write the tree. If `et` is not included, all decays would fail at the 2nd, perhaps 5th event, complaining that they couldn't find the DecReports. This was strange since the code that allows them to find the DecReports was the first thing we did when we started writing this tool.
>
> The fact that it wasn't straight away the first event lead Olli to wonder if the GEC was involved, and indeed we saw that for 1 event the GEC was throwing away, the `ExecutionReportsWriter` wasn't called, and therefore there were no DecReports for that event. We verified this by asking the `NoBiasLine` to accept everything, and amazingly everything worked for all decays.
>
> We then also saw that everything works now, for all decays, if `et` is not in the control flow as well.
>
> But this begs the question, if `MCDecayTreeTuple` was properly declaring its dependencies to the `ExecutionReportsWriter`, then shouldn't `ExecutionReportsWriter` be called for every event? So we looked in the debug output of the `HLTControlFlowMgr` and saw that indeed, `MCDecayTreeTuple` is not declaring any dependencies, whereas `EventTuple` is declaring its dependency to `ExecutionReportsWriter`.
>
> So the question is now, why doesn't `MCDecayTreeTuple` declare its dependencies properly, but `EventTuple` does? They seem to both inherit from `GaudiTupleAlg` in the end. I don't know enough about how the scheduler works to be able to find out where it works out the dependencies. @rmatev @nnolte any ideas how I can look for this?https://gitlab.cern.ch/lhcb/MooreAnalysis/-/issues/3Follow-up from "Add HltEfficiencyChecker tool to mooreanalysis" - check behav...2020-02-28T12:25:50+01:00Ross John HunterFollow-up from "Add HltEfficiencyChecker tool to mooreanalysis" - check behaviour of --dumpThe following discussion from !1 should be addressed:
- [ ] @dovombru started a [discussion](https://gitlab.cern.ch/lhcb/mooreanalysis/merge_requests/1#note_3142645): (+2 comments)
> While testing the scripts here, I noticed that ...The following discussion from !1 should be addressed:
- [ ] @dovombru started a [discussion](https://gitlab.cern.ch/lhcb/mooreanalysis/merge_requests/1#note_3142645): (+2 comments)
> While testing the scripts here, I noticed that the instructions in the MR here are not up to date. The command documented in `HltEfficiencyChecker.py` works though. Maybe make clear that the `--dump` option also needs the other arguments as input to work.https://gitlab.cern.ch/lhcb/MooreAnalysis/-/issues/4Get the pipeline in MooreAnalysis working.2020-02-22T18:57:47+01:00Ross John HunterGet the pipeline in MooreAnalysis working.See title.See title.https://gitlab.cern.ch/lhcb/MooreAnalysis/-/issues/5Fix build warning and make sure tests are run in the builds2020-06-09T17:16:48+02:00Ross John HunterFix build warning and make sure tests are run in the buildsSee this [warning](http://lhcb-nightlies.web.cern.ch/logs/build/nightly/lhcb-head/2507/x86_64-centos7-gcc9-opt/MooreAnalysis/#show_post-install/post-install_18) and the lack of testing of MooreAnalysis in any build. The test exists but f...See this [warning](http://lhcb-nightlies.web.cern.ch/logs/build/nightly/lhcb-head/2507/x86_64-centos7-gcc9-opt/MooreAnalysis/#show_post-install/post-install_18) and the lack of testing of MooreAnalysis in any build. The test exists but for some reason isn't being called.Ross John HunterRoss John Hunterhttps://gitlab.cern.ch/lhcb/MooreAnalysis/-/issues/6Change top level directory name from 'mooreanalysis' -> 'MooreAnalysis' to al...2020-03-30T11:56:47+02:00Ross John HunterChange top level directory name from 'mooreanalysis' -> 'MooreAnalysis' to align with other projectsRoss John HunterRoss John Hunterhttps://gitlab.cern.ch/lhcb/MooreAnalysis/-/issues/7Investigate using LokiMC to give you more flexibility in what to plot against...2023-06-17T16:45:11+02:00Ross John HunterInvestigate using LokiMC to give you more flexibility in what to plot against in HltEfficiencyCheckerThis has been suggested a couple of times. Put here so I don't forget.This has been suggested a couple of times. Put here so I don't forget.Ross John HunterRoss John Hunterhttps://gitlab.cern.ch/lhcb/MooreAnalysis/-/issues/8Documentation for HltEfficiencyChecker2020-07-02T11:30:02+02:00Ross John HunterDocumentation for HltEfficiencyCheckerRoss John HunterRoss John Hunterhttps://gitlab.cern.ch/lhcb/MooreAnalysis/-/issues/9Add test for rate calculation with HltEfficiencyChecker2020-06-09T17:07:05+02:00Ross John HunterAdd test for rate calculation with HltEfficiencyCheckerAs titled. Currently nothing is testing this.As titled. Currently nothing is testing this.Ross John HunterRoss John Hunterhttps://gitlab.cern.ch/lhcb/MooreAnalysis/-/issues/10Add rate correlation matrices to HltEfficiencyChecker2023-04-03T11:59:10+02:00Ross John HunterAdd rate correlation matrices to HltEfficiencyCheckerThe following discussion from !8 should be addressed:
- [ ] @rjhunter started a [discussion](https://gitlab.cern.ch/lhcb/MooreAnalysis/merge_requests/8#note_3258470): (+7 comments)
Alex suggested to make some correlation matrices....The following discussion from !8 should be addressed:
- [ ] @rjhunter started a [discussion](https://gitlab.cern.ch/lhcb/MooreAnalysis/merge_requests/8#note_3258470): (+7 comments)
Alex suggested to make some correlation matrices. This is a good idea for line building.Ross John HunterRoss John Hunterhttps://gitlab.cern.ch/lhcb/MooreAnalysis/-/issues/11lhcbStyle.C - where should this live?2020-06-09T17:07:05+02:00Ross John HunterlhcbStyle.C - where should this live?As titled. The lhcbStyle is an easy way to make plots look nice, but where should it live in the repository?As titled. The lhcbStyle is an easy way to make plots look nice, but where should it live in the repository?Ross John HunterRoss John Hunterhttps://gitlab.cern.ch/lhcb/MooreAnalysis/-/issues/12Infer HLT level (stage) automatically for analysis scripts2021-11-01T22:48:16+01:00Rosen MatevInfer HLT level (stage) automatically for analysis scriptsRoss John HunterRoss John Hunterhttps://gitlab.cern.ch/lhcb/MooreAnalysis/-/issues/13Run MuonID by default and disable for multi threaded tests2020-08-03T23:35:36+02:00Sascha StahlRun MuonID by default and disable for multi threaded testsThe following discussion from !17 should be addressed:
- [ ] @sstahl started a [discussion](https://gitlab.cern.ch/lhcb/MooreAnalysis/-/merge_requests/17#note_3607631): (+3 comments)
> I am wondering if we should enable muonid by ...The following discussion from !17 should be addressed:
- [ ] @sstahl started a [discussion](https://gitlab.cern.ch/lhcb/MooreAnalysis/-/merge_requests/17#note_3607631): (+3 comments)
> I am wondering if we should enable muonid by default and disable it for the multi-threaded tests explicitly. The selection part is not thread-safe as well.https://gitlab.cern.ch/lhcb/MooreAnalysis/-/issues/14Tests in HltEfficiencyChecker often time out2020-11-17T11:31:39+01:00Andre GuntherTests in HltEfficiencyChecker often time outTests in the HltEfficiencyChecker often have an unexpected timeout. See for example [here](https://lhcb-nightlies.web.cern.ch/logs/tests/nightly/lhcb-head-2/11/x86_64-centos7-clang8-dbg/MooreAnalysis/#all_0).
@rmatev @dovombruTests in the HltEfficiencyChecker often have an unexpected timeout. See for example [here](https://lhcb-nightlies.web.cern.ch/logs/tests/nightly/lhcb-head-2/11/x86_64-centos7-clang8-dbg/MooreAnalysis/#all_0).
@rmatev @dovombruRoss John HunterRoss John Hunterhttps://gitlab.cern.ch/lhcb/MooreAnalysis/-/issues/15Canvas windows popping up while running tests2021-03-10T19:54:46+01:00Rosen MatevCanvas windows popping up while running testsIf I run the tests with display forwarding, a lot of X windows are popping up (I guess while the analysis scripts run).
We should fix this somehow.If I run the tests with display forwarding, a lot of X windows are popping up (I guess while the analysis scripts run).
We should fix this somehow.https://gitlab.cern.ch/lhcb/MooreAnalysis/-/issues/16HltEfficiencyChecker: yaml parsing of rates groups doesn't work2023-12-14T17:28:17+01:00Ross John HunterHltEfficiencyChecker: yaml parsing of rates groups doesn't workDiscovered by @lpica - many thanks!
If you try a line like:
```yaml
rates_groups: Muon:Hlt1TrackMuonMVALineDecision,Hlt1SingleHighPtMuonLineDecision MVA:Hlt1TwoTrackMVALineDecision,Hlt1TrackMVALineDecision
```
The yaml translation to...Discovered by @lpica - many thanks!
If you try a line like:
```yaml
rates_groups: Muon:Hlt1TrackMuonMVALineDecision,Hlt1SingleHighPtMuonLineDecision MVA:Hlt1TwoTrackMVALineDecision,Hlt1TrackMVALineDecision
```
The yaml translation to command line args won't handle this properly. You'll get `'--rates-groups=Muon:Hlt1TrackMuonMVALineDecision,Hlt1SingleHighPtMuonLineDecision MVA:Hlt1TwoTrackMVALineDecision,Hlt1TrackMVALineDecision'`, and I think this doesn't work properly with the `nargs='+'` argparsing in `hlt_calculate_rates.py`. There should be no equals sign.https://gitlab.cern.ch/lhcb/MooreAnalysis/-/issues/17Use bookkeeping instead of TestFileDB2023-12-08T17:11:36+01:00Rosen MatevUse bookkeeping instead of TestFileDBThe TestFileDB is only really meant for small samples used in nightlty tests. It rarely has all the files in a production for a given entry, or even doesn't have entries at all for some interesting productions.
We should transition to u...The TestFileDB is only really meant for small samples used in nightlty tests. It rarely has all the files in a production for a given entry, or even doesn't have entries at all for some interesting productions.
We should transition to using the bookkeeping directly, building upon https://gitlab.cern.ch/lhcb-datapkg/PRConfig/-/merge_requests/170. The documentation at https://lhcbdoc.web.cern.ch/lhcbdoc/moore/master/tutorials/hltefficiencychecker.html needs to be revisited to be less centered on TestFileDB (perhaps not mention it at all), and possibly the scripts need to be adapted.
/cc @rjhunter @sklaver @mfontanahttps://gitlab.cern.ch/lhcb/MooreAnalysis/-/issues/19HltEfficiencyChecker should switch to using functors+FunTuple rather than Tup...2023-06-17T16:45:11+02:00Patrick KoppenburgHltEfficiencyChecker should switch to using functors+FunTuple rather than TupleTools+MCDecayTreeTupleWhat is the future of tupling in MooreAnalysis? I see development of TupleTools for MooreAnalysis https://gitlab.cern.ch/lhcb/Analysis/-/merge_requests/604 while on the other hand we want to remove all TupleTools https://gitlab.cern.ch/l...What is the future of tupling in MooreAnalysis? I see development of TupleTools for MooreAnalysis https://gitlab.cern.ch/lhcb/Analysis/-/merge_requests/604 while on the other hand we want to remove all TupleTools https://gitlab.cern.ch/lhcb-dpa/project/-/issues/7.
To satisfy https://gitlab.cern.ch/lhcb-dpa/project/-/issues/7, all remaining TupleTools were moved to `MooreAnalysis` in !55.
Tupling should be done with FunTuple+functors rather than the outdated MCDecayTreeTuple+TupleTools. Please see discussion below for progress/bottlenecks on this.
cc @rmatev @rjhunterhttps://gitlab.cern.ch/lhcb/MooreAnalysis/-/issues/20HltEfficiencyChecker does not work for Thor-based trigger line ?2021-07-23T14:57:45+02:00Mengzhen WangHltEfficiencyChecker does not work for Thor-based trigger line ?When trying to test BandQ trigger performance after using Thor functions (Moore MR924)[https://gitlab.cern.ch/lhcb/Moore/-/merge_requests/924], end up with
```
FunctionalChargedProtoParticleMa... INFO CUT: ' ( ( (TrTYPE==3) | (TrTYPE...When trying to test BandQ trigger performance after using Thor functions (Moore MR924)[https://gitlab.cern.ch/lhcb/Moore/-/merge_requests/924], end up with
```
FunctionalChargedProtoParticleMa... INFO CUT: ' ( ( (TrTYPE==3) | (TrTYPE==5) ) | (TrTYPE==4) ) '
FunctionalParticleMaker.LoKi::Hy... INFO CUT: ' ( (TrTYPE==3) &TrALL) '
FunctorFactory INFO Cache miss for functor: ::Functors::Filter( /* Predicate to filter the container with. */ operator&( operator&( operator&( operator&( operator&( operator&( operator&( operator>( ::Functors::Track::TransverseMomentum{}, 0.0f ), ::Functors::Track::IsMuon{} ), operator>( ::Functors::Track::Momentum{}, 0.0f ) ), operator>( ::Functors::Track::MinimumImpactParameterChi2<>( /* TES location of input [primary] vertices */ std::string{"/Event/LHCb__Converters__RecVertices__LHCbRecVerticesToVectorV2RecVertex/OutputVertices"} ), std::integral_constant<int, 9>{} ) ), operator>( ::Functors::Track::MinimumImpactParameter<>( /* TES location of input [primary] vertices */ std::string{"/Event/LHCb__Converters__RecVertices__LHCbRecVerticesToVectorV2RecVertex/OutputVertices"} ), 0.3f ) ), operator>( ::Functors::Track::PIDmu{}, -5.0f ) ), operator<( ::Functors::Track::GhostProbability{}, 0.4f ) ), operator<( ::Functors::Track::MinimumImpactParameterChi2<>( /* TES location of input [primary] vertices */ std::string{"/Event/LHCb__Converters__RecVertices__LHCbRecVerticesToVectorV2RecVertex/OutputVertices"} ), std::integral_constant<int, 10000000000000>{} ) ) ), now trying cling with headers [<string>, Event/Particle.h, Functors/Filter.h, Functors/TrackLike.h, GaudiKernel/NamedRange.h]
input_line_47:11:1505: error: non-type template argument evaluates to 10000000000000, which cannot be narrowed to type 'int' [-Wc++11-narrowing]
std::unique_ptr<Functors::AnyFunctor> functor_0xa945477f65a98317_cling() { return std::make_unique<Functors::Functor<SharedObjectsContainer<LHCb::Particle> (Gaudi::NamedRange_<std::vector<LHCb::Particle const*,std::allocator<LHCb::Particle const*> >,__gnu_cxx::__normal_iterator<LHCb::Particle const* const*,std::vector<LHCb::Particle const*,std::allocator<LHCb::Particle const*> > > > const&)>>( ::Functors::Filter( /* Predicate to filter the container with. */ operator&( operator&( operator&( operator&( operator&( operator&( operator&( operator>( ::Functors::Track::TransverseMomentum{}, 0.0f ), ::Functors::Track::IsMuon{} ), operator>( ::Functors::Track::Momentum{}, 0.0f ) ), operator>( ::Functors::Track::MinimumImpactParameterChi2<>( /* TES location of input [primary] vertices */ std::string{"/Event/LHCb__Converters__RecVertices__LHCbRecVerticesToVectorV2RecVertex/OutputVertices"} ), std::integral_constant<int, 9>{} ) ), operator>( ::Functors::Track::MinimumImpactParameter<>( /* TES location of input [primary] vertices */ std::string{"/Event/LHCb__Converters__RecVertices__LHCbRecVerticesToVectorV2RecVertex/OutputVertices"} ), 0.3f ) ), operator>( ::Functors::Track::PIDmu{}, -5.0f ) ), operator<( ::Functors::Track::GhostProbability{}, 0.4f ) ), operator<( ::Functors::Track::MinimumImpactParameterChi2<>( /* TES location of input [primary] vertices */ std::string{"/Event/LHCb__Converters__RecVertices__LHCbRecVerticesToVectorV2RecVertex/OutputVertices"} ), std::integral_constant<int, 10000000000000>{} ) ) ) ); }
```
But `EfficiencyChecker` works fine when testing `Hlt2Conf.lines.Bs2JpsiPhi` (default one in `MooreAnalysis/HltEfficiencyChecker/options/hlt2_rate_example.yaml`), where Thor function not used yet.https://gitlab.cern.ch/lhcb/MooreAnalysis/-/issues/21TOS matching in HltEfficiencyChecker fails for neutrals2023-12-14T17:29:11+01:00Ross John HunterTOS matching in HltEfficiencyChecker fails for neutralsTOS matching in `HltEfficiencyChecker` proceeds by comparing hits from tracks. Neutrals have no track, so it will seg fault.
Update: with !61 it will not seg fault, but rather give a zero TOS efficiency.TOS matching in `HltEfficiencyChecker` proceeds by comparing hits from tracks. Neutrals have no track, so it will seg fault.
Update: with !61 it will not seg fault, but rather give a zero TOS efficiency.https://gitlab.cern.ch/lhcb/MooreAnalysis/-/issues/22HltEfficiencyChecker: Better error messages when annotated decay descriptor i...2023-12-14T17:25:42+01:00Ross John HunterHltEfficiencyChecker: Better error messages when annotated decay descriptor isnt specified properly.Reported by @mkenzie - many thanks.
If you don't add the annotations you'll get
```
File "/home/epp/phskbk/Scratch/lhcb/LHCbStack/stack/Moore/Hlt/Hlt2Conf/options/bnoc/eff_Bs2KpiKpi.py", line 231, in <module>
run_moore_with_tuples...Reported by @mkenzie - many thanks.
If you don't add the annotations you'll get
```
File "/home/epp/phskbk/Scratch/lhcb/LHCbStack/stack/Moore/Hlt/Hlt2Conf/options/bnoc/eff_Bs2KpiKpi.py", line 231, in <module>
run_moore_with_tuples(
File "/home/epp/phskbk/Scratch/lhcb/LHCbStack/stack/MooreAnalysis/HltEfficiencyChecker/python/HltEfficiencyChecker/config.py", line 180, in run_moore_with_tuples
top_cf_node = add_efficiency_tuples(
File "/home/epp/phskbk/Scratch/lhcb/LHCbStack/stack/MooreAnalysis/HltEfficiencyChecker/python/HltEfficiencyChecker/config.py", line 279, in add_efficiency_tuples
decay, branches, _ = parse_descriptor_template(descriptor_template)
File "/home/epp/phskbk/Scratch/lhcb/LHCbStack/stack/MooreAnalysis/HltEfficiencyChecker/python/HltEfficiencyChecker/config.py", line 328, in parse_descriptor_template
parent = particles[
IndexError: list index out of range
```
which isn't super helpful. Also, if you place the `${name}` annotations inside the brackets rather than outside, you also won't get a helpful message.
These error messages should be improved.https://gitlab.cern.ch/lhcb/MooreAnalysis/-/issues/23Will HltEfficiencyChecker's "TOS" algorithm cause confusion w.r.t. old-school...2022-02-03T14:36:05+01:00Ross John HunterWill HltEfficiencyChecker's "TOS" algorithm cause confusion w.r.t. old-school TOS moving forward?Suggestion by @mengzhen - many thanks.
Our "TOS" algorithm in `HltEfficencyChecker` is different to the old-school "TOS" algorithm, and both algorithms will be in use in the future, albeit in different contexts. For example, the `HltEf...Suggestion by @mengzhen - many thanks.
Our "TOS" algorithm in `HltEfficencyChecker` is different to the old-school "TOS" algorithm, and both algorithms will be in use in the future, albeit in different contexts. For example, the `HltEfficiencyChecker` TOS algorithm works "online" and matches the true signal to the trigger reconstructed candidate, whereas the "TISTOS" method matches "offline" and matches an offline reconstructed candidate to the trigger reconstructed candidate. I haven't done a thorough review of the "TISTOS" code, but there are likely to be slight implementation differences, even if the main idea of the algorithm is the same.
As @mengzhen pointed out, there is a danger this could cause quite a bit of confusion, particularly with newcomers to the software.
Should we rename the algorithm used in `HltEfficiencyChecker`?https://gitlab.cern.ch/lhcb/MooreAnalysis/-/issues/24HltEfficiencyChecker: chain HLT1 and HLT2 so we can work out a total HLT effi...2021-11-22T12:36:25+01:00Ross John HunterHltEfficiencyChecker: chain HLT1 and HLT2 so we can work out a total HLT efficiencyFYI @rmatev @sstahl @mvesteri @gligorov
This feature has been requested by at least a couple of people now. @mkenzie put it very nicely to me today (quoting):
> It's also hard to tell how much better you can do. My Hlt2 line is curr...FYI @rmatev @sstahl @mvesteri @gligorov
This feature has been requested by at least a couple of people now. @mkenzie put it very nicely to me today (quoting):
> It's also hard to tell how much better you can do. My Hlt2 line is currently 50% efficient on C.R.C, but Hlt1 is only about the same. So is my line actually 100% on what it will eventually see?
You can get a HLT1 dst and manually filter out those events that fail your HLT2 lines, but it would be good if we could just chain HLT1-HLT2 jobs together in `HltEfficiencyChecker`, save decisions from both to the tuple and the construct e.g. HLT2 efficiency with an HLT1 line in the denominator.
In principle this doesn't sound too difficult to do, but not sure how much it would break things in practice.Ross John HunterRoss John Hunterhttps://gitlab.cern.ch/lhcb/MooreAnalysis/-/issues/25Compare HltEfficiencyChecker results to Allen standalone to test Allen-Moore ...2023-12-14T17:27:19+01:00Ross John HunterCompare HltEfficiencyChecker results to Allen standalone to test Allen-Moore integrationThe following discussion from !58 should be addressed:
- [ ] @sstahl started a [discussion](https://gitlab.cern.ch/lhcb/MooreAnalysis/-/merge_requests/58#note_4914225): (+6 comments)
> Thanks, I think it is great to have this func...The following discussion from !58 should be addressed:
- [ ] @sstahl started a [discussion](https://gitlab.cern.ch/lhcb/MooreAnalysis/-/merge_requests/58#note_4914225): (+6 comments)
> Thanks, I think it is great to have this functionality. I am just a bit worried that this is running Allen in Moore and as far as I know there is no automated test that checks that Allen in Moore and the production version of Allen are doing the same.
Further down this discussion I propose a way to address this.
Related to Moore#252https://gitlab.cern.ch/lhcb/MooreAnalysis/-/issues/26Options object which includes `decay`2023-12-05T15:30:42+01:00Sascha StahlOptions object which includes `decay`For automatization it might be useful to create a dedicated options object for MooreAnalysis which includes in addition a field for the decay. This way one can easier define a common file with the Hlt2 configuration for several jobs and ...For automatization it might be useful to create a dedicated options object for MooreAnalysis which includes in addition a field for the decay. This way one can easier define a common file with the Hlt2 configuration for several jobs and then just feed different input files into it.
So basically that one can do
```sh
gaudirun.py use_d2kkpi_sample.py use_fastest_reco.py
gaudirun.py use_bs2jpsiphi_sample.py use_fastest_reco.py
```https://gitlab.cern.ch/lhcb/MooreAnalysis/-/issues/27Suggested cleanups in HltEfficiencyChecker2023-05-09T09:52:02+02:00Ross John HunterSuggested cleanups in HltEfficiencyCheckerIn !58 @sstahl makes a few nice suggestions of how the Allen interface to `HltEfficiencyChecker` can be cleaned up. It was also noted that `config.py` can probably be refactored to remove redundant code.In !58 @sstahl makes a few nice suggestions of how the Allen interface to `HltEfficiencyChecker` can be cleaned up. It was also noted that `config.py` can probably be refactored to remove redundant code.https://gitlab.cern.ch/lhcb/MooreAnalysis/-/issues/28Multiple segfaults when running chained HLT1 + HLT2 jobs with MooreAnalysis2022-03-14T16:24:56+01:00Ryunosuke O'NeilMultiple segfaults when running chained HLT1 + HLT2 jobs with MooreAnalysisI have been noticed multiple (seemingly) random segmentation faults when running MooreAnalysis on the grid using `ganga`, which are also reproducible later on when running interactively. Here are a few examples, in addition to the one in...I have been noticed multiple (seemingly) random segmentation faults when running MooreAnalysis on the grid using `ganga`, which are also reproducible later on when running interactively. Here are a few examples, in addition to the one in the below issue:
![image](/uploads/54210ac6f5082544c9a4a9f9515a83e8/image.png)
![image](/uploads/2caa79b586bfc29b838562ba1b6e22cd/image.png)
![image](/uploads/62c61db785556af3ae74fce7092e7de6/image.png)
![image](/uploads/af455a2f6bef798949137023b24fe837/image.png)
Any ideas on how to track this down or work around it? It is not an issue with this specific file, because 45/100 files succeeded, and the other 55 or so jobs running on each of the other XDIGI files failed.
### Reproducing it
The stack or the latest version of the nightly may be used.
See https://gitlab.cern.ch/lhcb/Allen/-/issues/282https://gitlab.cern.ch/lhcb/MooreAnalysis/-/issues/29HltEfficiencyChecker for sprucing?2023-12-14T17:28:45+01:00Ross John HunterHltEfficiencyChecker for sprucing?FYI @nskidmor @rmatev @sstahl @mvesteri
Here's the work I think will be needed to make `HltEfficiencyChecker` work for sprucing:
* The user needs to specify the `level` when writing a `yaml`/"wizard" config file, which currently accept...FYI @nskidmor @rmatev @sstahl @mvesteri
Here's the work I think will be needed to make `HltEfficiencyChecker` work for sprucing:
* The user needs to specify the `level` when writing a `yaml`/"wizard" config file, which currently accepts `1`, `2` or `both`. This could be extended in `HltEfficencyChecker/options/options_template.py.jinja` to accept e.g. `Spruce`. However, `both` would no longer make sense, a better keyword would need to be dreamed up.
* Some sprucing examples (and tests) would need to be written for both `.yaml` and `.py`.
* Quite a few changes would be needed in `HltEfficiencyChecker/python/HltEfficiencyChecker/config.py` where different control flows are set up for HLT1 and HLT2.
* The TOS decisions are given by `Phys/DecayTreeTuple/src/MCTupleToolTOSHLT{1,2}`. The key difference is that the `HLT1` version gets LHCbIDs from the `SelReports`, whereas the `HLT2` tuple tool gets them from the persisted candidates. A sprucing version e.g. `MCTupleToolTOSSpruce` would be needed, although I guess it would be almost identical to the HLT2 version. Perhaps even the HLT2 tuple tool will just work, or work with minimal changes so an extra `MCTupleToolTOSSpruce` wouldn't be necessary.
* Then, the two analysis scripts that give efficiencies (`HltEfficiencyChecker/scripts/hlt_line_efficiencies.py`) and rates (`HltEfficiencyChecker/scripts/hlt_calculate_rates.py`) need a bit of work, although these scripts aren't that clever since they just make plots/printouts. The efficiencies script looks fairly agnostic to whether its HLT1 or HLT2, so shouldn't be too hard to extend it. The rates script has more to do.
* ... and then the documentation would need to be updated.
So I think it's quite a lot, but quite dull and shouldn't require too much thinking necessarily.https://gitlab.cern.ch/lhcb/MooreAnalysis/-/issues/30test_hlt1_allen_default test is failing with a segfault2022-03-09T14:47:56+01:00Rosen Matevtest_hlt1_allen_default test is failing with a segfaultThe HltEfficiencyChecker.test_hlt1_allen_default test is failing with a segfault.
In https://lhcb-nightlies.web.cern.ch/nightly/lhcb-master/1616/MooreAnalysis/x86_64_v2-centos7-gcc11-dbg/tests we get this:
```
#11 0x00007ff61ad5fd6c in...The HltEfficiencyChecker.test_hlt1_allen_default test is failing with a segfault.
In https://lhcb-nightlies.web.cern.ch/nightly/lhcb-master/1616/MooreAnalysis/x86_64_v2-centos7-gcc11-dbg/tests we get this:
```
#11 0x00007ff61ad5fd6c in THashTable::Clear (this=0x586042f0, option=0x7ff61afcd276 "nodelete") at /build/jenkins/workspace/lcg_release_pipeline/build/projects/ROOT-6.24.06/src/ROOT/6.24.06/core/cont/src/THashTable.cxx:177
#12 0x00007ff61ad5e3a1 in THashList::Delete (this=0x58258c60, option=0x7ff61a929ac4 "") at /build/jenkins/workspace/lcg_release_pipeline/build/projects/ROOT-6.24.06/src/ROOT/6.24.06/core/cont/src/THashList.cxx:214
#13 0x00007ff61a6e9615 in TDirectoryFile::Close (this=0x587df7a0, option=0x7ff5f65746f9 "") at /build/jenkins/workspace/lcg_release_pipeline/build/projects/ROOT-6.24.06/src/ROOT/6.24.06/io/io/src/TDirectoryFile.cxx:572
#14 0x00007ff61a7020ae in TFile::Close (this=0x587df7a0, option=0x7ff5f65746f9 "") at /build/jenkins/workspace/lcg_release_pipeline/build/projects/ROOT-6.24.06/src/ROOT/6.24.06/io/io/src/TFile.cxx:918
#15 0x00007ff5f63359c0 in ROOTService::~ROOTService (this=this
entry=0x5889df80, __in_chrg=<optimized out>) at ../main/src/ROOTService.cpp:56
#16 0x00007ff5f6a06040 in std::default_delete<ROOTService>::operator() (__ptr=0x5889df80, this=<optimized out>) at /cvmfs/lhcb.cern.ch/lib/lcg/releases/gcc/11.1.0-e80bf/x86_64-centos7/include/c++/11.1.0/bits/unique_ptr.h:85
#17 std::__uniq_ptr_impl<ROOTService, std::default_delete<ROOTService> >::reset (__p=0x0, this=0x585fedb0) at /cvmfs/lhcb.cern.ch/lib/lcg/releases/gcc/11.1.0-e80bf/x86_64-centos7/include/c++/11.1.0/bits/unique_ptr.h:182
#18 std::unique_ptr<ROOTService, std::default_delete<ROOTService> >::reset (__p=0x0, this=0x585fedb0) at /cvmfs/lhcb.cern.ch/lib/lcg/releases/gcc/11.1.0-e80bf/x86_64-centos7/include/c++/11.1.0/bits/unique_ptr.h:456
#19 RunAllen::finalize (this=0x585fd1c0) at ../Rec/Allen/src/RunAllen.cpp:187
#20 0x00007ff601ded78e in Gaudi::Algorithm::sysFinalize (this=0x585fd1c0) at ../GaudiKernel/src/Lib/Algorithm.cpp:483
#21 0x00007ff6008232c9 in operator() (alg=<optimized out>, __closure=0x7ffe98844c8f) at ../GaudiHive/src/HiveDataBroker.cpp:147
```
In another build we get something else (the root cause might be the same):
```
** Error in `python': munmap_chunk(): invalid pointer: 0x00000000597cb000 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x7f3e4)[0x7fcd347513e4]
/lib64/libc.so.6(+0x39d10)[0x7fcd3470bd10]
/lib64/libc.so.6(+0x39d37)[0x7fcd3470bd37]
/cvmfs/lhcb.cern.ch/lib/lcg/releases/ROOT/6.24.06-3455f/x86_64-centos7-gcc11-opt/lib/libCore.so(_ZN11TUnixSystem4ExitEib+0x1a)[0x7fcd2e87701a]
/cvmfs/lhcb.cern.ch/lib/lcg/releases/ROOT/6.24.06-3455f/x86_64-centos7-gcc11-opt/lib/libCore.so(_ZN11TUnixSystem15DispatchSignalsE8ESignals+0x41)[0x7fcd2e87ad01]
/lib64/libpthread.so.0(+0xf630)[0x7fcd351b8630]
/workspace/build/LHCb/InstallArea/x86_64_v3-centos7-gcc11-opt+g/lib/libHltDAQ.so(_ZNSt13unordered_mapIiS_INSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESt4pairIiiESt4hashIS5_ESt8equal_toIS5_ESaIS6_IKS5_S7_EEES8_IiESA_IiESaIS6_IKiSF_EEED1Ev+0x33)[0x7fcd07d49dd3]
/lib64/libc.so.6(+0x39ce9)[0x7fcd3470bce9]
/lib64/libc.so.6(+0x39d37)[0x7fcd3470bd37]
/cvmfs/lhcb.cern.ch/lib/lcg/releases/Python/3.9.6-b0f98/x86_64-centos7-gcc11-opt/bin/../lib/libpython3.9.so.1.0(+0x1d20a9)[0x7fcd357ce0a9]
/cvmfs/lhcb.cern.ch/lib/lcg/releases/Python/3.9.6-b0f98/x86_64-centos7-gcc11-opt/bin/../lib/libpython3.9.so.1.0(+0x1d664f)[0x7fcd357d264f]
/cvmfs/lhcb.cern.ch/lib/lcg/releases/Python/3.9.6-b0f98/x86_64-centos7-gcc11-opt/bin/../lib/libpython3.9.so.1.0(PyRun_SimpleFileExFlags+0x4f3)[0x7fcd357d3ae3]
/cvmfs/lhcb.cern.ch/lib/lcg/releases/Python/3.9.6-b0f98/x86_64-centos7-gcc11-opt/bin/../lib/libpython3.9.so.1.0(Py_RunMain+0x758)[0x7fcd357f0b48]
/cvmfs/lhcb.cern.ch/lib/lcg/releases/Python/3.9.6-b0f98/x86_64-centos7-gcc11-opt/bin/../lib/libpython3.9.so.1.0(Py_BytesMain+0x47)[0x7fcd357f1007]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7fcd346f4555]
```https://gitlab.cern.ch/lhcb/MooreAnalysis/-/issues/31HltEfficiencyChecker examples don't work2022-07-19T11:18:57+02:00Daniel CervenkovHltEfficiencyChecker examples don't workWith a fully updated stack (as of March 17), running
```
./MooreAnalysis/run MooreAnalysis/HltEfficiencyChecker/scripts/hlt_eff_checker.py --ignore-broken-inputs MooreAnalysis/HltEfficiencyChecker/options/hlt1_and_hlt2_eff_example.yaml
...With a fully updated stack (as of March 17), running
```
./MooreAnalysis/run MooreAnalysis/HltEfficiencyChecker/scripts/hlt_eff_checker.py --ignore-broken-inputs MooreAnalysis/HltEfficiencyChecker/options/hlt1_and_hlt2_eff_example.yaml
```
fails. The relevant error message seems to be
```
TransposeRawBanks ERROR Cannot find VPRetinaCluster raw bank.
```https://gitlab.cern.ch/lhcb/MooreAnalysis/-/issues/32HltEfficiencyChecker printing is mangled2022-07-19T11:18:56+02:00Daniel CervenkovHltEfficiencyChecker printing is mangled`HltEfficiencyChecker`/`hlt_calculate_rates.py` tries to be clever about aligning the decimal when printing the final rate table. However, it is not implemented correctly and ends up worse than nothing. See the output:
```
Line: Hlt2P...`HltEfficiencyChecker`/`hlt_calculate_rates.py` tries to be clever about aligning the decimal when printing the final rate table. However, it is not implemented correctly and ends up worse than nothing. See the output:
```
Line: Hlt2PID_BToKJpsi_JpsiToPmPpTagged_LineDecision Incl: 0.13348461589801777 +/- 0.09 kHz, Excl: 0.13348461589801777 +/- 0.09 kHz
Line: Hlt2PID_BToKJpsi_JpsiToPpPmTagged_LineDecision Incl: 0.0 +/- 0.0 kHz, Excl: 0.0 +/- 0.0 kHz
Line: Hlt2PID_DstToD0Pi_D0ToKPiPiPi_LineDecision Incl: 1.8020423146232396 +/- 0.34 kHz, Excl: 1.7353000066742308 +/- 0.34 kHz
```
This is due to the fact that the errors are [always truncated to 4 characters](https://gitlab.cern.ch/lhcb/MooreAnalysis/-/blob/master/HltEfficiencyChecker/scripts/hlt_calculate_rates.py#L248), while the central values are not.
Perhaps just limiting everything to two or three digits and right aligning would be better?https://gitlab.cern.ch/lhcb/MooreAnalysis/-/issues/33Error in <TTreeFormula::Compile>: Too many operators ! when processing all H...2023-12-17T18:42:07+01:00Miroslav Saurmiroslav.saur@cern.chError in <TTreeFormula::Compile>: Too many operators ! when processing all HLT2 linesObserved error in `<TTreeFormula::Compile>: Too many operators !` when processing all HLT2 lines currently implemented in master (more than 1000 lines).
Should be easy to solve by using [ROOT::v5::TFormula::SetMaxima](https://root.cer...Observed error in `<TTreeFormula::Compile>: Too many operators !` when processing all HLT2 lines currently implemented in master (more than 1000 lines).
Should be easy to solve by using [ROOT::v5::TFormula::SetMaxima](https://root.cern.ch/doc/master/classROOT_1_1v5_1_1TFormula.html#a93b0de2e86cdcf7250ffa4aa6aae9f6b)
FYI @rjhunterhttps://gitlab.cern.ch/lhcb/MooreAnalysis/-/issues/34Segmentation fault without message when running over a sample2022-08-01T09:58:25+02:00Louis Henrylouis.henry@cern.chSegmentation fault without message when running over a sample```
from Moore import options
from HltEfficiencyChecker.config import run_allen_in_moore_with_tuples
from RecoConf.hlt1_tracking import make_SPmixed_raw_event, make_RetinaCluster_raw_event, make_VeloClusterTrackingSIMD, make_velo_full_c...```
from Moore import options
from HltEfficiencyChecker.config import run_allen_in_moore_with_tuples
from RecoConf.hlt1_tracking import make_SPmixed_raw_event, make_RetinaCluster_raw_event, make_VeloClusterTrackingSIMD, make_velo_full_clusters
from PyConf.Algorithms import VeloRetinaClusterTrackingSIMD, VPRetinaFullClustering
from Configurables import HiveDataBrokerSvc
from Configurables import LHCb__ParticlePropertySvc as ParticlePropertySvc
ParticlePropertySvc().Particles = [ "H_10 87 25 0.0 0.5 1e-11 Higgs0 25 0.000000e+000" ]
HiveDataBrokerSvc().OutputLevel = 2
decay = ("[${B_s0}B_s0 => "
"( phi(1020) => ${Kplus0}K+ ${Kminus0}K-) "
"( phi(1020) => ${Kplus1}K+ ${Kminus1}K-) ]CC")
decay = ("[${Bplus}B+ => "
"( H_10 => ${muplus}mu+ ${muminus}mu-) "
"${Kplus}K+]CC")
options.evt_max = 10000
options.data_type = 'Upgrade'
options.input_type = 'ROOT'
options.simulation = True
options.conddb_tag = 'sim-20210617-vc-mu100'
options.dddb_tag = 'dddb-20210617'
import glob
options.input_files = ["/eos/lhcb/user/l/lohenry/LLP_Paper/lohenry.510/629644019/lohenry.510/629644019/Boole-Extended.digi"]*100
from RecoConf.hlt1_allen import allen_gaudi_node_barriers
from RecoConf.reconstruction_objects import reconstruction
from AllenCore.generator import make_transposed_raw_banks
with allen_gaudi_node_barriers.bind(sequence="hlt1_pp_veloSP"), reconstruction.bind( from_file=False), make_transposed_raw_banks.bind(rawbank_list=[
"ODIN", "Muon", "FTCluster", "UT", "VP", "Calo", "EcalPacked",
"HcalPacked"
]): # FIXME Dont like that I have to hard code this in.
run_allen_in_moore_with_tuples(options, decay)
```https://gitlab.cern.ch/lhcb/MooreAnalysis/-/issues/35HltEfficiencyChecker option for fastest reconstruction/change default2023-06-22T00:47:05+02:00Ross John HunterHltEfficiencyChecker option for fastest reconstruction/change defaultBrought up by @gunther in !81. I'm inclined to leave the default reconstruction sequence to be the sequence called "default", but happy to be convinced otherwise.Brought up by @gunther in !81. I'm inclined to leave the default reconstruction sequence to be the sequence called "default", but happy to be convinced otherwise.https://gitlab.cern.ch/lhcb/MooreAnalysis/-/issues/36HltEfficiencyChecker needs candidates in every event even if the line failed ...2023-12-14T17:30:12+01:00Ross John HunterHltEfficiencyChecker needs candidates in every event even if the line failed - can cause failuresFor HLT2, `MCDecayTreeTuple` has a datahandle-based dependency on the reconstructed candidates from each line running in each event, so that it can compare the reconstructed hits to the truth hits for `TrueSim` matching. At runtime, the ...For HLT2, `MCDecayTreeTuple` has a datahandle-based dependency on the reconstructed candidates from each line running in each event, so that it can compare the reconstructed hits to the truth hits for `TrueSim` matching. At runtime, the matching is smart enough to know to not try to match if the trigger line failed on that event. However, the data dependency means that the scheduler schedules the building of the candidate in every event regardless. So you can rarely run into issues where the candidate cannot be built and there is a failure, e.g. like
```
Hlt2RD_KS0ToPiE_Loose_Combiner ERROR Sel::calculateBestVertex : Empty vertex container passed to Sel::calculateBestVertex()
Hlt2RD_KS0ToPiE_Loose_Combiner ERROR Maximum number of errors ( 'ErrorMax':1) reached.
HLTControlFlowMgr FATAL Event failed in Node MCDecayTreeTuple/MCDecayTreeTuple : Error in algorithm execute
HLTControlFlowMgr FATAL *** Event 372 on slot 0 failed! ***
HLTControlFlowMgr INFO Dumping Alg Exec State for slot 0:
```
as observed recently. In the trigger line's control flow, it would stop before this point, because it has some filter like `require_pvs` that stops it getting this far, but such a filter is not part of the candidate-building, and so it is not run here.
### Ideas
* Can `MCDecayTreeTuple` take the output directly from the line, which should be e.g. an empty container when the line didn't fire?
* Can the combiners/candidate builders know the dependencies they have? If they knew they needed PVs, they could hopefully just return an empty container and stop, and no up-front `require_pvs` filter would be needed.https://gitlab.cern.ch/lhcb/MooreAnalysis/-/issues/37Recurring build warning in MooreAnalysis2023-02-07T17:00:27+01:00Miroslav Saurmiroslav.saur@cern.chRecurring build warning in MooreAnalysisBuild warnings appear in MooreAnalysis rather randomly for last few days.
All warning are of exactly same type:
```
Warning in &lt;TInterpreter::ReadRootmapFile&gt;: class IDecayFinder found in libDaVinciInterfacesDict.so is already i...Build warnings appear in MooreAnalysis rather randomly for last few days.
All warning are of exactly same type:
```
Warning in <TInterpreter::ReadRootmapFile>: class IDecayFinder found in libDaVinciInterfacesDict.so is already in libDecayTreeTupleBaseDict.so
4 occurrences: Phys/DaVinciDecayFinder, Phys/DecayTreeTuple, Phys/DaVinciAssociators, Phys/DecayTreeTupleBase
```
Appearing in:
```
Phys/DaVinciDecayFinder ( 1 warnings )
Phys/DecayTreeTuple ( 1 warnings )
Phys/DaVinciAssociators ( 1 warnings )
Phys/DecayTreeTupleBase ( 1 warnings )
```Ross John HunterRoss John Hunterhttps://gitlab.cern.ch/lhcb/MooreAnalysis/-/issues/38Outdated value of rate reported in HltEfficiencyChecker docs2023-05-11T15:02:41+02:00Michele AtzeniOutdated value of rate reported in HltEfficiencyChecker docsI was going through https://lhcbdoc.web.cern.ch/lhcbdoc/moore/master/tutorials/hltefficiencychecker.html and, with a freshly downloaded stack, I tried to run the example `MooreAnalysis/run MooreAnalysis/HltEfficiencyChecker/scripts/hlt_e...I was going through https://lhcbdoc.web.cern.ch/lhcbdoc/moore/master/tutorials/hltefficiencychecker.html and, with a freshly downloaded stack, I tried to run the example `MooreAnalysis/run MooreAnalysis/HltEfficiencyChecker/scripts/hlt_eff_checker.py MooreAnalysis/HltEfficiencyChecker/options/hlt2_rate_example.yaml` .
To my surprise, I got 0kHz rather than 30kHz. I guess with some of the most recent updates of Moore this value became outdated? If so it can be useful to correct the guide.https://gitlab.cern.ch/lhcb/MooreAnalysis/-/issues/39Issues with plotting overlap metrics and saving intersections in HltEfficienc...2023-05-02T15:53:10+02:00Jamie GoodingIssues with plotting overlap metrics and saving intersections in HltEfficiencyCheckerIssue reported by @miolocco, when plotting overlap metrics (seen in Jaccard matrix but equally applicable to cond. prob.), labels of cells where the error is >= 1 are shown as 0 +/- 0, even if cell value (which is coloured correctly) is ...Issue reported by @miolocco, when plotting overlap metrics (seen in Jaccard matrix but equally applicable to cond. prob.), labels of cells where the error is >= 1 are shown as 0 +/- 0, even if cell value (which is coloured correctly) is not 0.Micol OloccoJamie GoodingMicol Oloccohttps://gitlab.cern.ch/lhcb/MooreAnalysis/-/issues/40Crashing hlt1_and_hlt2_example when setting output_file to some value2023-12-08T17:10:40+01:00Louis Henrylouis.henry@cern.chCrashing hlt1_and_hlt2_example when setting output_file to some valueRunning `hlt1_and_hlt2_eff_example` with output_file set to a value, running either level = 2 or both leads to some ERRORS that are difficult to understand:
```
PyConf.utilities.ConfigurationError: Name collision detected for CombineRawB...Running `hlt1_and_hlt2_eff_example` with output_file set to a value, running either level = 2 or both leads to some ERRORS that are difficult to understand:
```
PyConf.utilities.ConfigurationError: Name collision detected for CombineRawBanks_for_default, identities are 0x24e10b3ca7b5514b and 0x7d0cefc82af19f7. Please make names unique. One possibility is to use a '_{hash}' suffix that will be automatically replaced by a unique hash
```
When adding a `hash` to the writers in Moore/config, the following ERROR is produced:
```
HiveDataBrokerSvc WARNING non-reentrant algorithm: EventTuple/EventTuple_2a49dff6
HiveDataBrokerSvc FATAL in sysInitialize(): exception with tag=mapProducers is caught
HiveDataBrokerSvc ERROR mapProducers multiple algorithms declare /Event/default as output (CombineRawBanks_for_default_7d0cefc8 and CombineRawBanks_for_default_24e10b3c at least). This is not allowed StatusCode=FAILURE
HiveDataBrokerSvc ERROR Exception stack trace
```
One could either force output_file to be set to "" or fix the underlying issue (no clue why writing fails here when providing a `hash`)https://gitlab.cern.ch/lhcb/MooreAnalysis/-/issues/41HltEfficiencyChecker: follow-up from !122; restore HLT1 True Sim efficiencies2023-12-14T17:29:42+01:00Ross John HunterHltEfficiencyChecker: follow-up from !122; restore HLT1 True Sim efficiencies* MooreAnalysis!116 removed tupling from HLT1 in HltEfficiencyChecker if no events were selected. This was because, in that chain of MRs, this error appeared:
```
HltSelReportsDecoder_3ce0f8f9 ERROR HltSelRepRawBank : sub-bank in...* MooreAnalysis!116 removed tupling from HLT1 in HltEfficiencyChecker if no events were selected. This was because, in that chain of MRs, this error appeared:
```
HltSelReportsDecoder_3ce0f8f9 ERROR HltSelRepRawBank : sub-bank index out of range in HltSelRepRawBank
HltSelReportsDecoder_3ce0f8f9 ERROR Maximum number of errors ( 'ErrorMax':1) reached.
HLTControlFlowMgr FATAL Event failed in Node FunTupleBase_MCParticles/MCFunTuple : Error in algorithm execute
```
* This error is related to reading of sel reports to compute Hlt1 True Sim efficiencies.
* Migrating to FunTuple (!122) hasn't made this go away, and we need the control flow in HltEfficiencyChecker to go back to `NONLAZY_AND`, i.e. tupling for every event regardless of whether triggers fired. Therefore in !122 I won't put back true sim efficiencies for HLT1.
* The error should be further investigated to put back these efficiencies. Normal, "DEC" efficiencies for HLT1 will be fine and available to the user.https://gitlab.cern.ch/lhcb/MooreAnalysis/-/issues/42Discussion about different results in rate when computing manually and with t...2023-09-18T06:19:27+02:00Nicole SchulteDiscussion about different results in rate when computing manually and with the efficiencyCheckerThis issue is opened after yesterday's discussion in the WP3 meeting.
All scripts that were used in the rates calculations for the topo can be found on the MooreAnalysis branch ```Topo_Neural_Net``` together with the commands that are u...This issue is opened after yesterday's discussion in the WP3 meeting.
All scripts that were used in the rates calculations for the topo can be found on the MooreAnalysis branch ```Topo_Neural_Net``` together with the commands that are used to launch the script.
For the hlt1-filtered Upgrade Minbias, the script: ```HltEfficiencyChecker/options/hlt2_2or3_toporate_UpgradeMinbias.py```
is used and leads to the following rates:
[UpgradeMinbiasRates.log](/uploads/8a5fcba38c9a73e5c50972a590ae4524/UpgradeMinbiasRates.log)
The notation is the following:
```Hlt2TwoBody_9669115942028985Decision``` means only the TwoBody Topo with an MVA Cut of ```0.9669115942028985``` is run.
When using a chained Allen approach on a Sim10U1 minbias from the bookkeeping, the script
``` HltEfficiencyChecker/options/hlt2_2or3_toporate_withAllen.py```
is used with the corresponding commands being named in the file. The rates that we get are:
[hlt1_and_hlt2_test.log](/uploads/63d3cc5c2b192f7e0e6c1358de5592d8/hlt1_and_hlt2_test.log)
with the same notion as before.
When calculating rates offline "per hand", we (@bldelane) get rates about an order of magnitude lower.
The tuples in used in the training are compiled via ganga and all the corresponding scripts can be found here:
```https://gitlab.cern.ch/nschulte/topo_neural_network/-/tree/master/ganga```
The topo line-file was amended for this calculation to the following (note the loop over different bins of MVA cuts in the end, which is only for developing purposes and not to be merged)
[__init__.py](/uploads/8bc03158536e013f7f9de207ac3e0aea/__init__.py)
@bldelane @gciezare @poluekt @rjhunterhttps://gitlab.cern.ch/lhcb/MooreAnalysis/-/issues/43Tests fail in DD4HEP builds2023-12-14T17:30:37+01:00Christopher Rob Jonesjonesc@hep.phy.cam.ac.ukTests fail in DD4HEP buildsMooreAnalysis tests for the most part fail in the DD4HEP builds. See e.g.
https://lhcb-nightlies.web.cern.ch/nightly/lhcb-master/2135/MooreAnalysis/x86_64_v2-centos7-gcc12-opt/tests
There are probably a number of reasons, but the main ...MooreAnalysis tests for the most part fail in the DD4HEP builds. See e.g.
https://lhcb-nightlies.web.cern.ch/nightly/lhcb-master/2135/MooreAnalysis/x86_64_v2-centos7-gcc12-opt/tests
There are probably a number of reasons, but the main one, affecting most (maybe all?) tests is the MC input data used is too old to be supported by DD4HEP. This needs to be addressed first, updating to newer samples that DD4HEP can support.
Then, there are a number of other possible problems (could be secondary effects from the input data).
Seg. fault from Allen code
```
Thread 2 (Thread 0x7efdd7810700 (LWP 43118) "python3"):
#0 0x00007efe14304549 in waitpid () from /lib64/libc.so.6
#1 0x00007efe14281f62 in do_system () from /lib64/libc.so.6
#2 0x00007efe14282311 in system () from /lib64/libc.so.6
#3 0x00007efe0f8f94ed in TUnixSystem::StackTrace() () from /cvmfs/lhcb.cern.ch/lib/lcg/releases/ROOT/6.28.00-a8528/x86_64-centos7-gcc12-opt/lib/libCore.so
#4 0x00007efe0fb8a2a3 in (anonymous namespace)::TExceptionHandlerImp::HandleException(int) () from /cvmfs/lhcb.cern.ch/lib/lcg/releases/ROOT/6.28.00-a8528/x86_64-centos7-gcc12-opt/lib/libcppyy_backend3_9.so
#5 0x00007efe0f8f8d21 in TUnixSystem::DispatchSignals(ESignals) () from /cvmfs/lhcb.cern.ch/lib/lcg/releases/ROOT/6.28.00-a8528/x86_64-centos7-gcc12-opt/lib/libCore.so
#6 <signal handler called>
#7 0x00007efde5b0e750 in void calculate_number_of_hits<3>(unsigned int const*, unsigned int*, UTBoards const&, UTRawBank<3> const&) () from /workspace/build/Allen/InstallArea/x86_64_v2-centos7-gcc12-opt/lib/libAllenLib.so
#8 0x00007efde5b11722 in void ut_calculate_number_of_hits::ut_calculate_number_of_hits<3, false>(ut_calculate_number_of_hits::Parameters, unsigned int, char const*, unsigned int const*, unsigned int const*) () from /workspace/build/Allen/InstallArea/x86_64_v2-centos7-gcc12-opt/lib/libAllenLib.so
#9 0x00007efde5b0f345 in ut_calculate_number_of_hits::ut_calculate_number_of_hits_t::operator()(Allen::Store::StoreRef<std::tuple<ut_calculate_number_of_hits::Parameters::host_number_of_events_t, ut_calculate_number_of_hits::Parameters::host_raw_bank_version_t, ut_calculate_number_of_hits::Parameters::dev_event_list_t, ut_calculate_number_of_hits::Parameters::dev_ut_raw_input_t, ut_calculate_number_of_hits::Parameters::dev_ut_raw_input_offsets_t, ut_calculate_number_of_hits::Parameters::dev_ut_raw_input_sizes_t, ut_calculate_number_of_hits::Parameters::dev_ut_hit_sizes_t, ut_calculate_number_of_hits::Parameters::block_dim_t>, std::tuple<ut_calculate_number_of_hits::Parameters::host_number_of_events_t, ut_calculate_number_of_hits::Parameters::host_raw_bank_version_t, ut_calculate_number_of_hits::Parameters::dev_event_list_t, ut_calculate_number_of_hits::Parameters::dev_ut_raw_input_t, ut_calculate_number_of_hits::Parameters::dev_ut_raw_input_offsets_t, ut_calculate_number_of_hits::Parameters::dev_ut_raw_input_sizes_t, ut_calculate_number_of_hits::Parameters::dev_ut_hit_sizes_t>, ut_calculate_number_of_hits::Parameters, std::tuple<> > const&, RuntimeOptions const&, Constants const&, Allen::Context const&) const () from /workspace/build/Allen/InstallArea/x86_64_v2-centos7-gcc12-opt/lib/libAllenLib.so
#10 0x00007efde15eeade in ut_calculate_number_of_hits_t::operator()(EventContext const&, std::vector<unsigned int, std::allocator<unsigned int> > const&, std::vector<int, std::allocator<int> > const&, std::vector<mask_t, std::allocator<mask_t> > const&, std::vector<char, std::allocator<char> > const&, std::vector<unsigned int, std::allocator<unsigned int> > const&, std::vector<unsigned int, std::allocator<unsigned int> > const&, RuntimeOptions const&, Constants const* const&) const () from /workspace/build/Allen/InstallArea/x86_64_v2-centos7-gcc12-opt/lib/libAllenAlgorithms.so
#11 0x00007efde15f30a3 in Gaudi::Functional::details::MultiTransformer<std::tuple<std::vector<unsigned int, std::allocator<unsigned int> > > (EventContext const&, std::vector<unsigned int, std::allocator<unsigned int> > const&, std::vector<int, std::allocator<int> > const&, std::vector<mask_t, std::allocator<mask_t> > const&, std::vector<char, std::allocator<char> > const&, std::vector<unsigned int, std::allocator<unsigned int> > const&, std::vector<unsigned int, std::allocator<unsigned int> > const&, RuntimeOptions const&, Constants const* const&), Gaudi::Functional::Traits::use_<Gaudi::Functional::Traits::BaseClass_t<Gaudi::Algorithm> >, false>::execute(EventContext const&) const () from /workspace/build/Allen/InstallArea/x86_64_v2-centos7-gcc12-opt/lib/libAllenAlgorithms.so
#12 0x00007efdecad9d01 in Gaudi::Algorithm::sysExecute(EventContext const&) () from /workspace/build/Gaudi/InstallArea/x86_64_v2-centos7-gcc12-opt/lib/libGaudiKernel.so
```
Invalid DD tags used with DD4HEP
```
LHCb::Det::LbDD4hep::DD4hepSvc ERROR 'dddb-20171126' is not a valid geometry version
```
and probably others.
Similar to what was done with Moore in lhcb&14, I will open an MR disabling the tests in dd4hep. Then via as part of lhcb&15 the work needed to re-enable them can be tracked.Ross John HunterRoss John Hunterhttps://gitlab.cern.ch/lhcb/MooreAnalysis/-/issues/44Bind of make_digits calo_raw_bank=False in HltEfficiencyChecker examples2023-12-17T18:41:58+01:00Jamie GoodingBind of make_digits calo_raw_bank=False in HltEfficiencyChecker examplesIn the HLT1+HLT2 chained examples, the following bind is called (e.g. at [hlt1_and_hlt2_eff_example.py#L35](https://gitlab.cern.ch/lhcb/MooreAnalysis/-/blob/master/HltEfficiencyChecker/options/hlt1_and_hlt2_eff_example.py#L35)):
```
make...In the HLT1+HLT2 chained examples, the following bind is called (e.g. at [hlt1_and_hlt2_eff_example.py#L35](https://gitlab.cern.ch/lhcb/MooreAnalysis/-/blob/master/HltEfficiencyChecker/options/hlt1_and_hlt2_eff_example.py#L35)):
```
make_digits.global_bind(calo_raw_bank=False)
```
When investigating the efficiencies of dilepton triggers recently with equivalent options, the inclusion of this bind caused a drop in efficiency of factor O(100). Without the bind, the options run absolutely fine, returning _sensible_ efficiencies.
Is this line really required/can it be removed?
(p.s. this issue is somewhat of a placeholder, planning to run tests with/without bind this week)Ross John HunterJamie GoodingRoss John Hunterhttps://gitlab.cern.ch/lhcb/MooreAnalysis/-/issues/45Retirement of MooreAnalysis2023-12-19T15:25:23+01:00Ross John HunterRetirement of MooreAnalysisFYI @erodrigu @rmatev @amathad
Decisions made in Analysis#52
As discussed there, `Analysis` is being retired and `FunTuple` is being moved to `DaVinci`. `HltEfficiencyChecker` (the original use case for `MooreAnalysis`) depends on `Fu...FYI @erodrigu @rmatev @amathad
Decisions made in Analysis#52
As discussed there, `Analysis` is being retired and `FunTuple` is being moved to `DaVinci`. `HltEfficiencyChecker` (the original use case for `MooreAnalysis`) depends on `FunTuple`, and so must therefore move to `DaVinci` with it. Furthermore, since `HltEfficiencyChecker` is the last remaining package/code in `MooreAnalysis`, this project can be retired once the move is complete.
Work has been ongoing this week in preparation for the move. The remaining timeline of work required is:
* [x] Merge MooreAnalysis!134 (and dependent Moore MR; doc update) to close #33 and #44 (the last two open issues).
* [x] Merge !131 and !132 to get them out of the way before the move. These two do not conflict, and are both ready to merge. They also should not conflict with !134.
(the above merges can all proceed in parallel/their order is not important as they do not conflict)
* [x] Rebase/update DaVinci!1005 to reflect above merges.
* [x] Make MooreAnalysis MR to remove HltEfficiencyChecker.
* [ ] Remake !130 in DaVinci on top of DaVinci!1005, as !130 doesn't look like it will be ready to merge before HltEfficiencyChecker is moved to DaVinci. Close !130. At this point there'll be no open MRs to MooreAnalysis and no open issues apart from this one and the MR to remove HltEfficiencyChecker.
* [x] Merge MooreAnalysis MR removing HltEfficiencyChecker, DaVinci moving MR and Moore!2812 (MR for docs changes) together, et voila.
* [ ] Retire MooreAnalysis (not sure what that means in practice for e.g. nightly builds, as I guess we don't actually delete the repo)Ross John HunterRoss John Hunter