trick to get JES variations in data rather than MC

Fix #157 (closed)

@euslan Please test

added changepatch statusongoing typesoftware labels

requested review from @lmoureau and @euslan

assigned to @paconnor

Hi @paconnor, I tried but I got an error when running applyJEScorrections with chs. I also got the same error for Puppi. :

$ try applyJEScorrections 1D_mergeNtuples/RunA/ 2D_applyJEScorrections/ /eos/user/e/euslan/EL9/DasAnalysisSystem/tables/jsonpog-integration/POG/JME/2018_UL/Daniel/ak4/jet_chs_jerc.json Summer19UL18_RunA_V6_DATA -f -s
applyJEScorrections 1D_mergeNtuples/RunA/ /eos/home-e/euslan/Root_Files_EL9/JetHT_UL18_CHS/2D_applyJEScorrections/0.root /eos/user/e/euslan/EL9/DasAnalysisSystem/tables/jsonpog-integration/POG/JME/2018_UL/Daniel/ak4/jet_chs_jerc.json Summer19UL18_RunA_V6_DATA -f -s -j 287519
Warning: you are overwriting "/eos/home-e/euslan/Root_Files_EL9/JetHT_UL18_CHS/2D_applyJEScorrections/0.root"
applyJEScorrections 0/287519 start
latest commit: fix Core!157
               by Patrick L.S. Connor (bf8499b0277bc7a)
ControlPlots	raw
ControlPlots	nominal
ControlPlots	AbsoluteStatDown
ControlPlots	AbsoluteStatUp
ControlPlots	AbsoluteScaleDown
ControlPlots	AbsoluteScaleUp
ControlPlots	AbsoluteMPFBiasDown
ControlPlots	AbsoluteMPFBiasUp
ControlPlots	FragmentationDown
ControlPlots	FragmentationUp
ControlPlots	SinglePionECALDown
ControlPlots	SinglePionECALUp
ControlPlots	SinglePionHCALDown
ControlPlots	SinglePionHCALUp
ControlPlots	FlavorQCDDown
ControlPlots	FlavorQCDUp
ControlPlots	RelativeJEREC1Down
ControlPlots	RelativeJEREC1Up
ControlPlots	RelativeJEREC2Down
ControlPlots	RelativeJEREC2Up
ControlPlots	RelativeJERHFDown
ControlPlots	RelativeJERHFUp
ControlPlots	RelativePtBBDown
ControlPlots	RelativePtBBUp
ControlPlots	RelativePtEC1Down
ControlPlots	RelativePtEC1Up
ControlPlots	RelativePtEC2Down
ControlPlots	RelativePtEC2Up
ControlPlots	RelativePtHFDown
ControlPlots	RelativePtHFUp
ControlPlots	RelativeBalDown
ControlPlots	RelativeBalUp
ControlPlots	RelativeSampleDown
ControlPlots	RelativeSampleUp
ControlPlots	RelativeFSRDown
ControlPlots	RelativeFSRUp
ControlPlots	RelativeStatFSRDown
ControlPlots	RelativeStatFSRUp
ControlPlots	RelativeStatECDown
ControlPlots	RelativeStatECUp
ControlPlots	RelativeStatHFDown
ControlPlots	RelativeStatHFUp
ControlPlots	PileUpDataMCDown
ControlPlots	PileUpDataMCUp
ControlPlots	PileUpPtRefDown
ControlPlots	PileUpPtRefUp
ControlPlots	PileUpPtBBDown
ControlPlots	PileUpPtBBUp
ControlPlots	PileUpPtEC1Down
ControlPlots	PileUpPtEC1Up
ControlPlots	PileUpPtEC2Down
ControlPlots	PileUpPtEC2Up
ControlPlots	PileUpPtHFDown
ControlPlots	PileUpPtHFUp
/eos/user/e/euslan/EL9/DasAnalysisSystem/Core/JEC/bin/applyJEScorrections.cc(110): Throw in function void DAS::JetEnergy::applyJEScorrections(std::vector<std::filesystem::__cxx11::path>, std::filesystem::__cxx11::path, const boost::property_tree::ptree&, int, Darwin::Tools::Slice)
Dynamic exception type: boost::wrapexcept<std::invalid_argument>
std::exception::what: `Summer19UL18_RunA_V6_DATA_AbsoluteStat_AK4PFchs` cannot be found in `/eos/user/e/euslan/EL9/DasAnalysisSystem/tables/jsonpog-integration/POG/JME/2018_UL/Daniel/ak4/jet_chs_jerc.json`
To reproduce error, you may run:
applyJEScorrections /eos/home-e/euslan/Root_Files_EL9/JetHT_UL18_CHS/1D_mergeNtuples/RunA /eos/home-e/euslan/Root_Files_EL9/JetHT_UL18_CHS/2D_applyJEScorrections/0.root /eos/user/e/euslan/EL9/DasAnalysisSystem/tables/jsonpog-integration/POG/JME/2018_UL/Daniel/ak4/jet_chs_jerc.json Summer19UL18_RunA_V6_DATA -j 287519 -s -f
try: Command '['applyJEScorrections', '1D_mergeNtuples/RunA/', '/eos/home-e/euslan/Root_Files_EL9/JetHT_UL18_CHS/2D_applyJEScorrections/0.root', '/eos/user/e/euslan/EL9/DasAnalysisSystem/tables/jsonpog-integration/POG/JME/2018_UL/Daniel/ak4/jet_chs_jerc.json', 'Summer19UL18_RunA_V6_DATA', '-f', '-s', '-j', '287519']' returned non-zero exit status 1.
Boost 1_82, Cpp C++20, Darwin a30e901bffe1fdf9b7f11974b793300495143f0c, ROOT 6.28/04, gpp 13.1.0, libgit2 1.1.1, Python 3.9
Consider opening a GitLab issue if you don't understand the cause of the failure.

added 2 commits

• bdcbf409 - implementing verbosity in ControlPlot
• 8f3a4b6d - minor bugfix

Compare with previous version

added 1 commit

• c5076b49 - debugging

Compare with previous version

So the problem is the unexpected mix of versions within the same file. Both V5 and V6 are used. I think that the problem should be solved upstream. I contacted @dsavoiu.

added 1 commit

• ad68dc5f - fully rivisiting the identification of the correction in a JSON JME correction file

Compare with previous version

Note to self: add a unit test

added 1 commit

• f10f4f2c - Adding a regex to identify the campaign without the data taking era

Compare with previous version

Note to self: make arguments of applyJERsmearing consistent

marked this merge request as ready

Note to self: fix the CI (Ntupliser/test/runAnalysis.sh)

mentioned in issue #55

added 1 commit

• 617e1da4 - harmonising the arguments for JES and JER application

Compare with previous version

added 1 commit

• c5e294ed - fix CI

Compare with previous version

added 2 commits

• 14598bff - reviving JEC tests
• e3c435f5 - testing the gymnastics to get the campaigns and the clustering algorithm automatically

Compare with previous version

resolved all threads

The pipeline is failing, but I ran the tests locally and successfully.