Skip to content

Commit and squash all time fit and toy pipeline script changes from dev_kheinick_sfit

Kevin Heinicke requested to merge dev_kh_dms_scripts into dev_kh_mergeoverview

This is 1 of 3 MRs containing a large number of changes for the delta ms time fit and toy study scripts.

To allow a better "diff view" the changes are split into three MRs. This MR contains only changes relevant for the toy generation, MDFit and sFit script and is therefore the most relevant MR to be reviewed. Feedback is very appreciated @adudziak @jobutter (I was told you also worked with the toyFactory :-) ) All changes are squashed into one commit. To automate the toy generation, a DeltaMs-specific Snakefile from the lhcb-b2oc/analyses/b2oc-deltams-run2 repository is used.

Changes include:

  • the text in the MDFit plot can be adjusted via the config file
  • trailing spaces and code formattig has been adjusted in several places (pep8 compliance still a long way to go)
  • Add some verbosity to WS import handler
  • Use pythons string formatting instead of (python string → TString → python string)
  • Remove the "shell" headings from plotSFit.py, runSFit_new.py, toyFactory.py
    • I don't know how this is supposed to be used and how the scripts should be called without the Gaudi python. If shell setup and environment variable massage is needed, this should probably be done in a separate env-setup script instead of each individual script.
  • Add workaround to bug reported here https://root-forum.cern.ch/t/segfaults-when-using-rooabsdata-tree-storange-and-handling-multiple-tfiles/37069
    • this should be fixed in ROOT v6.20 but that needs to be verified
    • the workaround includes using RooDataSet->GetClonedTree() instead of workspace->Write():
      • in prepareWorkspace.py, each individual tree of the RooWorkspaces are saved (can be turned on via --save-individual-trees flag)
      • furthermore, successive calls to workspace.Save() (through GeneralUtils.SaveWorkspace()) are replaced by a single final call. This might also be a "simple" ownership problem and not only due to the aforemention issue
      • in runSFit_new.py, replacing the getattr(workspace, 'import')(data) with a separate data.GetClonedTree().Write()
      • reading data into a RooAbsData.Vector storage in runMDFitter.py by default (this requrires a relatively large amount of memory for large datasets, though)
  • a output prefix can be defined in prepareWorkspace.py (defaulting to the previous value)
  • Add a vim modeline to help vim-people read the code

runMDFit.py

  • unfinished option to read toy workspaces in MDFit script is removed. Instead, the toyFactory generates workspaces that can be fed into the MDFit script directly
    • Due to different capitalization conventions, mdfitutils.py internall uses only lowercase mode captions in getSampleModeYear()
  • fix Bd2DhModels in the case of turning on widthRatio && scaleWidths
  • add some verbosity to Bs2DshModels.cxx

runSFit_new.py

  • removed unused function getCalibratedMistag() from runSFit_new.py
  • refactor getCalibrationParameter() in runSFit_new.py
  • add option to specify a mean scale factor for the time resolution model via getTrmMeanScaleFactor, allowing to properly define a decay time bias through the resolution model
    • this required turning off an assert in RooCubinSplineKnot, checking that the offset is 0. I don't see the reason for this assert (it is also disabled in P2VV fitter, why are there two versions of the RooCubicSpline anyway?)
  • add option to perform a "in-data" calibration of the decay time with a config entry looking like 
    myconfig["TimeCalibration"] = {
    "params": {
        "a": 1,
        "b": 0,
        "c": 0,
    }
}
    that applies a RooFormulaVar of the form t * (a + b * t_err) + c to the time variable. This has the same effect as setting the trmMeanScaleFactor.
  • Allow to toggle sWeight correction via config file
    • B2DXFitter-based through SFitUtils.ReadDataFromSWeights() via setting myconfigfile["SWeightCorrectionRead"] = True
    • RooFit-based through model.fitTo(RooFit.SumW2(True) by setting myconfigfile["SWeightCorrectionFit"] = True
  • Improve deltaMs blinding by using RooUnblindUniform instead of RooUnblindOffset
    • Print blinded fit result (also on data)
  • Add two Flavour Tagging calibration options:
    • via Espresso.GLM, by specifying myconfigfile['TaggingCalibration']['Type'] = 'GLM'. Note that serializing the RooBDecay with a GLM function currently breaks reading the fit model (causing a SegFault as soon as doing file.Get('workspace')).
      • To evaluate the fitresult, a copy of only the result is stored independently of the full workspace. This also speeds up reading the results for the pull plot generation.
    • via RooFormulaVar, by specifying myconfigfile['TaggingCalibration']['Type'] = 'Logit'. Note that this should yield the same result as the GLM fit, but seems to be missing some Espresso-internal details, hence yields more unstable fits than the GLM version
    • Both options can be configured to be shared accross the simultaneous fit datasets, by ommitting the "year" level in the config file.
  • make maxEntries configurable in SFitUtils.cxx
  • remove magic scale factor of TAUERR in SFitUtils.cxx

toyFactory.py

  • use a singleton to access target generationYield via getGenerationYield()
  • Allow to generate mistag distribution from data via RooHistPDF
  • refactor MockTimeErrorPDF generation
  • Only generate one polarity (up) to run merging in runMDFit.py
  • replace TrueID generation via a narrow gaussian with usage of RooCategories
  • Add IpatiaJohnsonSU and ExponentionPlusSignal generation model
  • Always generate TagDec from the PDF (withouth using ProtoData) by default. This can be forced off via myconfigfile['ForceProtoForTagging'] = True

The relevant commit history from dev_kheinick_sfit is:

  • 7bf32a15 - Addd debug config for GLM, store fitresult in a separate root file (3 days ago)
  • eeb98a6b - Fix bug that used shared GLM calibration between OS/SS (4 days ago)
  • 7e84d32a - Back to Minuit (Fortran version) (5 days ago)
  • eff36497 - Fix manual logit calibration, use unique names. Minor config tunings (5 days ago)
  • 05d4a0cc - Try to speed up debug config a bit more, help tag asym to converge (7 days ago)
  • 2433dd00 - Allow forceful usage of protodata for tagging variables (7 days ago)
  • ea6e7a68 - New toy setups, allow shared tagging parameters (10 days ago)
  • eb663ef5 - Add options to run GLM or manual logistic linear calibration (13 days ago)
  • 4ad1687a - Add option to store raw trees for RooDataSets (4 weeks ago)
  • 9194cbd5 - Properly use protodata for generation, fix non-sim fit options (4 weeks ago)
  • 7ccb353d - Help vim people read the code (4 weeks ago)
  • 535444b8 - Save workspace only once to prevent problems with RooTreeDataStore (5 weeks ago)
  • 274c4fd7 - Respect output_prefix in MassFitUtils::ObtainSignal, minor c++ format (5 weeks ago)
  • f5a93d20 - Remove dangerous assumption about meaning of background list (6 weeks ago)
  • 7c6c2a2c - some stdout fixes, minimal refactoring (6 weeks ago)
  • 63fd9c0f - Initialize Bd2DsPi background "ShiftedSignalIpatiaJohnsonSU" with R (6 weeks ago)
  • cb48e1f9 - Start update of nominal toys, allow per-mode workspace loading (7 weeks ago)
  • 6be30525 - Merge remote-tracking branch 'origin/master' into dev_kheinick_sfit (8 weeks ago) |
    | * 2e404194 - initial version of satellite project (3 months ago)
  • | 353e4973 - RooTreeDataStore handling, allow "manual" time bias in time fit (8 weeks ago)
  • | 9e9805b6 - Minor fix (8 weeks ago)
  • | 2548cccd - Add a little more verbosity (8 weeks ago)
  • | 8cf79b73 - Allow a bias in the ResolutionModel while using RooCubicSplineKnot (8 weeks ago)
  • | 86c36e4c - Try to work around current problems with RooTreeDataStore (8 weeks ago)
  • | d6b53c14 - Try to get RooTreeDataStore working again, reorder toy configs (4 months ago)
  • | 1ecaddbe - Tell editor how this should be indented (4 months ago)
  • | f7c0eaad - Allow to generrate IpatiaJohnsonSU "withWidthRatio" (4 months ago)
  • | dbc78d8d - Use a singleton random generator (4 months ago)
  • | c34af8b2 - Pass seed to poisson gen everywhere (4 months ago)
  • | a7386cbb - Restructure toy fitting configs (4 months ago)
  • | ab34c896 - Add documentation, set range of calibrated obs (4 months ago)
  • | 9c4d3411 - Properly shift decay time (4 months ago)
  • | 2d90e80c - Try to burn in decay time error (4 months ago)
  • | 10508173 - Remove trailing spaces (4 months ago)
  • | abc2e4a2 - Try to fix io problems after latest urania version (4 months ago)
  • | bf81fb6a - Use copy constructor instead of deepcopy, default AutoBinned again (4 months ago)
  • | 1ae392c5 - Re-silence compiler: Fix unused parameter warning (4 months ago)
  • | 42202cb3 - Use lowerTString method for GetModeCapital (4 months ago)
  • | 63e943ec - Merge remote-tracking branch 'origin/master' into dev_kheinick_sfit (4 months ago) |\
    | |/ | * 5716920b - fix assignment as condition (5 months ago) | * 146a75f1 - fix equality comparison result unused (5 months ago) | * f0094e7c - remove variable self assignment (5 months ago) | * 579e703f - fix condition in nested loop (5 months ago) | * 91854457 - avoid empty body in for loop (improve readability) (5 months ago) | * c942149f - fixed deprecated implicit operator= (5 months ago) | * c8f677e4 - Use unique_ptr instead of deprecated auto_ptr (5 months ago) | * e9c11927 - Initialize pointers (5 months ago) | * b13c0950 - Fix set but unused warnings (5 months ago) | * b1298340 - Fix unused argument warning (5 months ago) | * cfb07c3f - Fix overloaded-virtual warning (5 months ago) | * f4a309a1 - Avoir macro redefinition warning (5 months ago) | * 74f9a2c9 - Add missing virtual destructor (5 months ago) | * 46870bae - Merge branch 'Lb2Dsp' into 'master' (5 months ago) | |
    | | * 4bca70b8 - Changed MassFitUtils to be able to make Lb2Dsp Signal workspace (6 months ago) | * | a3a24d4b - (origin/B2DXFitter_DecRateCoeff) removing DecRateCoeff (5 months ago) | * | 1b9961ee - Replace RooFIter with suggested RooAbsCollection iterators (6 months ago) | |/ | * 9249c86f - update of lb2lcpi for run1, allowing sim/combined fit to run1 and run2 (8 months ago) | * 9c26a8ea - small fix for plotting (9 months ago) | * 0f596398 - prepare workspace flexibility, more config files and readme updates (9 months ago) | * ea01da37 - Merge branch 'adudziak_lb2dsp' into 'master' (9 months ago) | |
    | | * e06e5df9 - Lb2Dsp modifications (9 months ago) | * | 3531f759 - update for Bs2DsPi DGs (9 months ago) | |/
  • | e454c4a7 - Dont force merged polarities (4 months ago)
  • | a5cff6f7 - Fix toyFactory script (4 months ago)
  • | 1cfe84ce - slightly tune plot split (4 months ago)
  • | fa5544e5 - Start being case-insensitive + code formatting (5 months ago)
  • | a54f3343 - Mainly code formatting for readability (5 months ago)
  • | b59a93b8 - Minor code formatting + try to homogenize mdfit+toyfactory naming (5 months ago)
  • | a0904ba7 - (temporarily?) remove disabled/unmaintained toy switch (5 months ago)
  • | e4a327fa - Continued preparation for run2 DsK/Pi analysis (5 months ago)
  • | 977c0bc9 - Remove dead code (7 months ago)
  • | 2383c608 - Disable hardcoded tauerr calib (7 months ago)
  • | 1829d406 - Use more naive (but working) blinding (7 months ago)
  • | c5f0f8ab - Print verbose when running toyfit, different blinding string for toys (8 months ago) |/
  • ae82846b - Merge remote-tracking branch 'origin/master' into dev_kheinick_sfit (9 months ago) |
    | * 62020f0d - update configs for Bs2DsPi (9 months ago)
  • | b9e0c17b - Merge remote-tracking branch 'origin/master' into dev_kheinick_sfit (9 months ago) |\
    | |/ | * 4a6af512 - update config files for MC (9 months ago)
  • | d096f1ec - Merge remote-tracking branch 'origin/master' into dev_kheinick_sfit (9 months ago) |\
    | |/ | * 602e563a - update confing file for the final MDFit (9 months ago)
  • | 92c30e60 - Merge branch 'dev-kheinick-hotfix' into dev_kheinick_sfit (9 months ago) |\
    | |/ | * d4e910e9 - (origin/dev-kheinick-hotfix, dev-kheinick-hotfix) Fix plotting of non-sim fit (9 months ago) | * 4539fda7 - Reenable verbosity-blinding (for other fits) (9 months ago)
  • | fbad8d33 - Merge branch 'master' into dev_kheinick_sfit (9 months ago) |\
    | |/ | * a57a774b - (origin/dev_B2DXFitters_run2_kheinick, dev_B2DXFitters_run2_kheinick) Add WIP config for DsPi toyFactory and modify scripts for run2 namings (9 months ago) | * 006ed60b - Suppress warnings from nonexisting default RooFit arguments (9 months ago) | * 8abb8614 - Add ToyGenerator config and update toyFactory to run in current state (9 months ago) | * 727305bb - Add RooBlinding specifically to deltams (9 months ago)
  • | 193c6a25 - Add 2018 data to sfit config, update acceptance, ft (10 months ago)
  • | 19c1fe4e - Prepare sfit scripts for more generic simfit (10 months ago) |/
  • b652380b - Add _log suffix if plots are log-ged (10 months ago)
  • b6ab8eca - Add argument to allow config overrides via command line (10 months ago)
  • 2c986a28 - Add clone_* feature capability to circumvent default transformations (10 months ago)
  • e8d36cef - Allow specification of plot dir via cmdline (10 months ago)
  • 9d72a9e5 - Allow to define minimizer (10 months ago)
  • 933d6a3e - Allow to plot with mode == all (10 months ago)
  • b7ec7c4d - Fix prefix argument (10 months ago)
  • 81f2a1b1 - Add plot prefix, refactor plot name part, tune naming for snakemake (10 months ago)
  • c52bea67 - Minor refactor of runMDFitter lumi parsing (10 months ago)
  • e2573796 - Allow to specify numCPU; refactor parts of config parsing (10 months ago)
  • 74ef6498 - Prevent segfault with newer root versions (10 months ago)
  • 087b838e - Print filename when saying it will print a file (10 months ago)
  • 369a4d72 - Allow to define a prefix for temporary files. (10 months ago)
  • a87431ed - Dont let ROOT eat the cmdline options, fix misleading errors (11 months ago)
  • b3f85c7e - Add gitignore specifically for scripts directory (11 months ago)
  • 8576b962 - Reorder lines to prevent undefined errors (11 months ago)
  • fe22a98c - Fix undefined variables... (11 months ago)
Edited by Kevin Heinicke

Merge request reports