Master

• The inputs are now on json format

• A minimal of two json inputs by channel + the path to the fitdiagnostics is necessary

  • I think now ready to be added on law workflow as well (with the user giving the json inputs)

• The tables of yields (not yet uncertainties) are saved along the plots in dictionary format

  • To be passed to pandas to make manipulate tables easier (?)
  • @mrieger this is what you was using doing more manually, no? Does that help?
    • Take a look on the examples on $DHI_results/Prefits/Plots_demo/results, where $DHI_results is our EOS repo for combination results.

• it also saves a final json file by plot that allow reproduce only that plot with the main script with local modifications to it (instead of a large list as it was before)

Standalone example, provided that the datacards_run2 repo is up to date

python dhi/scripts/postfit_plots.py \
--output_folder $DHI_results/Prefits/Plots_demo/results \
--plot_options_dict $DHI_DATACARDS_RUN2/bbbb/plot_options.json \
--overwrite "$DHI_DATACARDS_RUN2/bbbb/plots_list.json" \
--overwrite_fitdiag $DHI_results/Prefits/Plots_demo/all_bbbb_wwbb_bbzz4l_fitdiagnostics.root


python dhi/scripts/postfit_plots.py \
--output_folder $DHI_results/Prefits/Plots_demo/results \
--plot_options_dict $DHI_DATACARDS_RUN2/bbzz4l/plot_options.json \
--overwrite "$DHI_DATACARDS_RUN2/bbzz4l/plots_list.json" \
--overwrite_fitdiag $DHI_results/Prefits/Plots_demo/all_bbbb_wwbb_bbzz4l_fitdiagnostics.root

To make the other channels dictionary in $DHI_DATACARDS_RUN2 will deppend on the acceptance of this MR.

dhi/scripts/README_postfit_plots.md

 -- to pass some info to online documentation in a proper format
 
-# inputs
+We use the plotter in standalone mode as:
 
-The list of user inputs to the plotter-only is:
-- fitDiagnosis (result of the PostFitShapes task)
-  - Made from a cards combination in which subcategories should be merged using a fixed naming convention for bins, eg
-  """
-  combineCards.py \
-  bbWW_SL_2017=datacard_hh_bb1l_hh_bb1l_cat_jet_2BDT_Wjj_simple_SM_Res_allReco_bbWW_nonresNLO_none_75_multisig_2017.txt \
-  bbWW_DL_2017=datacard_hh_bb2l_hh_bb2l_OS_SM_plainVars_inclusive_bbWW_nonresNLO_none_45_noSL_2017.txt>\
-  datacard_SL_DL_bbWW_nonresNLO_none_45_75_allSig_2017_noSingleH_naming.txt
-  """
-- dictionaries with plot options, minding the above-mentioned fixed naming convention for bins
+```
+python dhi/scripts/postfit_plots.py \
+--output_folder /where/save/the/plots/ \
+--plot_options_dict /where/is/plot_options.json
+```
 
-## Explaining the dictionary
+There are additional options, like eg  `--unblind` (the default is blinded). Please use `--help` to see further options.
 
+# Explaining the dictionary
 
-This is an example of dictionary booking one plot:
+This is an example of dictionary booking one plot, that should be saved in a .json file/format as `/where/is/plot_options.json` (on the example comand line above):
 
 ```
 {
-  "plotXX" : {
-      "fitdiagnosis"       : "/where/is/the/fitdiagnosis.root",
-      "bin_name_original"  : "none",
-      "datacard_original"  : "none",
-      "minY"               : 0.1,
-      "maxY"               : 100000000000.,
-      "minYerr"            : -0.28,
-      "maxYerr"            : 0.28,
-      "useLogPlot"         : True,
-      "era"                : 2016 ,
-      "labelX"             : "bla bla",
-      "header_legend"          : "bla bla can be latex",
-      "number_columns_legend" : 3,
-      "align_cats" : ["ch1","ch2",],
-      "align_cats_labels" : [["ch1 bla", "ch1 more details"], ["ch2 bla", "ch2 more details"]],
-      "align_cats_labelsX" : [3, 13],
-      "cats_labels_height" : 1000000.,
-      "merged_eras_fit" : False,
-      "procs_plot_options_bkg" : OrderedDict(
-          [
-          ("Other_bbWW",       {"color" : 205, "fillStype"   : 1001, "label" : "others"           , "make border" : True}),
-          ("VV",          {"color" : 823, "fillStype"   : 1001, "label" : "ZZ + WZ"          , "make border" : True}),
-          ("ST",         {"color" : 822, "fillStype" : 1001, "label" : "single top"         , "make border" : True}),
-          ("Fakes",  {"color" :  12, "fillStype" : 3345, "label" : "Fakes"  , "make border" : True}),
-          ("DY",          {"color" : 221, "fillStype" : 1001, "label" : "DY"         , "make border" : True}),
-          ("WJets",           {"color" : 5, "fillStype" : 1001, "label" : 'W + jets'   , "make border" : True}),
-          ("TT",          {"color" : 17, "fillStype"  : 1001, "label" : 't#bar{t} + jets'   , "make border" : True})
-          ]
-      ),
-      'procs_plot_options_sig' : OrderedDict(
-       [
-       ('ggHH_kl_1_kt_1' ,      {'color' : 5, 'fillStype'  : 3351, 'label' : 'GGF HH SM', 'scaleBy' : 1.}),
-       ('ggHH_kl_5_kt_1',       {'color' : 221, 'fillStype'  : 3351, 'label' : 'GGF HH #kappa#lambda = 5', 'scaleBy' : 1.}),
-       ('ggHH_kl_2p45_kt_1',    {'color' : 2, 'fillStype'  : 3351, 'label' : 'GGF HH #kappa#lambda = 2.45', 'scaleBy' : 1.}),
-       ('qqHH_CV_1_C2V_1_kl_1', {'color' : 8, 'fillStype'  : 3351, 'label' : 'VBF HH SM', 'scaleBy' : 1.}),
-       ]
-       ),
-  },
-}
+  "plotXX_2018":
+  {
+    "fitdiagnosis": "/where/is/the/fitdiagnosis.root",
+    "datacard_original": "none",
+    "bin_name_original": "none",
+    "procs_plot_options_bkg": "plot_options_BKG_colors.json",
+    "procs_plot_options_sig": "plot_options_sig_colors.json",
+    "header_legend": "bla bla can be latex",
+    "labelX": "bla bla can be latex",
+    "maxY": 10000.0,
+    "minY": 0.001,
+    "maxYerr": 20.01,
+    "minYerr": -5.01,
+    "maxYerr_postfit": 15.01,
+    "minYerr_postfit": -15.01,
+    "cats_labels_height": 8.0,
+    "number_columns_legend": 3,
+    "useLogPlot": true,
+    "era": 2018,
+    "align_cats": ["ch1","ch2",],
+    "align_cats_labelsX": [6, 20],
+    "align_cats_labels": [["ch1 bla", "ch1 more details"], ["ch2 bla", "ch2 more details"]],
+    "merged_eras_fit": false
+    }
+  }
 ```
 
+## Plot options
+
+The name of the file containing the plot will contain the main key, in the example above `"plotXX_2018"`.
+
 The keys of the dictionary are the names of the bins to plot distributions for (declared on the combineCards command). The entries are bellow:
 
 - "datacard_original": the datacard.root file with shapes for that bin (what goes along with datacard.txt)
 - "bin_name_original" : For some analyses the bin can be in a folder inside the datacard.root , if this is the case put the name of this folder. If there is no internal folder, just put "none".
 - Y-axis of the shapes distributions (top pad) : "minY" / "maxY"
-- Y-axis of the bottom pad : "minYerr", "maxYerr"
+- Y-axis of the bottom pad for prefit plot: "minYerr", "maxYerr"
+- Y-axis of the bottom pad for postfit plot: "minYerr_postfit", "maxYerr_postfit" (if it is not given it will use the ones for prefit, defined above)
 - "useLogPlot", for the shapes distributions (top pad)
 - "era", to decide which lumi put on plot header
 - "labelX" is the variable being plotted
 - options for legends "header_legend", "number_columns_legend"
+- "procs_plot_options_bkg" is the name of the json file containing the list of BKG processes to be drawn and options for plotting. See point bellow.
+- "procs_plot_options_sig" is the name of the json file containing the list of signal processes to be drawn and options for plotting. See point bellow.
+-  "align_cats" is a list of bins/channels that we want drawn one after another in the same plot
+  - It can be one, that is the most common use
+-   "align_cats_labels" are the labels for "align_cats"
+-  "align_cats_labelsX" : the X positions for the labels "align_cats_labels"
+-   "cats_labels_height" : the Y positions for the labels "align_cats_labels"
+- "merged_eras_fit" : if true it will try to read the single H processes with era in name (e.g. "ttH_2017_hbb" instead of "ttH_hbb")
+
+
+- TODO: make the dictionary example on datacards_run2 repo, and make the path to the original datacard.root or relative to the datacards_run2 when I do the example with cards from datacards_run2 repo
+
+### Processes dictionaries
+
+There are mentions for two files json files, if just the plain name of a file it will assume that they are in the same folder than `/where/is/plot_options.json`, if it is a path it will use the string as an absolute path.
+
+#### Explaining procs_plot_options_bkg
+
 - "procs_plot_options_bkg" is the list of processes to be stacked in the plot, with options
   - the keys are the exact names of the processes on datacard.root
   - the order you write there is going to be the order of the stack and of the legend entries
   - if you put a process that is not on the datacard it will skip it (not break)
   - if you do not put a process that is on the datacard, it will not add it to the stack (useful to negligible processes)
   - How to merge processes: put them subsequently. Put the desired  "label" and "make border"==True only on the last of the list, on the others put "label"="none" and "make border"=False. Example here [link from datacards_run2 uploaded example].
-- "procs_plot_options_sig" is the list of signals (processes to not be added to the stack)
+
+An explicit example is bellow:
+
+```
+{
+  "bbzz4l_others" : {"color": 205, "fillStype": 1001, "make border": true, "label": "others"},
+  "ZX"            : {"color": 208, "fillStype": 1001, "make border": true, "label": "ZX"},
+  "ttZ"           : {"color": 9, "fillStype": 1001, "make border": true, "label": "ttZ"},
+  "ggZZ"          : {"color": 822, "fillStype": 1001, "make border": true, "label": "ggZZ"},
+  "qqZZ"          : {"color": 221, "fillStype": 1001, "make border": true, "label": "qqZZ"}
+}
+```
+
+#### Explaining procs_plot_options_sig
+
+- "procs_plot_options_sig" is the list of signals (processes to be drawn but not added to the stack)
   - each entry will be added as one overlaid histogram, summing up the processes listed in "processes" and scaled by "scaleBy"
--  "align_cats" is a list of bins/channels that we want drawn one after another in the same plot
--   "align_cats_labels" are the labels for "align_cats"
--  "align_cats_labelsX" : the X positions for the labels "align_cats_labels"
--   "cats_labels_height" : the Y positions for the labels "align_cats_labels"
-- "merged_eras_fit" : if true it will try to read the single H processes with era in name (e.g. "ttH_2017_hbb" instead of "ttH_hbb")
+  - the order you write there is going to be the legend entries
 
-- TODO: make the dictionary example on datacards_run2 repo, and make the path to the original datacard.root or relative to the datacards_run2 when I do the example with cards from datacards_run2 repo
+An explicit example is bellow:
+
+```
+{
+  "ggHH_kl_1_kt_1"       : {"color": 5, "fillStype": 3351, "scaleBy": 1.0, "label": "GGF SM"},
+  "ggHH_kl_5_kt_1"       : {"color": 221, "fillStype": 3351, "scaleBy": 1.0, "label": "GGF #kappa#lambda = 5"},
+  "ggHH_kl_2p45_kt_1"    : {"color": 2, "fillStype": 3351, "scaleBy": 1.0, "label": "GGF #kappa#lambda = 2.45"},
+  "qqHH_CV_1_C2V_1_kl_1" : {"color": 8, "fillStype": 3351, "scaleBy": 1.0, "label": "VBF SM"},
+  "qqHH_CV_1_C2V_2_kl_1" : {"color": 6, "fillStype": 3351, "scaleBy": 1.0, "label": "VBF c2V = 2"}
+}
+```
 
-## Common options that we may like to change
+### Common options that we may like to change
 
 Right now:
-- single H processes are added to the stack of backgrounds, as first entry
 - the total band is the sum of BKGs only
 
-We would like:
-- use the fact that there is a naming convention to signal and avoiod the need of "procs_plot_options_sig" on the dictionary (right nonow we did not agreed on the suffixes, so that is not so eazy)
-
-## Pending/suggestions
+### Pending/suggestions
 
 - make the options flow in law scheme
 
@@ -104,7 +126,21 @@ We would like:
 
 - save the log of the plot (what the script prints) along with the plot.pdf/root/png, so the person running can check it and the person implementing a new dictionary can debug mistakes/lists of processes
 
-- I still did not removed completely the functionality to plot different categories one after the other in a same canvas -- I will remove when I am sure nobody will use == after some tests
 
-## Task missing:
+
+## On the shapes input (FitDiagnostics)
+
+The list of user inputs to the plotter-only is:
+- fitDiagnosis (result of the PostFitShapes task)
+  - Made from a cards combination in which subcategories should be merged using a fixed naming convention for bins, that you will enter on the dictionary as "align_cats", eg (following the example dictionary above)
+  """
+  combineCards.py \
+  ch1=datacard_ch1.txt \
+  ch2=datacard_ch2.txt>\
+  datacard_combo.txt
+  """
+- dictionaries with plot options, minding the above-mentioned fixed naming convention for bins
+
+
+# Task missing:
 - make prefit/postfit table of yields with uncertainties