diff --git a/docs/DL1r-instructions.md b/docs/DL1r-instructions.md index df7de11fa40cd9e5a39b5c5f0656a51c499fdb73..462418498f49a21f577fa4b46eaa7efacde1e759 100644 --- a/docs/DL1r-instructions.md +++ b/docs/DL1r-instructions.md @@ -41,52 +41,48 @@ model_file: # Add training file train_file: ////train_file.h5 +# Defining templates for the variable cuts +.variable_cuts_ttbar: &variable_cuts_ttbar + variable_cuts: + - pt_btagJes: + operator: "<=" + condition: 2.5e5 + +.variable_cuts_zpext: &variable_cuts_zpext + variable_cuts: + - pt_btagJes: + operator: ">" + condition: 2.5e5 + + # Add validation files validation_files: ttbar_r21_val: path: /MC16d_hybrid_odd_100_PFlow-no_pTcuts-file_0.h5 label: "$t\\bar{t}$ Release 21" - variable_cuts: - - pt_btagJes: - operator: "<=" - condition: 250000 + <<: *variable_cuts_ttbar zprime_r21_val: path: /MC16d_hybrid-ext_odd_0_PFlow-no_pTcuts-file_0.h5 label: "$Z'$ Release 21" - variable_cuts: - - pt_btagJes: - operator: ">" - condition: 250000 + <<: *variable_cuts_zpext test_files: ttbar_r21: path: ////ttbar_r21_test_file.h5 - variable_cuts: - - pt_btagJes: - operator: "<=" - condition: 250000 + <<: *variable_cuts_ttbar ttbar_r22: path: ////ttbar_r22_test_file.h5 - variable_cuts: - - pt_btagJes: - operator: "<=" - condition: 250000 + <<: *variable_cuts_ttbar zpext_r21: path: ////zpext_r21_test_file.h5 - variable_cuts: - - pt_btagJes: - operator: ">" - condition: 250000 + <<: *variable_cuts_zpext zpext_r22: path: ////zpext_r22_test_file.h5 - variable_cuts: - - pt_btagJes: - operator: ">" - condition: 250000 + <<: *variable_cuts_zpext # Path to Variable dict used in preprocessing var_dict: ///DL1r_Variables.yaml @@ -230,6 +226,8 @@ The different options are briefly explained here: | `train_file` | String | Necessary | Path to the training sample. This is given by the `preprocessing` step of Umami | | `validation_files` | Dict | Optional | Here you can define different validation samples that are used in the training and the `plotting_epoch_performance.py` script. Those validation samples need to be defined in a dict structure shown in the example. The name of the dict entry is relevant and is the unique identifier for this sample (DO NOT USE IT MULTIPLE TIMES). `path` gives the path to the file. | | `test_files` | Dict | Optional | Here you can define different test samples that are used in the [`evaluate_model.py`](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/umami/evaluate_model.py). Those test samples need to be defined in a dict structure shown in the example. The name of the dict entry is relevant and is the unique identifier in the results file which is produced by the [`evaluate_model.py`](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/umami/evaluate_model.py). `Path` gives the path to the file. For test samples, all samples from the training-dataset-dumper can be used without preprocessing although the preprocessing of Umami produces test samples to ensure orthogonality of the jets with respect to the train sample. | +| `path` | String | Necessary | Path to the validation/test file which is to be used. Using wildcards is possible. | +| `variable_cuts` | Dict | Optional | Dict of cuts which are applied when loading the different test files. Only jet variables can be cut on. These are in this example defined as templates for the different samples types. | | `var_dict` | String | Necessary | Path to the variable dict used in the `preprocess_config` to produce the train sample. | | `exclude` | List | Necessary | List of variables that are excluded from training. Only compatible with DL1r training. To include all, just give an empty list. | | `NN_structure` | None | Necessary | A dict where all important information for the training are defined. | @@ -266,7 +264,6 @@ The different options are briefly explained here: | `tagger` | List | Necessary | List of taggers used for comparison. This needs to be a list of string or a single string. The name of the taggers must be same as in the evaluation file. For example, if the DL1d probabilities in the test samples are called `DL1dLoose20210607_pb`, the name you need to add to the list is `DL1dLoose20210607`. | | `frac_values_comp` | Dict | Necessary | Dict with the fraction values for the comparison taggers. For all flavour (except the main flavour), you need to add values here which add up to one. | | `frac_values` | Dict | Necessary | Dict with the fraction values for the freshly trained tagger. For all flavour (except the main flavour), you need to add values here which add up to one. | -| `variable_cuts` | Dict | Necessary | Dict of cuts which are applied when loading the different test files. Only jet variables can be cut on. | | `WP` | Float | Necessary | Working point which is used in the validation and evaluation. | | `eff_min` | Float | Optional | Minimal main class efficiency considered for ROC. | diff --git a/docs/Dips-instructions.md b/docs/Dips-instructions.md index 4c81e76687927cef4703193e24a0c241c4c1bc36..9a4fd343de3d74b8df3523afddaa325c1a7db4ef 100644 --- a/docs/Dips-instructions.md +++ b/docs/Dips-instructions.md @@ -34,52 +34,48 @@ model_file: # Add training file train_file: ////train_file.h5 +# Defining templates for the variable cuts +.variable_cuts_ttbar: &variable_cuts_ttbar + variable_cuts: + - pt_btagJes: + operator: "<=" + condition: 2.5e5 + +.variable_cuts_zpext: &variable_cuts_zpext + variable_cuts: + - pt_btagJes: + operator: ">" + condition: 2.5e5 + + # Add validation files validation_files: ttbar_r21_val: path: /MC16d_hybrid_odd_100_PFlow-no_pTcuts-file_0.h5 label: "$t\\bar{t}$ Release 21" - variable_cuts: - - pt_btagJes: - operator: "<=" - condition: 250000 + <<: *variable_cuts_ttbar zprime_r21_val: path: /MC16d_hybrid-ext_odd_0_PFlow-no_pTcuts-file_0.h5 label: "$Z'$ Release 21" - variable_cuts: - - pt_btagJes: - operator: ">" - condition: 250000 + <<: *variable_cuts_zpext test_files: ttbar_r21: path: ////ttbar_r21_test_file.h5 - variable_cuts: - - pt_btagJes: - operator: "<=" - condition: 250000 + <<: *variable_cuts_ttbar ttbar_r22: path: ////ttbar_r22_test_file.h5 - variable_cuts: - - pt_btagJes: - operator: "<=" - condition: 250000 + <<: *variable_cuts_ttbar zpext_r21: path: ////zpext_r21_test_file.h5 - variable_cuts: - - pt_btagJes: - operator: ">" - condition: 250000 + <<: *variable_cuts_zpext zpext_r22: path: ////zpext_r22_test_file.h5 - variable_cuts: - - pt_btagJes: - operator: ">" - condition: 250000 + <<: *variable_cuts_zpext # Path to Variable dict used in preprocessing var_dict: ///Dips_Variables.yaml @@ -187,6 +183,8 @@ The different options are briefly explained here: | `train_file` | String | Necessary | Path to the training sample. This is given by the `preprocessing` step of Umami | | `validation_files` | Dict | Optional | Here you can define different validation samples that are used in the training and the `plotting_epoch_performance.py` script. Those validation samples need to be defined in a dict structure shown in the example. The name of the dict entry is relevant and is the unique identifier for this sample (DO NOT USE IT MULTIPLE TIMES). `path` gives the path to the file. | | `test_files` | Dict | Optional | Here you can define different test samples that are used in the [`evaluate_model.py`](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/umami/evaluate_model.py). Those test samples need to be defined in a dict structure shown in the example. The name of the dict entry is relevant and is the unique identifier in the results file which is produced by the [`evaluate_model.py`](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/umami/evaluate_model.py). `Path` gives the path to the file. For test samples, all samples from the training-dataset-dumper can be used without preprocessing although the preprocessing of Umami produces test samples to ensure orthogonality of the jets with respect to the train sample. | +| `path` | String | Necessary | Path to the validation/test file which is to be used. Using wildcards is possible. | +| `variable_cuts` | Dict | Optional | Dict of cuts which are applied when loading the different test files. Only jet variables can be cut on. These are in this example defined as templates for the different samples types. | | `var_dict` | String | Necessary | Path to the variable dict used in the `preprocess_config` to produce the train sample. | | `exclude` | List | Necessary | List of variables that are excluded from training. Only compatible with DL1r training. To include all, just give an empty list. | |`tracks_name`| String| Necessary* | Name of the tracks data-set to use for training and evaluation, default is "tracks".
* ***This option is necessary when using tracks, but, when working with old preprpocessed files (before January 2022) this option has to be removed form the config file to ensure compatibility*** | @@ -224,7 +222,6 @@ The different options are briefly explained here: | `tagger` | List | Necessary | List of taggers used for comparison. This needs to be a list of string or a single string. The name of the taggers must be same as in the evaluation file. For example, if the DL1d probabilities in the test samples are called `DL1dLoose20210607_pb`, the name you need to add to the list is `DL1dLoose20210607`. | | `frac_values_comp` | Dict | Necessary | Dict with the fraction values for the comparison taggers. For all flavour (except the main flavour), you need to add values here which add up to one. | | `frac_values` | Dict | Necessary | Dict with the fraction values for the freshly trained tagger. For all flavour (except the main flavour), you need to add values here which add up to one. | -| `variable_cuts` | Dict | Necessary | Dict of cuts which are applied when loading the different test files. Only jet variables can be cut on. | | `WP` | Float | Necessary | Working point which is used in the validation and evaluation. | | `Calculate_Saliency` | Bool | Optional | Decide, if the saliency maps are calculated or not. This takes a lot of time and resources! | diff --git a/examples/DL1r-PFlow-Training-config.yaml b/examples/DL1r-PFlow-Training-config.yaml index a87c1a2d4b11521df279e962fe68570b8a3d647d..71b4978576e986d28c4001722cdca20c24c28b36 100644 --- a/examples/DL1r-PFlow-Training-config.yaml +++ b/examples/DL1r-PFlow-Training-config.yaml @@ -9,52 +9,47 @@ model_file: # Add training file train_file: /PFlow-hybrid-preprocessed_shuffled.h5 +# Defining templates for the variable cuts +.variable_cuts_ttbar: &variable_cuts_ttbar + variable_cuts: + - pt_btagJes: + operator: "<=" + condition: 2.5e5 + +.variable_cuts_zpext: &variable_cuts_zpext + variable_cuts: + - pt_btagJes: + operator: ">" + condition: 2.5e5 + # Add validation files validation_files: ttbar_r21_val: path: /MC16d_hybrid_odd_100_PFlow-no_pTcuts-file_0.h5 label: "$t\\bar{t}$ Release 21" - variable_cuts: - - pt_btagJes: - operator: "<=" - condition: 250000 + <<: *variable_cuts_ttbar zprime_r21_val: path: /MC16d_hybrid-ext_odd_0_PFlow-no_pTcuts-file_0.h5 label: "$Z'$ Release 21" - variable_cuts: - - pt_btagJes: - operator: ">" - condition: 250000 + <<: *variable_cuts_zpext test_files: ttbar_r21: path: /MC16d_hybrid_odd_100_PFlow-no_pTcuts-file_1.h5 - variable_cuts: - - pt_btagJes: - operator: "<=" - condition: 250000 + <<: *variable_cuts_ttbar ttbar_r22: path: /MC16d_hybrid-r22_odd_100_PFlow-no_pTcuts-file_1.h5 - variable_cuts: - - pt_btagJes: - operator: "<=" - condition: 250000 + <<: *variable_cuts_ttbar zpext_r21: path: /MC16d_hybrid-ext_odd_0_PFlow-no_pTcuts-file_1.h5 - variable_cuts: - - pt_btagJes: - operator: ">" - condition: 250000 + <<: *variable_cuts_zpext zpext_r22: path: /MC16d_hybrid-r22-ext_odd_0_PFlow-no_pTcuts-file_1.h5 - variable_cuts: - - pt_btagJes: - operator: ">" - condition: 250000 + <<: *variable_cuts_zpext # Path to Variable dict used in preprocessing var_dict: umami/configs/DL1r_Variables.yaml @@ -147,18 +142,6 @@ Eval_parameters_validation: "ujets": 0.982, } - # Cuts which are applied to the different datasets used for evaluation - variable_cuts: - validation_file: - - pt_btagJes: - operator: "<=" - condition: 250000 - - add_validation_file: - - pt_btagJes: - operator: ">" - condition: 250000 - # A list to add available variables to the evaluation files add_variables_eval: ["actualInteractionsPerCrossing"] diff --git a/examples/Dips-Cond-Att-PFlow-Training-config.yaml b/examples/Dips-Cond-Att-PFlow-Training-config.yaml index aabad9b1838ba148a06b3a02f17ac1ddf60711fb..cbc2145d5cd4e98a8d5cfa6d61d8ef65c4c3bdc6 100644 --- a/examples/Dips-Cond-Att-PFlow-Training-config.yaml +++ b/examples/Dips-Cond-Att-PFlow-Training-config.yaml @@ -9,52 +9,47 @@ model_file: # Add training file train_file: /PFlow-hybrid-preprocessed_shuffled.h5 +# Defining templates for the variable cuts +.variable_cuts_ttbar: &variable_cuts_ttbar + variable_cuts: + - pt_btagJes: + operator: "<=" + condition: 2.5e5 + +.variable_cuts_zpext: &variable_cuts_zpext + variable_cuts: + - pt_btagJes: + operator: ">" + condition: 2.5e5 + # Add validation files validation_files: ttbar_r21_val: path: /MC16d_hybrid_odd_100_PFlow-no_pTcuts-file_0.h5 label: "$t\\bar{t}$ Release 21" - variable_cuts: - - pt_btagJes: - operator: "<=" - condition: 250000 + <<: *variable_cuts_ttbar zprime_r21_val: path: /MC16d_hybrid-ext_odd_0_PFlow-no_pTcuts-file_0.h5 label: "$Z'$ Release 21" - variable_cuts: - - pt_btagJes: - operator: ">" - condition: 250000 + <<: *variable_cuts_zpext test_files: ttbar_r21: path: /MC16d_hybrid_odd_100_PFlow-no_pTcuts-file_1.h5 - variable_cuts: - - pt_btagJes: - operator: "<=" - condition: 250000 + <<: *variable_cuts_ttbar ttbar_r22: path: /MC16d_hybrid-r22_odd_100_PFlow-no_pTcuts-file_1.h5 - variable_cuts: - - pt_btagJes: - operator: "<=" - condition: 250000 + <<: *variable_cuts_ttbar zpext_r21: path: /MC16d_hybrid-ext_odd_0_PFlow-no_pTcuts-file_1.h5 - variable_cuts: - - pt_btagJes: - operator: ">" - condition: 250000 + <<: *variable_cuts_zpext zpext_r22: path: /MC16d_hybrid-r22-ext_odd_0_PFlow-no_pTcuts-file_1.h5 - variable_cuts: - - pt_btagJes: - operator: ">" - condition: 250000 + <<: *variable_cuts_zpext # Path to Variable dict used in preprocessing var_dict: /umami/umami/configs/Dips_Variables.yaml @@ -158,18 +153,6 @@ Eval_parameters_validation: # Charm fraction value used for evaluation of the trained model frac_values: {"cjets": 0.018, "ujets": 0.982} - # Cuts which are applied to the different datasets used for evaluation - variable_cuts: - validation_file: - - pt_btagJes: - operator: "<=" - condition: 250000 - - add_validation_file: - - pt_btagJes: - operator: ">" - condition: 250000 - # Working point used in the evaluation WP: 0.77 diff --git a/examples/Dips-PFlow-Training-config.yaml b/examples/Dips-PFlow-Training-config.yaml index b9fedaaa81e34d4c8948324e592b09a41d983a20..b9f373f17243d17634d3e4eb37dee2ce7e67d048 100755 --- a/examples/Dips-PFlow-Training-config.yaml +++ b/examples/Dips-PFlow-Training-config.yaml @@ -9,52 +9,47 @@ model_file: # Add training file train_file: /PFlow-hybrid-preprocessed_shuffled.h5 +# Defining templates for the variable cuts +.variable_cuts_ttbar: &variable_cuts_ttbar + variable_cuts: + - pt_btagJes: + operator: "<=" + condition: 2.5e5 + +.variable_cuts_zpext: &variable_cuts_zpext + variable_cuts: + - pt_btagJes: + operator: ">" + condition: 2.5e5 + # Add validation files validation_files: ttbar_r21_val: path: /MC16d_hybrid_odd_100_PFlow-no_pTcuts-file_0.h5 label: "$t\\bar{t}$ Release 21" - variable_cuts: - - pt_btagJes: - operator: "<=" - condition: 250000 + <<: *variable_cuts_ttbar zprime_r21_val: path: /MC16d_hybrid-ext_odd_0_PFlow-no_pTcuts-file_0.h5 label: "$Z'$ Release 21" - variable_cuts: - - pt_btagJes: - operator: ">" - condition: 250000 + <<: *variable_cuts_zpext test_files: ttbar_r21: path: /MC16d_hybrid_odd_100_PFlow-no_pTcuts-file_1.h5 - variable_cuts: - - pt_btagJes: - operator: "<=" - condition: 250000 + <<: *variable_cuts_ttbar ttbar_r22: path: /MC16d_hybrid-r22_odd_100_PFlow-no_pTcuts-file_1.h5 - variable_cuts: - - pt_btagJes: - operator: "<=" - condition: 250000 + <<: *variable_cuts_ttbar zpext_r21: path: /MC16d_hybrid-ext_odd_0_PFlow-no_pTcuts-file_1.h5 - variable_cuts: - - pt_btagJes: - operator: ">" - condition: 250000 + <<: *variable_cuts_zpext zpext_r22: path: /MC16d_hybrid-r22-ext_odd_0_PFlow-no_pTcuts-file_1.h5 - variable_cuts: - - pt_btagJes: - operator: ">" - condition: 250000 + <<: *variable_cuts_zpext # Path to Variable dict used in preprocessing var_dict: /umami/umami/configs/Dips_Variables.yaml diff --git a/examples/evalute_comp_taggers.yaml b/examples/evalute_comp_taggers.yaml index 6548512e9ed62b3ee2e68c82bc7c284fc89cb32c..090ec5a51b0364b622e1451b36f35f9a378a22d7 100644 --- a/examples/evalute_comp_taggers.yaml +++ b/examples/evalute_comp_taggers.yaml @@ -4,34 +4,35 @@ model_name: Eval_results # Set the option to evaluate a freshly trained model to False evaluate_trained_model: False +# Defining templates for the variable cuts +.variable_cuts_ttbar: &variable_cuts_ttbar + variable_cuts: + - pt_btagJes: + operator: "<=" + condition: 2.5e5 + +.variable_cuts_zpext: &variable_cuts_zpext + variable_cuts: + - pt_btagJes: + operator: ">" + condition: 2.5e5 + test_files: ttbar_r21: path: ////ttbar_r21_test_file.h5 - variable_cuts: - - pt_btagJes: - operator: "<=" - condition: 250000 + <<: *variable_cuts_ttbar ttbar_r22: path: ////ttbar_r22_test_file.h5 - variable_cuts: - - pt_btagJes: - operator: "<=" - condition: 250000 + <<: *variable_cuts_ttbar zpext_r21: path: ////zpext_r21_test_file.h5 - variable_cuts: - - pt_btagJes: - operator: ">" - condition: 250000 + <<: *variable_cuts_zpext zpext_r22: path: ////zpext_r22_test_file.h5 - variable_cuts: - - pt_btagJes: - operator: ">" - condition: 250000 + <<: *variable_cuts_zpext # Values for the neural network NN_structure: diff --git a/examples/umami-PFlow-Training-config.yaml b/examples/umami-PFlow-Training-config.yaml index 944c4fe983747cea6e1ecd8b1a91116c24830d5c..25ce80629ae7312c36828813e119843a45161b45 100644 --- a/examples/umami-PFlow-Training-config.yaml +++ b/examples/umami-PFlow-Training-config.yaml @@ -9,52 +9,47 @@ model_file: # Add training file train_file: /PFlow-hybrid_70-test-preprocessed_shuffled.h5 +# Defining templates for the variable cuts +.variable_cuts_ttbar: &variable_cuts_ttbar + variable_cuts: + - pt_btagJes: + operator: "<=" + condition: 2.5e5 + +.variable_cuts_zpext: &variable_cuts_zpext + variable_cuts: + - pt_btagJes: + operator: ">" + condition: 2.5e5 + # Add validation files validation_files: ttbar_r21_val: path: /MC16d_hybrid_odd_100_PFlow-no_pTcuts-file_0.h5 label: "$t\\bar{t}$ Release 21" - variable_cuts: - - pt_btagJes: - operator: "<=" - condition: 250000 + <<: *variable_cuts_ttbar zprime_r21_val: path: /MC16d_hybrid-ext_odd_0_PFlow-no_pTcuts-file_0.h5 label: "$Z'$ Release 21" - variable_cuts: - - pt_btagJes: - operator: ">" - condition: 250000 + <<: *variable_cuts_zpext test_files: ttbar_r21: path: /MC16d_hybrid_odd_100_PFlow-no_pTcuts-file_1.h5 - variable_cuts: - - pt_btagJes: - operator: "<=" - condition: 250000 + <<: *variable_cuts_ttbar ttbar_r22: path: /MC16d_hybrid-r22_odd_100_PFlow-no_pTcuts-file_1.h5 - variable_cuts: - - pt_btagJes: - operator: "<=" - condition: 250000 + <<: *variable_cuts_ttbar zpext_r21: path: /MC16d_hybrid-ext_odd_0_PFlow-no_pTcuts-file_1.h5 - variable_cuts: - - pt_btagJes: - operator: ">" - condition: 250000 + <<: *variable_cuts_zpext zpext_r22: path: /MC16d_hybrid-r22-ext_odd_0_PFlow-no_pTcuts-file_1.h5 - variable_cuts: - - pt_btagJes: - operator: ">" - condition: 250000 + <<: *variable_cuts_zpext # Path to Variable dict used in preprocessing var_dict: /umami/umami-git/umami/configs/Umami_Variables.yaml diff --git a/umami/evaluate_model.py b/umami/evaluate_model.py index 407297577cb0f1cdcdb97e3edcd557eda27629c6..119b327f4287cbf70c49b90cd76c2cb8c8376235 100644 --- a/umami/evaluate_model.py +++ b/umami/evaluate_model.py @@ -110,16 +110,14 @@ def EvaluateModel( """ # Get train parameters - test_file_options = train_config.test_files[data_set_name] + test_set_config = train_config.test_files[data_set_name] Eval_params = train_config.Eval_parameters_validation class_labels = train_config.NN_structure["class_labels"] main_class = train_config.NN_structure["main_class"] frac_values_comp = Eval_params["frac_values_comp"] tracks_name = train_config.tracks_name var_cuts = ( - test_file_options["variable_cuts"] - if "variable_cuts" in test_file_options - else None + test_set_config["variable_cuts"] if "variable_cuts" in test_set_config else None ) # Print a warning that no variable cuts are used for the file @@ -342,16 +340,14 @@ def EvaluateModelDips( raise ValueError("You need to give an epoch which is to be evaluated!") # Get train parameters - test_file_options = train_config.test_files[data_set_name] + test_set_config = train_config.test_files[data_set_name] Eval_params = train_config.Eval_parameters_validation class_labels = train_config.NN_structure["class_labels"] main_class = train_config.NN_structure["main_class"] frac_values_comp = Eval_params["frac_values_comp"] tracks_name = train_config.tracks_name var_cuts = ( - test_file_options["variable_cuts"] - if "variable_cuts" in test_file_options - else None + test_set_config["variable_cuts"] if "variable_cuts" in test_set_config else None ) # Print a warning that no variable cuts are used for the file @@ -588,15 +584,13 @@ def EvaluateModelDL1( """ # Get train parameters - test_file_options = train_config.test_files[data_set_name] + test_set_config = train_config.test_files[data_set_name] Eval_params = train_config.Eval_parameters_validation class_labels = train_config.NN_structure["class_labels"] main_class = train_config.NN_structure["main_class"] frac_values_comp = Eval_params["frac_values_comp"] var_cuts = ( - test_file_options["variable_cuts"] - if "variable_cuts" in test_file_options - else None + test_set_config["variable_cuts"] if "variable_cuts" in test_set_config else None ) # Print a warning that no variable cuts are used for the file @@ -818,25 +812,31 @@ if __name__ == "__main__": # TODO Change this in python 3.10 if tagger_name == "dl1": logger.info("Start evaluating DL1 with test files...") - for test_file_i in training_config.test_files: + for ( + test_file_identifier, + test_file_config, + ) in training_config.test_files.items(): EvaluateModelDL1( args=parser_args, train_config=training_config, preprocess_config=preprocessing_config, - test_file=training_config.test_files[test_file_i]["path"], - data_set_name=test_file_i, - test_file_entry=test_file_i, + test_file=test_file_config["path"], + data_set_name=test_file_identifier, + test_file_entry=test_file_identifier, ) elif tagger_name in ("dips", "dips_cond_att"): logger.info("Start evaluating DIPS with test files...") - for test_file_i in training_config.test_files: + for ( + test_file_identifier, + test_file_config, + ) in training_config.test_files.items(): EvaluateModelDips( args=parser_args, train_config=training_config, preprocess_config=preprocessing_config, - test_file=training_config.test_files[test_file_i]["path"], - data_set_name=test_file_i, + test_file=test_file_config["path"], + data_set_name=test_file_identifier, tagger=tagger_name, ) @@ -847,13 +847,16 @@ if __name__ == "__main__": else: logger.info("Start evaluating UMAMI with test files...") - for test_file_i in training_config.test_files: + for ( + test_file_identifier, + test_file_config, + ) in training_config.test_files.items(): EvaluateModel( args=parser_args, train_config=training_config, preprocess_config=preprocessing_config, - test_file=training_config.test_files[test_file_i]["path"], - data_set_name=test_file_i, + test_file=test_file_config["path"], + data_set_name=test_file_identifier, ) else: diff --git a/umami/tests/integration/test_train.py b/umami/tests/integration/test_train.py index 81ee791b8f8c049ce7de22f8effc8eae4b3b1b5d..f81228b8a7be027d9dc0ca7054ed7939ec354775 100644 --- a/umami/tests/integration/test_train.py +++ b/umami/tests/integration/test_train.py @@ -144,7 +144,7 @@ def prepareConfig( "ttbar_r21": { "path": f"{test_file_ttbar}", "variable_cuts": [ - {"pt_btagJes": {"operator": "<=", "condition": 250000}} + {"pt_btagJes": {"operator": "<=", "condition": 250_000}} ], }, "zpext_r21": { @@ -157,7 +157,7 @@ def prepareConfig( config_file["NN_structure"]["batch_size"] = 50 config_file["NN_structure"]["epochs"] = 2 config_file["NN_structure"]["nJets_train"] = 100 - config_file["Eval_parameters_validation"]["n_jets"] = 4000 + config_file["Eval_parameters_validation"]["n_jets"] = 4_000 config_file["Eval_parameters_validation"]["eff_min"] = 0.60 # Add some validation files for testing config_file.update( @@ -167,14 +167,14 @@ def prepareConfig( "path": f"{test_dir}/ci_ttbar_testing.h5", "label": "$t\\bar{t}$ Release 21", "variable_cuts": [ - {"pt_btagJes": {"operator": "<=", "condition": 250000}} + {"pt_btagJes": {"operator": "<=", "condition": 250_000}} ], }, "zprime_r21_val": { "path": f"{test_dir}/ci_zpext_testing.h5", "label": "$Z'$ Release 21", "variable_cuts": [ - {"pt_btagJes": {"operator": ">", "condition": 250000}} + {"pt_btagJes": {"operator": ">", "condition": 250_000}} ], }, } diff --git a/umami/tests/unit/train_tools/test_NN_tools.py b/umami/tests/unit/train_tools/test_NN_tools.py index 875ba7d6bf08b52654d30440bfe65d478f097943..9edf5865c248d9a9ee73a98bc6e12bd01981ab69 100644 --- a/umami/tests/unit/train_tools/test_NN_tools.py +++ b/umami/tests/unit/train_tools/test_NN_tools.py @@ -273,8 +273,6 @@ class MyCallback_TestCase(unittest.TestCase): self.nFeatures = 15 self.nClasses = len(self.class_labels) self.target_beff = 0.77 - # TODO: we could add a properly initialized val_data_dict here, but seems like - # the test further down doesn't use it anyway self.val_data_dict = {} self.frac_dict = { "cjets": 0.018, @@ -399,15 +397,13 @@ class GetSamples_TestCase(unittest.TestCase): "path": f"{self.test_dir.name}/ci_ttbar_testing.h5", "label": "$t\\bar{t}$ Release 21", "variable_cuts": [ - {"pt_btagJes": {"operator": "<=", "condition": 250000}} + {"pt_btagJes": {"operator": "<=", "condition": 250_000}} ], }, "zprime_r21_val": { "path": f"{self.test_dir.name}/ci_zpext_testing.h5", "label": "$Z'$ Release 21", - "variable_cuts": [ - {"pt_btagJes": {"operator": ">", "condition": 250000}} - ], + "variable_cuts": None, }, } self.class_labels = ["bjets", "cjets", "ujets"] diff --git a/umami/tests/unit/train_tools/test_Plotting.py b/umami/tests/unit/train_tools/test_Plotting.py index 93ff2c105bea03e827df4e4d4dc09b0a66a43527..8957008cd7a6c71bc65c993bb860bd5f38b7d886 100644 --- a/umami/tests/unit/train_tools/test_Plotting.py +++ b/umami/tests/unit/train_tools/test_Plotting.py @@ -32,9 +32,6 @@ class GetRejection_TestCase(unittest.TestCase): self.tmp_dir = tempfile.TemporaryDirectory() self.tmp_plot_dir = f"{self.tmp_dir.name}/" self.Control_plots_dir = os.path.join(os.path.dirname(__file__), "plots/") - # for updating the control plots uncomment the following line and run one - # time with this configuration - # self.tmp_plot_dir = self.Control_plots_dir list_of_keys = [ "epoch", "loss", @@ -83,25 +80,22 @@ class GetRejection_TestCase(unittest.TestCase): "path": "dummy", "label": "$t\\bar{t}$ Release 21", "variable_cuts": [ - {"pt_btagJes": {"operator": "<=", "condition": 250000}} + {"pt_btagJes": {"operator": "<=", "condition": 250_000}} ], }, "zprime_r21_val": { "path": "dummy", "label": "$Z'$ Release 21", "variable_cuts": [ - {"pt_btagJes": {"operator": ">", "condition": 250000}} + {"pt_btagJes": {"operator": ">", "condition": 250_000}} ], }, } self.validation_unique_identifiers = self.val_files.keys() - # TODO: - # create a pseudo results dataframe where the results of each key specified - # above just increases linearly. In order to be able to see all of them when - # plotted together, the slope changes for each key (e.g. we want to plot - # different validation accuracies together --> if they have the same values we - # only see the most recently plotted line) + # TODO: Change the plotted data for each key? Atm we plot only straight lines + # on top of each other, which maybe makes the test less robust? + self.df_results = dict( zip( list_of_keys, diff --git a/umami/train_tools/NN_tools.py b/umami/train_tools/NN_tools.py index 9e59c6832672367e416fdc6f4a1225e70a567e75..3a7eb10ed1c7521c14b372a34ace46fffc59924d 100644 --- a/umami/train_tools/NN_tools.py +++ b/umami/train_tools/NN_tools.py @@ -1284,6 +1284,7 @@ def evaluate_model_umami( ) # Write metrics to results dict + # TODO Change this in python 3.9 result_dict.update( { f"val_loss_{val_file_identifier}": loss, diff --git a/umami/train_tools/Plotting.py b/umami/train_tools/Plotting.py index e6bd75fc5ae9fd05dba72045db06c47c007eca9d..01f5459cd248390c787c07e6d14e536f7cdb709b 100644 --- a/umami/train_tools/Plotting.py +++ b/umami/train_tools/Plotting.py @@ -202,6 +202,8 @@ def PlotDiscCutPerEpoch( Number of columns in the legend, by default 1 plot_datatype : str, optional Datatype of the plot, by default "pdf" + **kwargs + Arbitrary keyword arguments. """ # Apply ATLAS style @@ -293,6 +295,8 @@ def PlotDiscCutPerEpochUmami( Number of columns in the legend, by default 1 plot_datatype : str, optional Datatype of the plot, by default "pdf" + **kwargs + Arbitrary keyword arguments. """ # Apply ATLAS style @@ -420,6 +424,8 @@ def PlotRejPerEpochComparison( Datatype of the plot., by default "pdf" legFontSize : int, optional Fontsize of the legend., by default 10 + **kwargs + Arbitrary keyword arguments. """ if figsize is None: figsize = [8, 6] @@ -675,6 +681,8 @@ def PlotRejPerEpoch( Datatype of the plot., by default "pdf" legFontSize : int, optional Fontsize of the legend., by default 10 + **kwargs + Arbitrary keyword arguments. """ if figsize is None: figsize = [8, 6] @@ -844,6 +852,8 @@ def PlotLosses( Manually set ymin. Overwrites yAxisIncrease, by default None ymax : float, optional Manually set ymax. Overwrites yAxisIncrease, by default None + **kwargs + Arbitrary keyword arguments. """ # Apply ATLAS style @@ -944,6 +954,8 @@ def PlotAccuracies( Manually set ymin. Overwrites yAxisIncrease, by default None ymax : float, optional Manually set ymax. Overwrites yAxisIncrease, by default None + **kwargs + Arbitrary keyword arguments. """ # Apply ATLAS style @@ -1044,6 +1056,8 @@ def PlotLossesUmami( Manually set ymin. Overwrites yAxisIncrease, by default None ymax : float, optional Manually set ymax. Overwrites yAxisIncrease, by default None + **kwargs + Arbitrary keyword arguments. """ # Apply ATLAS style @@ -1159,6 +1173,8 @@ def PlotAccuraciesUmami( Manually set ymin. Overwrites yAxisIncrease, by default None ymax : float, optional Manually set ymax. Overwrites yAxisIncrease, by default None + **kwargs + Arbitrary keyword arguments. """ # Apply ATLAS style @@ -1311,7 +1327,6 @@ def RunPerformanceCheck( main_class=main_class, ) ) - # logger.warning(comp_tagger_rej_dict) else: # Define the dicts as None if compare tagger is False