Commit 5622fc0a authored by Alexander Froch's avatar Alexander Froch Committed by Manuel Guth
Browse files

Merging DL1r and DIPS doc doublings

parent 36994392
This diff is collapsed.
# Instructions to train DIPS with the umami framework
The following instructions are meant to give a guidline how to reproduce the DIPS results presented in the [DIPS Note](https://cds.cern.ch/record/2718948). It is focused on the PFlow training.
## Sample Preparation
The first step is to obtain the samples for the training. All the samples are listed in [MC-Samples.md](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/docs/MC-Samples.md). For the PFlow training only the ttbar and extended Z' samples from 2017 data taking period (MC16d) were used.
The training ntuples are produced using the [training-dataset-dumper](https://gitlab.cern.ch/atlas-flavor-tagging-tools/training-dataset-dumper) which dumps the jets from the FTAG1 derivations directly into hdf5 files. The processed ntuples are also listed in the table in [MC-Samples.md](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/docs/MC-Samples.md) which can be used for training.
### Ntuple preparation
After the previous step the ntuples need to be further processed. We can use different resampling approaches to achieve the same pt and eta distribution for all of the used flavour categories.
This processing can be done using the preprocessing capabilities of Umami via the [`preprocessing.py`](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/umami/preprocessing.py) script.
Please refer to the [documentation on preprocessing](preprocessing.md) for additional information.
Note, that for running Dips tracks have to be stored in the output hybrid sample. Therefore, the `save_tracks` argument in the preprocessing config need to be set while the preprocessing the samples.
## Config File
After all the files are ready we can start with the training. The config file for the Dips training is [Dips-PFlow-Training-config.yaml](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/examples/Dips-PFlow-Training-config.yaml). This will look for example like this:
```yaml
# Set modelname and path to Pflow preprocessing config file
model_name: <MODELNAME>
preprocess_config: <path>/<to>/<preprocessing>/<config>/PFlow-Preprocessing.yaml
# Add here a pretrained model to start with.
# Leave empty for a fresh start
model_file:
# Add training file
train_file: <path>/<to>/<train>/<samples>/train_file.h5
# Defining templates for the variable cuts
.variable_cuts_ttbar: &variable_cuts_ttbar
variable_cuts:
- pt_btagJes:
operator: "<="
condition: 2.5e5
.variable_cuts_zpext: &variable_cuts_zpext
variable_cuts:
- pt_btagJes:
operator: ">"
condition: 2.5e5
# Add validation files
validation_files:
ttbar_r21_val:
path: <path_palce_holder>/MC16d_hybrid_odd_100_PFlow-no_pTcuts-file_0.h5
label: "$t\\bar{t}$ Release 21"
<<: *variable_cuts_ttbar
zprime_r21_val:
path: <path_palce_holder>/MC16d_hybrid-ext_odd_0_PFlow-no_pTcuts-file_0.h5
label: "$Z'$ Release 21"
<<: *variable_cuts_zpext
test_files:
ttbar_r21:
path: <path>/<to>/<preprocessed>/<samples>/ttbar_r21_test_file.h5
<<: *variable_cuts_ttbar
ttbar_r22:
path: <path>/<to>/<preprocessed>/<samples>/ttbar_r22_test_file.h5
<<: *variable_cuts_ttbar
zpext_r21:
path: <path>/<to>/<preprocessed>/<samples>/zpext_r21_test_file.h5
<<: *variable_cuts_zpext
zpext_r22:
path: <path>/<to>/<preprocessed>/<samples>/zpext_r22_test_file.h5
<<: *variable_cuts_zpext
# Path to Variable dict used in preprocessing
var_dict: <path>/<to>/<variables>/Dips_Variables.yaml
exclude: null
# Values for the neural network
NN_structure:
# Decide, which tagger is used
tagger: "dips"
# NN Training parameters
lr: 0.001
batch_size: 15000
epochs: 200
# Number of jets used for training
# To use all: Fill nothing
nJets_train:
# Dropout rate. If = 0, dropout is disabled
dropout: 0
# Define which classes are used for training
# These are defined in the global_config
class_labels: ["ujets", "cjets", "bjets"]
# Main class which is to be tagged
main_class: "bjets"
# Decide if Batch Normalisation is used
Batch_Normalisation: True
# Structure of the dense layers for each track
ppm_sizes: [100, 100, 128]
# Structure of the dense layers after summing up the track outputs
dense_sizes: [100, 100, 100, 30]
# Options for the Learning Rate reducer
LRR: True
# Option if you want to use sample weights for training
use_sample_weights: False
# Plotting settings for training metrics plots
Validation_metrics_settings:
# Define which taggers should also be plotted
taggers_from_file: ["rnnip", "DL1r"]
# Label for the freshly trained tagger
tagger_label: "DIPS"
# Define which freshly trained taggers should be plotted
trained_taggers:
dipsReference:
path: "dips_Loose/validation_WP0p77_300000jets_Dict.json"
label: "DIPS Reference"
# Enable/Disable atlas tag
UseAtlasTag: True
# fc_value and WP_b are autmoatically added to the plot label
AtlasTag: "Internal Simulation"
SecondTag: "\n$\\sqrt{s}=13$ TeV, PFlow jets"
# Set the datatype of the plots
plot_datatype: "pdf"
# Eval parameters for validation evaluation while training
Eval_parameters_validation:
# Number of jets used for validation
n_jets: 3e5
# Define taggers that are used for comparison in evaluate_model
# This can be a list or a string for only one tagger
tagger: ["rnnip", "DL1r"]
# Define fc values for the taggers
frac_values_comp:
{
"rnnip": {"cjets": 0.08, "ujets": 0.92},
"DL1r": {"cjets": 0.018, "ujets": 0.982},
}
# Charm fraction value used for evaluation of the trained model
frac_values: {"cjets": 0.018, "ujets": 0.982}
# Working point used in the evaluation
WP: 0.77
# Decide, if the Saliency maps are calculated or not.
Calculate_Saliency: True
```
It contains the information about the neural network architecture and the training as well as about the files for training, validation and testing. Also evaluation parameters are given for the training evaluation which is performed by the [plotting_epoch_performance.py](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/umami/plotting_epoch_performance.py) script.
The different options are briefly explained here:
| Options | Data Type | Necessary/Optional | Explanation |
|---------|-----------|--------------------|-------------|
| `model_name` | String | Necessary | Name of the model which is to be trained. Also the foldername where everything of the model will be saved. |
| `preprocess_config` | String | Necessary | Path to the `preprocess_config` which was used to produce the training samples. |
| `model_file` | String | Optional | If you already have a model and want to continue the training of this model, you can give the path to this model here. This model will be loaded and used instead of init a new one. |
| `train_file` | String | Necessary | Path to the training sample. This is given by the `preprocessing` step of Umami |
| `validation_files` | Dict | Optional | Here you can define different validation samples that are used in the training and the `plotting_epoch_performance.py` script. Those validation samples need to be defined in a dict structure shown in the example. The name of the dict entry is relevant and is the unique identifier for this sample (DO NOT USE IT MULTIPLE TIMES). `path` gives the path to the file. |
| `test_files` | Dict | Optional | Here you can define different test samples that are used in the [`evaluate_model.py`](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/umami/evaluate_model.py). Those test samples need to be defined in a dict structure shown in the example. The name of the dict entry is relevant and is the unique identifier in the results file which is produced by the [`evaluate_model.py`](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/umami/evaluate_model.py). `Path` gives the path to the file. For test samples, all samples from the training-dataset-dumper can be used without preprocessing although the preprocessing of Umami produces test samples to ensure orthogonality of the jets with respect to the train sample. |
| `path` | String | Necessary | Path to the validation/test file which is to be used. Using wildcards is possible. |
| `variable_cuts` | Dict | Optional | Dict of cuts which are applied when loading the different test files. Only jet variables can be cut on. These are in this example defined as templates for the different samples types. |
| `var_dict` | String | Necessary | Path to the variable dict used in the `preprocess_config` to produce the train sample. |
| `exclude` | List | Necessary | List of variables that are excluded from training. Only compatible with DL1r training. To include all, just give an empty list. |
|`tracks_name`| String| Necessary* | Name of the tracks data-set to use for training and evaluation, default is "tracks". <br />* ***This option is necessary when using tracks, but, when working with old preprpocessed files (before January 2022) this option has to be removed form the config file to ensure compatibility*** |
| `NN_structure` | None | Necessary | A dict where all important information for the training are defined. |
| `tagger` | String | Necessary | Name of the tagger that is used/to be trained. |
| `load_optimiser` | Bool | Optional | When loading a model (via `model_file`), you can load the optimiser state for continuing a training (`True`) or initialize a new optimiser to use the model as a start point for a fresh training (`False`). Default is `True`.
| `lr` | Float | Necessary | Learning rate which is used for training. |
| `batch_size` | Int | Necessary | Batch size which is used for training. |
| `epochs` | Int | Necessary | Number of epochs of the training. |
| `nJets_train` | Int | Necessary | Number of jets used for training. Leave empty to use all. |
| `dropout` | Float | Necessary | Dropout factor used in the _ϕ_ network. If 0, dropout is not used. |
| `class_labels` | List | Necessary | List of flavours used in training. NEEDS TO BE THE SAME AS IN THE `preprocess_config`. Even the ordering needs to be the same! |
| `main_class` | String | Necessary | Main class which is to be tagged. Needs to be in `class_labels`. |
| `Batch_Normalisation` | Bool | Necessary | Decide, if batch normalisation is used in the _ϕ_ network. |
| `ppm_sizes` | List | Necessary | List of nodes per layer of the _ϕ_ network. Every entry is one layer. The numbers need to be ints! |
| `dense_sizes` | List | Necessary | List of nodes per layer of the _F_ network. Every entry is one layer. The numbers need to be ints! |
| `LRR` | Bool | Optional | Decide, if a Learning Rate Reducer (LRR) is used or not. If yes, the following options can be added. |
| `use_sample_weights` | Bool | Optional | Applies the weights, you calculated with the `--weighting` flag from the preprocessing to the training loss function. |
| `LRR_monitor` | String | Optional | Quantity to be monitored. Default: "loss" |
| `LRR_factor` | Float | Optional | Factor by which the learning rate will be reduced. `new_lr = lr * factor`. Default: 0.8 |
| `LRR_patience` | Int | Optional | Number of epochs with no improvement after which learning rate will be reduced. Default: 3 |
| `LRR_verbose` | Int | Optional | 0: Quiet, 1: Update messages. Default: 1 |
| `LRR_mode` | String | Optional | One of `{"auto", "min", "max"}`. In "min" mode, the learning rate will be reduced when the quantity monitored has stopped decreasing; in "max" mode it will be reduced when the quantity monitored has stopped increasing; in "auto" mode, the direction is automatically inferred from the name of the monitored quantity. Default: "auto" |
| `LRR_cooldown` | Int | Optional | Number of epochs to wait before resuming normal operation after lr has been reduced. Default: 5 |
| `LRR_min_lr` | Float | Optional | Lower bound on the learning rate. Default: 0.000001 |
| `Validation_metrics_settings` | None | Necessary | Plotting settings for the validation plots which are produced by the [plotting_epoch_performance.py](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/umami/plotting_epoch_performance.py) script. |
| `taggers_from_file` | List | Optional | List of taggers that are available in the .h5 samples. The here given taggers are plotted as reference lines in the rejection per epoch plots. |
| `tagger_label` | String | Optional | Name for the legend of the freshly trained tagger for the rejection per epoch plots. |
| `trained_taggers` | Dict | Optional | A dict with local trained taggers which shall be plotted in the rejection per epoch plots. You need to provide a dict with a `path` and a `label`. The path is the path to the validation metrics .json file, where the rejections per epoch are saved. The `label` is the label which will be shown in the legend in the rejection per epoch plots. The `dipsReference` in the example here is just an internal naming. It will not be shown anywhere. |
| `UseAtlasTag` | Bool | Optional | Decide, if the ATLAS tag is printed at the top left of the plot. |
| `AtlasTag` | String | Optional | Main ATLAS tag which is right to "ATLAS" |
| `SecondTag` | String | Optional | Second line below the ATLAS tag |
| `plot_datatype` | String | Necessary | Datatype of the plots that are produced using the [plotting_epoch_performance.py](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/umami/plotting_epoch_performance.py) script. |
| `Eval_parameters_validation` | None | Necessary | A dict where all important information for the training are defined. |
| `n_jets` | Int | Necessary | Number of jets used for evaluation. This should not be to high, due to the fact that Callback function also uses this amount of jets after each epoch for validation. |
| `tagger` | List | Necessary | List of taggers used for comparison. This needs to be a list of string or a single string. The name of the taggers must be same as in the evaluation file. For example, if the DL1d probabilities in the test samples are called `DL1dLoose20210607_pb`, the name you need to add to the list is `DL1dLoose20210607`. |
| `frac_values_comp` | Dict | Necessary | Dict with the fraction values for the comparison taggers. For all flavour (except the main flavour), you need to add values here which add up to one. |
| `frac_values` | Dict | Necessary | Dict with the fraction values for the freshly trained tagger. For all flavour (except the main flavour), you need to add values here which add up to one. |
| `WP` | Float | Necessary | Working point which is used in the validation and evaluation. |
| `Calculate_Saliency` | Bool | Optional | Decide, if the saliency maps are calculated or not. This takes a lot of time and resources! |
## Training
Before starting the training, you need to set some paths for the umami package to find all the tools. Change to the umami dir and run the `setup.py`.
```bash
python setup.py install
```
Note that with the `install` setup, changes that are performed to the scripts after setup are not included! For development and usage of changes without resetup everything, use
```bash
source run_setup.sh
```
This script sets the python path to a specific folder where the executables are directly the code you are working on.
After that, you can switch to the folder `umami/umami` and run the training, using the following command
```bash
train.py -c ${EXAMPLES}/Dips-PFlow-Training-config.yaml
```
The results after each epoch will be saved to the `umami/umami/MODELNAME/` folder. The modelname is the name defined in the [Dips-PFlow-Training-config.yaml](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/examples/Dips-PFlow-Training-config.yaml).
If you want instant performance checks of the model after each epoch during the training, you can use
```bash
plotting_epoch_performance.py -c ${EXAMPLES}/Dips-PFlow-Training-config.yaml
```
which will write out plots for the not- main flavour rejections, accuracy and loss per epoch to `umami/umami/MODELNAME/plots/`. In this form, the performance measurements, like the rejection performances, will be recalculated using the working point, the `frac_values` and the number of validation jets defined in the [Dips-PFlow-Training-config.yaml](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/examples/Dips-PFlow-Training-config.yaml). If you don't want to recalculate it, you can give the path to the existing dict with the option `--dict`. For example:
```bash
plotting_epoch_performance.py -c ${EXAMPLES}/Dips-PFlow-Training-config.yaml --dict dips_Loose_lr_0.001_bs_15000_epoch_200_nTrainJets_Full/validation_WP0p77_fc0p018_300000jets_Dict.json
```
## Evaluating the results
After the training is over, the different epochs can be evaluated with ROC plots, output scores, saliency maps and confusion matrices etc. using the build-in scripts. Before plotting these, the model needs to be evaluated using the [evaluate_model.py](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/umami/evaluate_model.py).
```bash
evaluate_model.py -c ${EXAMPLES}/Dips-PFlow-Training-config.yaml -e 5
```
The `-e` options (here `5`) allows to set the training epoch which should be evaluated.
It will produce .h5 and .pkl files with the evaluations which will be saved in the model folder in an extra folder called `results/`. After, the [plotting_umami.py](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/umami/plotting_umami.py) script can be used to plot the results. For an explanation, look in the [plotting_umami documentation](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/docs/plotting_umami.md)
# Instructions to train the Umami-tagger with the umami framework
## Preprocessing
Same as [Dl1r](DL1r-instructions.md) except for also writing the track information to the samples. Note that tau jets are not yet supported in Umami.
## Training
After all the files are ready we can start with the training. The config file for the Umami training is [umami-PFlow-Training-config.yaml](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/examples/umami-PFlow-Training-config.yaml).
It contains the information about the neural network architecture as well as about the files for training, validation and testing.
To run the training, use the following command
```bash
train_umami.py -c examples/umami-PFlow-Training-config.yaml
```
## Plotting
### Rejection Rates for Validation Sample
During the training the important metrics are saved in a json file and from this file the rejection rates per epoch can be plotted:
```bash
plotting_epoch_performance.py -c examples/umami-PFlow-Training-config.yaml -d umami_dev/validation_WP0p77_fc0p018_100000jets_Dict.json
```
If you want to re-evaluate the training with the valdiation samples but different charm fraction, working point or number of jets then you can do so by omitting the `-d` option like this:
```bash
plotting_epoch_performance.py -c examples/umami-PFlow-Training-config.yaml --beff 0.6 --cfrac 0.08 --nJets 300000
```
### Detailed Performance Evaluation on Test Sample
Finally we can evaluate our model.
You first need to choose which epoch you want to use for the evaluation (easiest by looking at the performance plots vs. epochs) and then run
```bash
evaluate_model.py -c examples/umami-PFlow-Training-config.yaml -e 230
```
Next you need to adapt the plotting config file [examples/plotting_umami_config_Umami.yaml](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/examples/plotting_umami_config_Umami.yaml) providing the epoch and model name in the `Eval_parameters`. The plots can then be created by running the following command
```bash
plotting_umami.py -c examples/plotting_umami_config_Umami.yaml
```
# Instructions to Train the different Taggers with the UMAMI Framework
The following instructions are meant to give a guidline how to train the different taggers currently living in the UMAMI framework. Currently supported taggers are
* DL1r / DL1d
* DIPS
* Umami
* DIPS Attention / CADS
## Sample Preparation
Before we can start training the different taggers, we need to produce our training,
validation and test datasets. This done using the preprocessing, which is explained
[here](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/docs/preprocessing.md).
For the different taggers, different information need to be inside the preprocessed
samples. While DL1r / DL1d and DIPS work on one specific information set (DL1r/DL1d: jet information, DIPS/DIPS Attention: track information), Umami and CADS need both information. Due to the fact that the jet information are always preprocessed (due to truth info needed), you need to check if you need track information. If this is the case, you need to set the `save_tracks` option to `True`. The rest of the preprocessing (with example files etc.) is explained [here](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/docs/preprocessing.md).
## Train Config
After all files are preprocessed, we can start with the training. The train config files for the different trainings can be found [here](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/tree/master/examples). While the basic options needed/provided inside this config files are the same for all taggers, some options are only available for some other. A list with all options/explanations, if the option is necessary or optional and for which tagger the option can be used, is provided here.
| Options | Tagger | Data Type | Necessary, Optional | Explanation |
|---------|--------|-----------|--------------------|-------------|
| `model_name` | All | `str` | Necessary | Name of the model you want to train. This will be the name of the folder, where all results etc. will be saved in. This folder will automatically be created if not existing. |
| `preprocess_config` | All | `str` | Necessary | Path to your preprocess config you used producing your train datasets. When you start the training and the folder for the model is created, this file is copied to the `metadata/` folder inside the model folder. Also, the path here in the train config will be changed to the new path of the preprocess config inside the `metadata/` folder. |
| `model_file` | All | `str` | Optional | If you already have a model and want to continue the training of this model, you can give the path to this model here. This model will be loaded and used instead of init a new one. |
| `train_file` | All | `str` | Necessary | Path to the training sample. This is given by the `preprocessing` step of Umami |
| `validation_files` | All | `dict` | Necessary | Here you can define different validation samples that are used in the training and the `plotting_epoch_performance.py` script. Those validation samples need to be defined in a dict structure shown in the example. The name of the dict entry is relevant and is the unique identifier for this sample (DO NOT USE IT MULTIPLE TIMES). `path` gives the path to the file. If you don't want to use validation files, leave this blank. |
| `test_files` | All | `dict` | Necessary | Here you can define different test samples that are used in the [`evaluate_model.py`](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/umami/evaluate_model.py). Those test samples need to be defined in a dict structure shown in the example. The name of the dict entry is relevant and is the unique identifier in the results file which is produced by the [`evaluate_model.py`](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/umami/evaluate_model.py). `Path` gives the path to the file. For test samples, all samples from the training-dataset-dumper can be used without preprocessing although the preprocessing of Umami produces test samples to ensure orthogonality of the jets with respect to the train sample. If you don't want to use test files, leave this blank. |
| `path` | All | `str` | Necessary | Path to the validation/test file which is to be used. Using wildcards is possible. |
| `variable_cuts` | All | `dict` | Optional | `dict` of cuts which are applied when loading the different test files. Only jet variables can be cut on. These are in this example defined as templates for the different samples types. |
| `var_dict` | All | `str` | Necessary | Path to the variable dict used in the `preprocess_config` to produce the train sample. |
| `exclude` | DL1r, DL1d | `list` | Necessary | List of variables that are excluded from training. Only compatible with DL1r training. To include all, just give an empty list. |
|`tracks_name`| DIPS, DIPS Attention, Umami, CADS | `str` | Necessary* | Name of the tracks data-set to use for training and evaluation, default is "tracks". <br />* ***This option is necessary when using tracks, but, when working with old preprpocessed files (before January 2022, Tag 05 or older) this option has to be removed form the config file to ensure compatibility*** |
| `NN_structure` | All | `dict` | Necessary | A dict where all important information for the training are defined. |
| `tagger` | All | `str` | Necessary | Name of the tagger that is used/to be trained. |
| `load_optimiser` | All | `bool` | Optional | When loading a model (via `model_file`), you can load the optimiser state for continuing a training (`True`) or initialize a new optimiser to use the model as a start point for a fresh training (`False`).
| `lr` | All | `float` | Necessary | Learning rate which is used for training. |
| `batch_size` | All | `int` | Necessary | Batch size which is used for training. |
| `epochs` | All | `int` | Necessary | Number of epochs of the training. |
| `nJets_train` | All | `int` | Necessary | Number of jets used for training. Leave empty to use all. |
| `dropout` | All | `float` | Necessary | Dropout factor used in the _ϕ_ network. If 0, dropout is not used. |
| `class_labels` | All | `list` | Necessary | List of flavours used in training. NEEDS TO BE THE SAME AS IN THE `preprocess_config`. Even the ordering needs to be the same! |
| `main_class` | All | `str` | Necessary | Main class which is to be tagged. Needs to be in `class_labels`. |
| `Batch_Normalisation` | All | `bool` | Necessary | Decide, if batch normalisation is used in the network. (Look in the model files where this is used for the specific models) |
| `ppm_sizes` | DIPS, DIPS Attention, Umami, CADS | `list` | Necessary | List of nodes per layer of the _ϕ_ network. Every entry is one layer. The numbers need to be ints! |
| `dense_sizes` | All | `list` | Necessary | List of nodes per layer of the _F_ network (DIPS/DIPS Attention/Umami/CADS). Every entry is one layer. The numbers need to be ints! For DL1r/DL1d, this is the number of nodes per layer. |
| `LRR` | All | `bool` | Optional | Decide, if a Learning Rate Reducer (LRR) is used or not. If yes, the following options can be added. |
| `use_sample_weights` | All | `bool` | Optional | Applies the weights, you calculated with the `--weighting` flag from the preprocessing to the training loss function. |
| `LRR_monitor` | All | `str` | Optional | Quantity to be monitored. Default: "loss" |
| `LRR_factor` | All | `float` | Optional | Factor by which the learning rate will be reduced. `new_lr = lr * factor`. Default: 0.8 |
| `LRR_patience` | All | `int` | Optional | Number of epochs with no improvement after which learning rate will be reduced. Default: 3 |
| `LRR_verbose` | All | `int` | Optional | 0: Quiet, 1: Update messages. Default: 1 |
| `LRR_mode` | All | `str` | Optional | One of `{"auto", "min", "max"}`. In "min" mode, the learning rate will be reduced when the quantity monitored has stopped decreasing; in "max" mode it will be reduced when the quantity monitored has stopped increasing; in "auto" mode, the direction is automatically inferred from the name of the monitored quantity. Default: "auto" |
| `LRR_cooldown` | All | `int` | Optional | Number of epochs to wait before resuming normal operation after lr has been reduced. Default: 5 |
| `LRR_min_lr` | All | `float` | Optional | Lower bound on the learning rate. Default: 0.000001 |
| `Validation_metrics_settings` | All | `dict` | Necessary | Plotting settings for the validation plots which are produced by the [plotting_epoch_performance.py](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/umami/plotting_epoch_performance.py) script. |
| `taggers_from_file` | All | `list` | Optional | List of taggers that are available in the .h5 samples. The here given taggers are plotted as reference lines in the rejection per epoch plots. |
| `tagger_label` | All | `str` | Optional | Name for the legend of the freshly trained tagger for the rejection per epoch plots. |
| `trained_taggers` | All | `dict` | Optional | A dict with local trained taggers which shall be plotted in the rejection per epoch plots. You need to provide a dict with a `path` and a `label`. The path is the path to the validation metrics .json file, where the rejections per epoch are saved. The `label` is the label which will be shown in the legend in the rejection per epoch plots. The `dipsReference` in the example here is just an internal naming. It will not be shown anywhere. |
| `UseAtlasTag` | All | `bool` | Optional | Decide, if the ATLAS tag is printed at the top left of the plot. |
| `AtlasTag` | All | `str` | Optional | Main ATLAS tag which is right to "ATLAS" |
| `SecondTag` | All | `str` | Optional | Second line below the ATLAS tag |
| `plot_datatype` | All | `str` | Necessary | Datatype of the plots that are produced using the [plotting_epoch_performance.py](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/umami/plotting_epoch_performance.py) script. |
| `Eval_parameters_validation` | All | `dict` | Necessary | A dict where all important information for the training are defined. |
| `n_jets` | All | `int` | Necessary | Number of jets used for evaluation. This should not be to high, due to the fact that Callback function also uses this amount of jets after each epoch for validation. |
| `tagger` | All | `list` | Necessary | List of taggers used for comparison. This needs to be a list of `str` or a single `str`. The name of the taggers must be same as in the evaluation file. For example, if the DL1d probabilities in the test samples are called `DL1dLoose20210607_pb`, the name you need to add to the list is `DL1dLoose20210607`. |
| `frac_values_comp` | All | `dict` | Necessary | `dict` with the fraction values for the comparison taggers. For all flavour (except the main flavour), you need to add values here which add up to one. |
| `frac_values` | All | `dict` | Necessary | `dict` with the fraction values for the freshly trained tagger. For all flavour (except the main flavour), you need to add values here which add up to one. |
| `WP` | All | `float` | Necessary | Working point which is used in the validation and evaluation. |
| `eff_min` | All | `float` | Optional | Minimal main class efficiency considered for ROC.
| `eff_max` | All | `float` | Optional | Maximal main class efficiency considered for ROC.
| `Calculate_Saliency` | DIPS | `bool` | Optional | Decide, if the saliency maps are calculated or not. This takes a lot of time and resources! |
| `add_variables_eval` | DL1r, DL1d | `list` | Optional | A list to add available variables to the evaluation files. |
| `shapley` | DL1r, DL1d | `dict` | Optional | `dict` with the options for the feature importance explanation with SHAPley |
| `feature_sets` | DL1r, DL1d | `int` | Optional | Over how many full sets of features it should calculate over. Corresponds to the dots in the beeswarm plot. 200 takes like 10-15 min for DL1r on a 32 core-cpu. |
| `model_output` | DL1r, DL1d | `int` | Optional | Defines which of the model outputs (flavour) you want to explain. This is the index of the flavour in `class_labels`. |
| `bool_all_flavor_plot` | DL1r, DL1d | `bool` | Optional | You can also choose if you want to plot the magnitude of feature importance for all output nodes (flavors) in another plot. This will give you a bar plot of the mean SHAP value magnitudes. |
| `averaged_sets` | DL1r, DL1d | `int` | Optional | As this takes much longer you can average the feature_sets to a smaller set, 50 is a good choice for DL1r. |
| `plot_size` | DL1r, DL1d | `list` | Optional | Figure size of the SHAPley plot. This is a list with `[width, height]` |
## Training
Before starting the training, you need to set some paths for the umami package to find all the tools. How to set this all up is explained [here](https://umami-docs.web.cern.ch/installation/).
Note: When working with `Singularity`, the `python install` option is not writable and therefore will fail. In this case, switch to the umami folder and run the following command.
```bash
source run_setup.sh
```
This will create links to the executables. Note: If you know change something in the files, this will immediately become active (Like for `python setup.py develop`). A more detailed explanation can be found [here](https://umami-docs.web.cern.ch/installation/).
After that, you can switch to the folder `umami/umami` and run the training, using the following command
```bash
train.py -c <path>/<to>/<train>/<config>
```
The results after each epoch will be saved to the `umami/umami/MODELNAME/` folder. The modelname is the name defined in the train config.
## Plotting the Training Metrics
If you want to check how your model is performing while training, you can use the following command:
```bash
plotting_epoch_performance.py -c <path>/<to>/<train>/<config> --dict <path>/<to>/<validation>/<json>
```
This will write out plots for the non-main flavour rejections, accuracy and loss per epoch to `umami/umami/MODELNAME/plots/`. The validation json mentioned here will be produced by the `MyCallback` function which is running on each epoch end. As a result, a json file will be filled with different metrics. The validation json is updated after each epoch. The file will be stored in the `umami/umami/MODELNAME/` folder.
If you don't provide a json with the `--dict` argument, the performance measurements, like the rejection performances, will be recalculated using the working point, the `frac_values` and the number of validation jets defined in the train config. This can take a long time without a GPU, because each saved model is loaded and evaluated with the validation files. We strongly advice you to only do that if you changed the validation files!
## Evaluating the results
After the training is over, the different epochs can be evaluated with ROC plots, output scores, saliency maps and confusion matrices etc. using the build-in scripts. Before plotting these, a model needs to be evaluated using the [evaluate_model.py](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/umami/evaluate_model.py).
```bash
evaluate_model.py -c <path>/<to>/<train>/<config> -e 5
```
The `-e` options (here `5`) allows to set the training epoch which should be evaluated.
It will produce .h5 and .pkl files with the evaluations which will be saved in the model folder in an extra folder called `results/`. After, the [plotting_umami.py](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/umami/plotting_umami.py) script can be used to plot the results. For an explanation, look in the [plotting_umami documentation](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/docs/plotting_umami.md).
......@@ -16,9 +16,7 @@ nav:
- Development: development.md
- MC samples: mc-samples.md
- Preprocessing: preprocessing.md
- Running DL1r: DL1r-instructions.md
- Running Dips: Dips-instructions.md
- Running Umami: Umami-instructions.md
- Training: train-instructions.md
- Running GNN: GNN-instructions.md
- LWTNN Conversion: LWTNN-conversion.md
- Evaluate Taggers in Samples: WO_trained_model.md
......
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment