- Fixing Callback error when LRR is not used [!567](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/567)
- Fixing stacking issue for the jet variables in the PDFSampling [!565](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/565)
- Fixing problem with 4 classes integration test [!564](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/564)
- Rework saliency plots to use puma [!556](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/556)
- Fixing generation of class ids for only one class [!563](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/563)
- Removing hardcoded tmp directories in the integration tests [!562](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/562)
- Fixing x range in metrics plots + correct tagger name in results files [!560](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/560)
- Fixing issue with the PDFSampling shuffling + Fixing small issue with the loaders [!558](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/558)
- Fixing ylabel issue in ROC plots [!555](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/555)
- Adding verbose option to executable scripts [!557](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/557)
- Moving Plotting Files in one folder [!554](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/554)
- Adding classes to global config (light-flavour jets split by quark flavour/gluons, leptonic b-hadron decays) to define extended tagger output [!553](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/553)
- Fixing issues with trained_taggers and taggers_from_file in plotting_epoch_performance.py [!549](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/549)
- Adding plotting API to Contour plots + Updating plotting_umami docs [!537](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/537)
- Adding unit test for prepare_model and minor bug fixes [!546](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/546)
- Adding unit tests for tf generators[!542](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/542)
- Fix epoch bug in continue_training[!543](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/543)
- Fix epoch bug in continue_training[!543](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/543)
- Updating tensorflow to version `2.9.0` and pytorch to `1.11.0-cuda11.3-cudnn8-runtime`[!547](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/547)
- Removing plotting API code and switch to puma [!540](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/540)[!548](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/548)
- Fix epoch bug in continue_training[!543](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/543)
The input variables for different files can also be plotted using the `plot_input_variables.py` script. Its also steered by a yaml file. An example for such a file can be found [here](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/examples/plotting_input_vars.yaml). The structure is close to the one from `plotting_umami` but still a little bit different.
To start the plotting of the input variables, you need to run the following command
```bash
plot_input_vars.py -c <path/to/config> --tracks
```
or
```bash
plot_input_vars.py -c <path/to/config> --jets
```
which will plot either all plots defined using jet- or track variables. You can also give the `-f` or `--format` option where you can decide on a format for the plots. The default is `pdf`.
### Yaml File
In the following, the possible configration parameters are listed with a brief description.
...
...
@@ -7,17 +20,17 @@ In the following, the possible configration parameters are listed with a brief d
#### Variable dict and number of jets
Here you can define the number of jets that are used and also the variable dict, where all the variables that are available are saved.
??? example "Click to see corresponding code highlighted in the [example config file](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/examples/plotting_input_vars.yaml)"
```yaml linenums="1", hl_lines="9-14"
§§§examples/plotting_input_vars.yaml§§§
??? example "Click to see corresponding code in the [example config file](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/examples/plotting_input_vars.yaml)"
```yaml
§§§examples/plotting_input_vars.yaml:9:14§§§
```
#### Number of Tracks per Jet
The number of tracks per jet can be plotted for all different files. This can be given like this:
??? example "Click to see corresponding code highlighted in the [example config file](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/examples/plotting_input_vars.yaml)"
```yaml linenums="1", hl_lines="117-133"
§§§examples/plotting_input_vars.yaml§§§
??? example "Click to see corresponding code in the [example config file](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/examples/plotting_input_vars.yaml)"
```yaml
§§§examples/plotting_input_vars.yaml:91:108§§§
```
| Options | Data Type | Necessary/Optional | Explanation |
...
...
@@ -36,9 +49,9 @@ The number of tracks per jet can be plotted for all different files. This can be
#### Input Variables Tracks
To plot the track input variables, the following options are used.
??? example "Click to see corresponding code highlighted in the [example config file](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/examples/plotting_input_vars.yaml)"
```yaml linenums="1", hl_lines="135-168"
§§§examples/plotting_input_vars.yaml§§§
??? example "Click to see corresponding code in the [example config file](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/examples/plotting_input_vars.yaml)"
```yaml
§§§examples/plotting_input_vars.yaml:110:144§§§
```
...
...
@@ -58,9 +71,9 @@ To plot the track input variables, the following options are used.
#### Input Variables Jets
To plot the jet input variables, the following options are used.
??? example "Click to see corresponding code highlighted in the [example config file](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/examples/plotting_input_vars.yaml)"
```yaml linenums="1", hl_lines="16-115"
§§§examples/plotting_input_vars.yaml§§§
??? example "Click to see corresponding code in the [example config file](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/examples/plotting_input_vars.yaml)"
```yaml
§§§examples/plotting_input_vars.yaml:16:89§§§
```
| Options | Data Type | Necessary/Optional | Explanation |
...
...
@@ -81,9 +94,9 @@ The `plot_settings` section is similar for all three cases described above.
In order to define some settings you want to apply to all plots, use yaml anchors
as shown here:
??? example "Click to see corresponding code highlighted in the [example config file](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/examples/plotting_input_vars.yaml)"
```yaml linenums="1", hl_lines="1-7"
§§§examples/plotting_input_vars.yaml§§§
??? example "Click to see corresponding code in the [example config file](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/examples/plotting_input_vars.yaml)"
```yaml
§§§examples/plotting_input_vars.yaml:1:7§§§
```
Most of the plot settings are valid for all types of input variable plots
| `xmin` | `float` | Optional | Set the minimum b efficiency in the plot (which is the xmin limit). |
| `ymax` | `float` | Optional | The maximum y axis. |
| `working_points` | `list` | Optional | The specified WPs are calculated and at the calculated b-tagging discriminant there will be a vertical line with a small label on top which prints the WP. |
...
...
@@ -124,10 +124,26 @@ Plot the b efficiency/c-rejection/light-rejection against the pT. For example:
| `flavour` | `str` | Necessary | Flavour class rejection which is to be plotted. |
| `class_labels` | List of class labels that were used in the preprocessing/training. They must be the same in all three files! Order is important! |
| `main_class` | `str` | Class which is to be tagged. |
| `WP` | `float` | Necessary | Float of the working point that will be used. |
| `WP_line` | `float` | Optional | Print a horizontal line at this value efficiency. |
| `working_point` | `float` | Necessary | Float of the working point that will be used. |
| `working_point_line` | `float` | Optional | Print a horizontal line at this value efficiency. |
| `fixed_eff_bin` | `bool` | Optional | Calculate the WP cut on the discriminant per bin. |
#### Saliency Plots
To evaluate the impact of the track variables to the final b-tagging discriminant can't be found using SHAPley. To make the impact visible (for each track of the jet), so-called Saliency maps are used. These maps are calculated when evaluating the model you have trained (if it is activated). A lot of different options can be set. An example is given here:
| `type` | `str` | Necessary | This gives the type of plot function used. Must be `"saliency"` here. |
| `data_set_name` | `str` | Necessary | Name of the dataset that is used. This is the name of the test_file which you want to use. |
| `target_eff` | `float` | Necessary | Efficiency of the target flavour you want to use (Which WP you want to use). The value is given between 0 and 1. |
| `jet_flavour` | `str` | Necessary | Name of flavour you want to plot. |
| `PassBool` | `str` | Necessary | Decide if the jets need to pass the working point discriminant cut or not. `False` would give you, for example, truth b-jets which does not pass the working point discriminant cut and are therefore not tagged a b-jets. |
| `nFixedTrks` | `int` | Necessary | The saliency maps can only be calculated for jets with a fixed number of tracks. This number of tracks can be set with this parameter. For example, if this value is `8`, than only jets which have exactly 8 tracks are used for the saliency maps. This value needs to be set in the train config when you run the evaluation! If you run the evaluation with, for example `5`, you can't plot the saliency map for `8`. |
#### Fraction Contour Plot
Plot two rejections against each other for a given working point with different fraction values.
@@ -16,7 +16,7 @@ After the previous step the ntuples need to be further processed. We can use dif
This processing can be done using the preprocessing capabilities of Umami via the [`preprocessing.py`](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/umami/preprocessing.py) script.
Please refer to the [documentation on preprocessing](preprocessing.md) for additional information.
Please refer to the [documentation on preprocessing](https://umami-docs.web.cern.ch/preprocessing/preprocessing/) for additional information.
For the GNN, we use the `PFlow-Preprocessing-GNN.yaml` config file, found [here](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami-config-tags/-/blob/master/offline/PFlow-Preprocessing-GNN.yaml).
@@ -112,7 +112,7 @@ Here are all important settings defined for the evaluation process (evaluating v
| `frac_step` | All | `float` | Optional | Step size of the fraction value scan. Please keep in mind that the fractions given to the background classes need to add up to one! All combinations that do not add up to one are ignored. If you choose a combination `frac_min`, `frac_max` or `frac_step` where the fractions of the brackground classes do not add up to one, you will get an error while running `evaluate_model.py` |
| `frac_min` | All | `float` | Optional | Minimal fraction value which is set for a background class in the fraction scan. |
| `frac_max` | All | `float` | Optional | Maximal fraction value which is set for a background class in the fraction scan. |
| `Calculate_Saliency` | DIPS | `bool` | Optional | Decide, if the saliency maps are calculated or not. This takes a lot of time and resources! |
| `calculate_saliency` | DIPS | `bool` | Optional | Decide, if the saliency maps are calculated or not. This takes a lot of time and resources! |
| `add_variables_eval` | DL1r, DL1d | `list` | Optional | A list to add available variables to the evaluation files. |
| `shapley` | DL1r, DL1d | `dict` | Optional | `dict` with the options for the feature importance explanation with SHAPley |
| `feature_sets` | DL1r, DL1d | `int` | Optional | Over how many full sets of features it should calculate over. Corresponds to the dots in the beeswarm plot. 200 takes like 10-15 min for DL1r on a 32 core-cpu. |
