Commit 032a57c1 authored by Manuel Guth's avatar Manuel Guth
Browse files

Merge branch alfroch-fix-saliency with refs/heads/master into refs/merge-requests/556/train

parents f44e66e7 c9f139d7
Pipeline #4105544 passed with stages
in 11 minutes and 38 seconds
include umami/preprocessing_tools/configs/*.yaml
include umami/tools/PyATLASstyle/fonts/*.ttf
include umami/configs/global_config.yaml
\ No newline at end of file
......@@ -6,6 +6,7 @@
- Fixing stacking issue for the jet variables in the PDFSampling [!565](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/565)
- Fixing problem with 4 classes integration test [!564](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/564)
- Rework saliency plots to use puma [!556](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/556)
- Fixing generation of class ids for only one class [!563](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/563)
- Removing hardcoded tmp directories in the integration tests [!562](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/562)
- Fixing x range in metrics plots + correct tagger name in results files [!560](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/560)
......
......@@ -128,6 +128,22 @@ Plot the b efficiency/c-rejection/light-rejection against the pT. For example:
| `working_point_line` | `float` | Optional | Print a horizontal line at this value efficiency. |
| `fixed_eff_bin` | `bool` | Optional | Calculate the WP cut on the discriminant per bin. |
#### Saliency Plots
To evaluate the impact of the track variables to the final b-tagging discriminant can't be found using SHAPley. To make the impact visible (for each track of the jet), so-called Saliency maps are used. These maps are calculated when evaluating the model you have trained (if it is activated). A lot of different options can be set. An example is given here:
```yaml
§§§examples/plotting_umami_config_dips.yaml:149:160§§§
```
| Options | Data Type | Necessary/Optional | Explanation |
|---------|-----------|--------------------|-------------|
| `type` | `str` | Necessary | This gives the type of plot function used. Must be `"saliency"` here. |
| `data_set_name` | `str` | Necessary | Name of the dataset that is used. This is the name of the test_file which you want to use. |
| `target_eff` | `float` | Necessary | Efficiency of the target flavour you want to use (Which WP you want to use). The value is given between 0 and 1. |
| `jet_flavour` | `str` | Necessary | Name of flavour you want to plot. |
| `PassBool` | `str` | Necessary | Decide if the jets need to pass the working point discriminant cut or not. `False` would give you, for example, truth b-jets which does not pass the working point discriminant cut and are therefore not tagged a b-jets. |
| `nFixedTrks` | `int` | Necessary | The saliency maps can only be calculated for jets with a fixed number of tracks. This number of tracks can be set with this parameter. For example, if this value is `8`, than only jets which have exactly 8 tracks are used for the saliency maps. This value needs to be set in the train config when you run the evaluation! If you run the evaluation with, for example `5`, you can't plot the saliency map for `8`. |
#### Fraction Contour Plot
Plot two rejections against each other for a given working point with different fraction values.
......
......@@ -112,7 +112,7 @@ Here are all important settings defined for the evaluation process (evaluating v
| `frac_step` | All | `float` | Optional | Step size of the fraction value scan. Please keep in mind that the fractions given to the background classes need to add up to one! All combinations that do not add up to one are ignored. If you choose a combination `frac_min`, `frac_max` or `frac_step` where the fractions of the brackground classes do not add up to one, you will get an error while running `evaluate_model.py` |
| `frac_min` | All | `float` | Optional | Minimal fraction value which is set for a background class in the fraction scan. |
| `frac_max` | All | `float` | Optional | Maximal fraction value which is set for a background class in the fraction scan. |
| `Calculate_Saliency` | DIPS | `bool` | Optional | Decide, if the saliency maps are calculated or not. This takes a lot of time and resources! |
| `calculate_saliency` | DIPS | `bool` | Optional | Decide, if the saliency maps are calculated or not. This takes a lot of time and resources! |
| `add_variables_eval` | DL1r, DL1d | `list` | Optional | A list to add available variables to the evaluation files. |
| `shapley` | DL1r, DL1d | `dict` | Optional | `dict` with the options for the feature importance explanation with SHAPley |
| `feature_sets` | DL1r, DL1d | `int` | Optional | Over how many full sets of features it should calculate over. Corresponds to the dots in the beeswarm plot. 200 takes like 10-15 min for DL1r on a 32 core-cpu. |
......
......@@ -163,4 +163,4 @@ Eval_parameters_validation:
WP: 0.77
# Decide, if the Saliency maps are calculated or not.
Calculate_Saliency: False
calculate_saliency: False
......@@ -145,4 +145,4 @@ Eval_parameters_validation:
WP: 0.77
# Decide, if the Saliency maps are calculated or not.
Calculate_Saliency: True
calculate_saliency: True
......@@ -149,12 +149,12 @@ confusion_matrix_Dips_ttbar:
Dips_saliency_b_WP77_passed_ttbar:
type: "saliency"
data_set_name: "ttbar_r21"
target_eff: 0.77
jet_flavour: "bjets"
PassBool: True
nFixedTrks: 8
plot_settings:
title: "Saliency map for $b$ jets from \n $t\\bar{t}$ who passed WP = 77% \n with exactly 8 tracks"
target_beff: 0.77
jet_flavour: "cjets"
PassBool: True
FlipAxis: True
use_atlas_tag: True # Enable/Disable atlas_first_tag
atlas_first_tag: "Simulation Internal"
atlas_second_tag: "$\\sqrt{s}=13$ TeV, PFlow jets"
......
......@@ -297,7 +297,7 @@ def evaluate_model(
)
# Get the discriminant values and probabilities of each tagger for each jet
df_discs_dict = uet.GetScoresProbsDict(
df_discs_dict = uet.get_scores_probs_dict(
jets=jets,
y_true=truth_internal_labels,
tagger_preds=tagger_preds,
......@@ -329,7 +329,7 @@ def evaluate_model(
)
# Get the rejections, discs and effs of the taggers
tagger_rej_dicts = uet.GetRejectionPerEfficiencyDict(
tagger_rej_dicts = uet.get_rej_per_eff_dict(
jets=jets,
y_true=truth_internal_labels,
tagger_preds=tagger_preds,
......@@ -367,7 +367,7 @@ def evaluate_model(
h5_file.attrs["N_test"] = len(jets)
# Get the rejections, discs and f_* values for the taggers
tagger_fraction_rej_dict = uet.GetRejectionPerFractionDict(
tagger_fraction_rej_dict = uet.get_rej_per_frac_dict(
jets=jets,
y_true=truth_internal_labels,
tagger_preds=tagger_preds,
......@@ -583,7 +583,7 @@ def evaluate_model_dips(
)
# Get the discriminant values and probabilities of each tagger for each jet
df_discs_dict = uet.GetScoresProbsDict(
df_discs_dict = uet.get_scores_probs_dict(
jets=jets,
y_true=truth_internal_labels,
tagger_preds=[pred_dips],
......@@ -610,7 +610,7 @@ def evaluate_model_dips(
)
# Get the rejections, discs and effs of the taggers
tagger_rej_dicts = uet.GetRejectionPerEfficiencyDict(
tagger_rej_dicts = uet.get_rej_per_eff_dict(
jets=jets,
y_true=truth_internal_labels,
tagger_preds=[pred_dips],
......@@ -644,7 +644,7 @@ def evaluate_model_dips(
h5_file.attrs["N_test"] = len(jets)
# Get the rejections, discs and f_* values for the taggers
tagger_fraction_rej_dict = uet.GetRejectionPerFractionDict(
tagger_fraction_rej_dict = uet.get_rej_per_frac_dict(
jets=jets,
y_true=truth_internal_labels,
tagger_preds=[pred_dips],
......@@ -677,11 +677,11 @@ def evaluate_model_dips(
h5_file.attrs["N_test"] = len(jets)
if (
"Calculate_Saliency" in eval_params
and eval_params["Calculate_Saliency"] is True
"calculate_saliency" in eval_params
and eval_params["calculate_saliency"] is True
):
# Get the saliency map dict
saliency_map_dict = uet.GetSaliencyMapDict(
saliency_map_dict = uet.get_saliency_map_dict(
model=model,
model_pred=pred_dips,
X_test=x_comb,
......@@ -689,6 +689,14 @@ def evaluate_model_dips(
class_labels=class_labels,
main_class=main_class,
frac_dict=eval_params["frac_values"],
var_dict_path=train_config.var_dict,
tracks_name=tracks_name,
nTracks=eval_params["saliency_ntrks"]
if "saliency_ntrks" in eval_params
else None,
effs=eval_params["saliency_effs"]
if "saliency_effs" in eval_params
else None,
)
# Create results dir and pickle file
......@@ -867,7 +875,7 @@ def evaluate_model_dl1(
# Get the discriminant values and probabilities of each tagger
# for each jet
df_discs_dict = uet.GetScoresProbsDict(
df_discs_dict = uet.get_scores_probs_dict(
jets=jets,
y_true=truth_internal_labels,
tagger_preds=[pred_dl1],
......@@ -900,7 +908,7 @@ def evaluate_model_dl1(
)
# Get the rejections, discs and effs of the taggers
tagger_rej_dicts = uet.GetRejectionPerEfficiencyDict(
tagger_rej_dicts = uet.get_rej_per_eff_dict(
jets=jets,
y_true=truth_internal_labels,
tagger_preds=[pred_dl1],
......@@ -934,7 +942,7 @@ def evaluate_model_dl1(
h5_file.attrs["N_test"] = len(jets)
# Get the rejections, discs and f_* values for the taggers
tagger_fraction_rej_dict = uet.GetRejectionPerFractionDict(
tagger_fraction_rej_dict = uet.get_rej_per_frac_dict(
jets=jets,
y_true=truth_internal_labels,
tagger_preds=[pred_dl1],
......
# flake8: noqa
# pylint: skip-file
from umami.evaluation_tools.eval_tools import (
GetRejectionPerEfficiencyDict,
GetRejectionPerFractionDict,
GetSaliencyMapDict,
GetScoresProbsDict,
RecomputeScore,
get_rej_per_eff_dict,
get_rej_per_frac_dict,
get_saliency_map_dict,
get_scores_probs_dict,
recompute_score,
)
......@@ -13,6 +13,7 @@ from tensorflow.keras.layers import Lambda # pylint: disable=import-error
from tensorflow.keras.models import Model # pylint: disable=import-error
import umami.metrics as umt
from umami.preprocessing_tools import GetVariableDict
from umami.tools import check_main_class_input
......@@ -86,7 +87,7 @@ def calculate_fraction_dict(
return dict_list
def GetRejectionPerFractionDict(
def get_rej_per_frac_dict(
jets,
y_true: np.ndarray,
tagger_preds: list,
......@@ -232,7 +233,7 @@ def GetRejectionPerFractionDict(
return tagger_rej_dict
def GetRejectionPerEfficiencyDict(
def get_rej_per_eff_dict(
jets,
y_true,
tagger_preds: list,
......@@ -377,7 +378,7 @@ def GetRejectionPerEfficiencyDict(
return {**tagger_disc_cut_dicts, **tagger_rej_dicts}
def GetScoresProbsDict(
def get_scores_probs_dict(
jets,
y_true,
tagger_preds: list,
......@@ -515,14 +516,18 @@ def GetScoresProbsDict(
return df_discs_dict
def GetSaliencyMapDict(
def get_saliency_map_dict(
model: object,
model_pred,
X_test,
Y_test,
model_pred: np.ndarray,
X_test: np.ndarray,
Y_test: np.ndarray,
class_labels: list,
main_class: str,
frac_dict: dict,
var_dict_path: str,
tracks_name: str,
nTracks: int = None,
effs: list = None,
nJets: int = int(10e4),
) -> dict:
"""
......@@ -544,17 +549,53 @@ def GetSaliencyMapDict(
The main discriminant class. For b-tagging obviously "bjets".
frac_dict : dict
Dict with the fraction values for the tagger.
nJets : int
var_dict_path : str
Path to the variable dict which was used for training the tagger
(to retrieve the inputs).
tracks_name : str
Name of the tracks which are used in the training.
nTracks : int
Number of tracks each jet needs to have. Saliency maps can
only be calculated for a fixed number of tracks per jet.
Only jets with this amount of tracks are used for calculation.
effs : list, optional
List with the efficiencies which are tested.
If None is given, the default WPs of 60, 70, 77 and 85
are tested. By default None.
nJets : int, optional
Number of jets to use to calculate the saliency maps.
By default 10e4
Returns
-------
Map_dict : dict
Dict with the saliency values
Raises
------
ValueError
If given efficiencies are neither a list nor a int.
"""
logger.info("Calculate gradients for inputs")
# Check if default nTracks must be used
if nTracks is None:
nTracks = 8
# Check effs for None
if effs is None:
effs = [60, 70, 77, 85]
elif isinstance(effs, int):
effs = [effs]
elif not isinstance(effs, list):
raise ValueError(
"Efficiencies for saliency calculation must be a list "
f"or an int! Given type: {type(effs)}"
)
# Cut off last layer of the model for saliency maps
cutted_model = model.layers[-1].output
......@@ -587,14 +628,23 @@ def GetSaliencyMapDict(
frac_dict=frac_dict,
)
# Load the variable dict
var_dict = GetVariableDict(var_dict_path)
# Extract track variables from the dict
trk_variables_dict = var_dict["track_train_variables"][tracks_name]
# Flatten the track variables in one list
trk_variables_list = [i for j in trk_variables_dict for i in trk_variables_dict[j]]
# Init small dict
map_dict = {}
map_dict = {"Variables_list": trk_variables_list}
# Get spartial class id
class_indices = list(range(len(class_labels)))
# Iterate over different beff, jet flavours and passed options
for target_beff in [60, 70, 77, 85]:
for target_beff in effs:
for (jet_flavour, class_index) in zip(class_labels, class_indices):
for PassBool in [True, False]:
......@@ -604,15 +654,15 @@ def GetSaliencyMapDict(
# Get the cutvalue for the specific WP
cutvalue = np.percentile(Disc_values_flavour, (100 - target_beff))
# Check for correct flavour and number of tracks
mask = Y_test[:, class_index].astype(bool)
mask = mask & (nTrks == nTracks)
# Set PassBool masking
if PassBool is True:
mask = Y_test[:, class_index].astype(bool)
mask = mask & (nTrks == 8)
if PassBool:
mask = mask & (Disc_values > cutvalue)
elif PassBool is False:
mask = Y_test[:, class_index].astype(bool)
mask = mask & (nTrks == 8)
else:
mask = mask & (Disc_values < cutvalue)
# Get gradient map
......@@ -631,7 +681,7 @@ def GetSaliencyMapDict(
return map_dict
def RecomputeScore(
def recompute_score(
df,
model_tagger: str,
main_class: str,
......
"""Collection of plotting function for ftag performance plots."""
# pylint: disable=consider-using-f-string, invalid-name
# TODO: switch to new plotting API with pep8 conform naming
from umami.configuration import global_config, logger # isort:skip
from collections import OrderedDict
import matplotlib.pyplot as plt
......@@ -15,13 +12,14 @@ from puma import (
FractionScanPlot,
Histogram,
HistogramPlot,
PlotBase,
Roc,
RocPlot,
VarVsEff,
VarVsEffPlot,
)
import umami.tools.PyATLASstyle.PyATLASstyle as pas
from umami.configuration import global_config, logger
from umami.plotting_tools.utils import translate_kwargs
......@@ -476,19 +474,12 @@ def plot_roc(
def plot_saliency(
maps_dict: dict,
plot_name: str,
title: str,
target_beff: float = 0.77,
jet_flavour: str = "bjets",
PassBool: bool = True,
nFixedTrks: int = 8,
fontsize: int = 14,
xlabel: str = "Tracks sorted by $s_{d0}$",
use_atlas_tag: bool = True,
atlas_first_tag: str = "Simulation Internal",
atlas_second_tag: str = r"$\sqrt{s}$ = 13 TeV, $t\bar{t}$ PFlow Jets",
yAxisAtlasTag: float = 0.925,
FlipAxis: bool = False,
dpi: int = 400,
target_eff: float,
jet_flavour: str,
PassBool: bool,
nFixedTrks: int,
cmap: str = "PiYG",
**kwargs,
):
"""Plot the saliency map given in maps_dict.
......@@ -498,9 +489,7 @@ def plot_saliency(
Dict with the saliency values inside
plot_name : str
Path, Name and format of the resulting plot file.
title : str
Title of the plot
target_beff : float, optional
target_eff : float, optional
Working point to use, by default 0.77
jet_flavour : str, optional
Class which is to be plotted, by default "bjets"
......@@ -509,151 +498,93 @@ def plot_saliency(
the working point cut, by default True
nFixedTrks : int, optional
Decide, how many tracks the jets need to have, by default 8
fontsize : int, optional
Fontsize to use in the title, legend, etc, by default 14
xlabel : str, optional
x-label, by default "Tracks sorted by {d0}$"
use_atlas_tag : bool, optional
Use the ATLAS Tag in the plots, by default True
atlas_first_tag : str, optional
First row of the ATLAS Tag, by default "Simulation Internal"
atlas_second_tag : str, optional
Second Row of the ATLAS Tag, by default
"$sqrt{s}$ = 13 TeV, $t bar{t}$ PFlow Jets"
yAxisAtlasTag : float, optional
y position of the ATLAS Tag, by default 0.925
FlipAxis : bool, optional
Decide, if the x- and y-axis are switched, by default False
dpi : int, optional
Sets a DPI value for the plot that is produced (mainly for png),
by default 400
"""
# Transform to percent
target_beff = 100 * target_beff
cmap : str, optional
Colour map to use for the saliency map. By defaulkt PiYG
**kwargs
Keyword arguments handed to the plotting API
# Little Workaround
atlas_first_tag = " " + atlas_first_tag
gradient_map = maps_dict[f"{int(target_beff)}_{jet_flavour}_{PassBool}"]
colorScale = np.max(np.abs(gradient_map))
cmaps = {
"ujets": "RdBu",
"cjets": "PuOr",
"bjets": "PiYG",
}
nFeatures = gradient_map.shape[0]
if FlipAxis is True:
fig = plt.figure(figsize=(0.7 * nFeatures, 0.7 * nFixedTrks))
gradient_map = np.swapaxes(gradient_map, 0, 1)
Raises
------
ValueError
If the number of variables in the given dict is not the same
as the number of variables used to calculate the saliency map.
"""
# Get list of variables used for saliency maps
variable_list = maps_dict["Variables_list"]
# Load gradient map from file
gradient_map = maps_dict[f"{int(target_eff * 100)}_{jet_flavour}_{PassBool}"][
:, :nFixedTrks
]
# Check kwargs for fontsize
kwargs["fontsize"] = (
kwargs["fontsize"]
if "fontsize" in kwargs and kwargs["fontsize"] is not None
else 12
)
plt.yticks(
np.arange(nFixedTrks),
np.arange(1, nFixedTrks + 1),
fontsize=fontsize,
)
# Check kwargs for figsize
kwargs["figsize"] = (
kwargs["figsize"]
if "figsize" in kwargs and kwargs["figsize"] is not None
else (0.9 * nFixedTrks, 0.5 * len(variable_list))
)
plt.ylabel(xlabel, fontsize=fontsize)
plt.ylim(-0.5, nFixedTrks - 0.5)
# ylabels. Order must be the same as in the Vardict
xticklabels = [
"$s_{d0}$",
"$s_{z0}$",
"PIX1 hits",
"IBL hits",
"shared IBL hits",
"split IBL hits",
"shared pixel hits",
"split pixel hits",
"shared SCT hits",
"log" + r" $p_T^{frac}$",
"log" + r" $\mathrm{\Delta} R$",
"nPixHits",
"nSCTHits",
"$d_0$",
r"$z_0 \sin \theta$",
]
# Set log of x and y axis to False
kwargs["logx"] = False
kwargs["logy"] = False
plt.xticks(
np.arange(nFeatures),
xticklabels[:nFeatures],
rotation=45,
fontsize=fontsize,
# Check that the number of variables are similar in the map and the var_dict
if len(variable_list) != gradient_map.shape[0]:
raise ValueError(
"Number of variables in variable list and in gradient dict are "
"not the same!"
)
else:
fig = plt.figure(figsize=(0.7 * nFixedTrks, 0.7 * nFeatures))
# Define the colour scales and maps for the flavours
colour_scale = np.max(np.abs(gradient_map))
plt.xticks(
np.arange(nFixedTrks),
np.arange(1, nFixedTrks + 1),
fontsize=fontsize,
)
# Init the saliency plot
saliency_plot = PlotBase(**kwargs)
saliency_plot.initialise_figure()
plt.xlabel(xlabel, fontsize=fontsize)
plt.xlim(-0.5, nFixedTrks - 0.5)
# ylabels. Order must be the same as in the Vardict
yticklabels = [
"$s_{d0}$",
"$s_{z0}$",
"PIX1 hits",
"IBL hits",
"shared IBL hits",
"split IBL hits",
"shared pixel hits",
"split pixel hits",
"shared SCT hits",
"log" + r" $p_T^{frac}$",
"log" + r" $\mathrm{\Delta} R$",
"nPixHits",
"nSCTHits",
"$d_0$",
r"$z_0 \sin \theta$",
]
plt.yticks(np.arange(nFeatures), yticklabels[:nFeatures], fontsize=fontsize)
# Set x- and y-ticks
saliency_plot.axis_top.set_xticks(
ticks=np.arange(nFixedTrks),
labels=np.arange(1, nFixedTrks + 1),
fontsize=kwargs["fontsize"],
)
saliency_plot.axis_top.set_yticks(
ticks=np.arange(len(variable_list)),
labels=variable_list,
fontsize=kwargs["fontsize"],
)
im = plt.imshow(
gradient_map,
cmap=cmaps[jet_flavour],
# Plot saliency