Merge branch 'train_implementation' into 'master'

Train implementation

See merge request mguth/umami!28

README.md

@@ -44,7 +44,7 @@ There are two different labeling available, the ```HadronConeExclTruthLabelID```
 | 44            | double c-jets   | 
 | 55            | double b-jets   | 
 
-For the ```HadronConeExclTruthLabelID``` labeling, the categories `4` and `44` as well as `5` and `55` are combined.
+For the ```HadronConeExclTruthLabelID``` labeling, the categories `4` and `44` as well as `5`, `54` and `55` are combined.
 
 ### Ntuple preparation for b-,c- & light-jets
 These jets are taken from ttbar and Z' events.

docs/MC-Samples.md

@@ -5,15 +5,18 @@ The FTAG1 derivations and the most recent ntuples for PFlow with the new RNNIP,
 
 ## Default FTAG Samples (ttbar and Z')
 
-| Sample | h5 ntuples    |  FTAG1 derivations| AOD |
-| ------------- | ---------------- | ---------------- | ---------------- |
-|MC16a - ttbar             |  | mc16_13TeV.410470.PhPy8EG_A14_ttbar_hdamp258p75_nonallhad.deriv.DAOD_FTAG1.e6337_s3126_r9364_p3985 | |
-|MC16a - Z'  |           |     mc16_13TeV.427080.Pythia8EvtGen_A14NNPDF23LO_flatpT_Zprime.deriv.DAOD_FTAG1.e5362_s3126_r9364_p3985 | mc16_13TeV.427080.Pythia8EvtGen_A14NNPDF23LO_flatpT_Zprime.recon.AOD.e5362_s3126_r9364 |
-|MC16d - ttbar             | user.mguth.410470.btagTraining.e6337_s3126_r10201_p3985.EMPFlow.2020-02-14-T232210-R26303_output.h5 | mc16_13TeV.410470.PhPy8EG_A14_ttbar_hdamp258p75_nonallhad.deriv.DAOD_FTAG1.e6337_s3126_r10201_p3985 | |
-|MC16d - Z'             | user.mguth.427080.btagTraining.e5362_s3126_r10201_p3985.EMPFlow.2020-02-14-T232210-R26303_output.h5 | mc16_13TeV.427080.Pythia8EvtGen_A14NNPDF23LO_flatpT_Zprime.deriv.DAOD_FTAG1.e5362_s3126_r10201_p3985 | mc16_13TeV.427080.Pythia8EvtGen_A14NNPDF23LO_flatpT_Zprime.recon.AOD.e5362_s3126_r10201 |
-|MC16d - Z' extended             |  user.mguth.427081.btagTraining.e6928_e5984_s3126_r10201_r10210_p3985.EMPFlow.2020-02-15-T225316-R8334_output.h5 | mc16_13TeV.427081.Pythia8EvtGen_A14NNPDF23LO_flatpT_Zprime_Extended.deriv.DAOD_FTAG1.e6928_e5984_s3126_r10201_r10210_p3985 | mc16_13TeV.427081.Pythia8EvtGen_A14NNPDF23LO_flatpT_Zprime_Extended.recon.AOD.e6928_s3126_r10201 |
-|MC16e - ttbar             |  | mc16_13TeV.410470.PhPy8EG_A14_ttbar_hdamp258p75_nonallhad.deriv.DAOD_FTAG1.e6337_s3126_r10724_p3985 | |
-|MC16e - Z'      |   | mc16_13TeV.427080.Pythia8EvtGen_A14NNPDF23LO_flatpT_Zprime.deriv.DAOD_FTAG1.e5362_s3126_r10724_p3985 | mc16_13TeV.427080.Pythia8EvtGen_A14NNPDF23LO_flatpT_Zprime.recon.AOD.e5362_s3126_r10724 |
+| Sample | h5 ntuples | h5 ntuples (looser track selection)   |  FTAG1 derivations| AOD |
+| ------------- | ---------------- | ---------------- | ---------------- | ---------------- |
+|MC16a - ttbar           |  |  | mc16_13TeV.410470.PhPy8EG_A14_ttbar_hdamp258p75_nonallhad.deriv.DAOD_FTAG1.e6337_s3126_r9364_p3985 | |
+|MC16a - Z'  |         |  |     mc16_13TeV.427080.Pythia8EvtGen_A14NNPDF23LO_flatpT_Zprime.deriv.DAOD_FTAG1.e5362_s3126_r9364_p3985 | mc16_13TeV.427080.Pythia8EvtGen_A14NNPDF23LO_flatpT_Zprime.recon.AOD.e5362_s3126_r9364 |
+|MC16d - ttbar             | user.mguth.410470.btagTraining.e6337_s3126_r10201_p3985.EMPFlow.2020-02-14-T232210-R26303_output.h5 | user.mguth.410470.btagTraining.e6337_s3126_r10201_p3985.EMPFlow_looser-track_selection.2020-07-01-T193555-R26654_output.h5 | mc16_13TeV.410470.PhPy8EG_A14_ttbar_hdamp258p75_nonallhad.deriv.DAOD_FTAG1.e6337_s3126_r10201_p3985 | |
+|MC16d - Z'             | user.mguth.427080.btagTraining.e5362_s3126_r10201_p3985.EMPFlow.2020-02-14-T232210-R26303_output.h5 | user.mguth.427080.btagTraining.e5362_s3126_r10201_p3985.EMPFlow_looser-track_selection.2020-07-01-T193555-R26654_output.h5| mc16_13TeV.427080.Pythia8EvtGen_A14NNPDF23LO_flatpT_Zprime.deriv.DAOD_FTAG1.e5362_s3126_r10201_p3985 | mc16_13TeV.427080.Pythia8EvtGen_A14NNPDF23LO_flatpT_Zprime.recon.AOD.e5362_s3126_r10201 |
+|MC16d - Z' extended             |  user.mguth.427081.btagTraining.e6928_e5984_s3126_r10201_r10210_p3985.EMPFlow.2020-02-15-T225316-R8334_output.h5 | user.mguth.427081.btagTraining.e6928_e5984_s3126_r10201_r10210_p3985.EMPFlow_looser-track_selection.2020-07-01-T195748-R1_output.h5 | mc16_13TeV.427081.Pythia8EvtGen_A14NNPDF23LO_flatpT_Zprime_Extended.deriv.DAOD_FTAG1.e6928_e5984_s3126_r10201_r10210_p3985 | mc16_13TeV.427081.Pythia8EvtGen_A14NNPDF23LO_flatpT_Zprime_Extended.recon.AOD.e6928_s3126_r10201 |
+|MC16e - ttbar           |  |  | mc16_13TeV.410470.PhPy8EG_A14_ttbar_hdamp258p75_nonallhad.deriv.DAOD_FTAG1.e6337_s3126_r10724_p3985 | |
+|MC16e - Z'      | |  | mc16_13TeV.427080.Pythia8EvtGen_A14NNPDF23LO_flatpT_Zprime.deriv.DAOD_FTAG1.e5362_s3126_r10724_p3985 | mc16_13TeV.427080.Pythia8EvtGen_A14NNPDF23LO_flatpT_Zprime.recon.AOD.e5362_s3126_r10724 |
+
+
+
 
 The Z' & Z+jets FTAG 1 derivations were requested [here](https://its.cern.ch/jira/browse/ATLFTAGDPD-216)
 

examples/DL1r-PFlow-Training-config.yaml

@@ -2,7 +2,9 @@ model_name: DL1r-PFlow_new-taggers-stats-22M
 preprocess_config: examples/PFlow-Preprocessing.yaml
 train_file: /work/ws/nemo/fr_mg1150-umami-0/ntuples-p3985/umami_default/hybrids/preprocessed/PFlow-hybrid_70-test-preprocessed_shuffled_test.h5
 validation_file: /work/ws/nemo/fr_mg1150-umami-0/ntuples-p3985/umami_default/hybrids/MC16d_hybrid-odd_1_PFlow-no_pTcuts-file_0.h5 
+test_file: /work/ws/nemo/fr_mg1150-umami-0/ntuples-p3985/umami_default/hybrids/MC16d_hybrid-odd_1_PFlow-no_pTcuts-file_1.h5 
 add_validation_file: /work/ws/nemo/fr_mg1150-umami-0/ntuples-p3985/umami_default/hybrids/MC16d_hybrid-ext_odd_0_PFlow-no_pTcuts-file_0.h5
+add_test_file: /work/ws/nemo/fr_mg1150-umami-0/ntuples-p3985/umami_default/hybrids/MC16d_hybrid-ext_odd_0_PFlow-no_pTcuts-file_1.h5
 var_dict: umami/configs/DL1r_Variables.yaml
 NN_structure:  
     lr: 0.01