Umami merge requestshttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests2021-10-19T14:05:21+02:00https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/194Merging Preprocessing-Remake in Master2021-10-19T14:05:21+02:00Alexander FrochMerging Preprocessing-Remake in MasterPreprocessing rewriteAlexander FrochAlexander Frochhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/171Adding new tf_tools and new conditional deep sets model2021-10-19T11:47:28+02:00Alexander FrochAdding new tf_tools and new conditional deep sets modelThis MR restructures the definition of the neural network models. All tensorflow/keras related functions (model definitions, custom layers, generators) are now in a separate folder `umami/tf_tools.
Also the new conditional deep sets...This MR restructures the definition of the neural network models. All tensorflow/keras related functions (model definitions, custom layers, generators) are now in a separate folder `umami/tf_tools.
Also the new conditional deep sets model + all needed custom layers from @mguth and @jraine will be added in this step.Preprocessing rewriteAlexander FrochAlexander Frochhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/162Remove hard-coded labels for DIPS2021-08-18T15:58:22+02:00Alexander FrochRemove hard-coded labels for DIPSThis MR will remove the hard-coded labels in the training process for DIPS. I did this only for DIPS in one go, because the MR is already very large (Sorry for that...) but I wasn't able to make it smaller. But the next MRs concerning th...This MR will remove the hard-coded labels in the training process for DIPS. I did this only for DIPS in one go, because the MR is already very large (Sorry for that...) but I wasn't able to make it smaller. But the next MRs concerning the rewriting of the DL1r and UMAMI tools should be less.
This is a copy of the old branch. The old was buggy and needed to be killed. All the changes and comments from the old branch/MR are added here. For the comments look at !155Preprocessing rewriteAlexander FrochAlexander Frochhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/161Removing hard-coded labels from train_tools for UMAMI and DL1r2021-08-24T15:12:27+02:00Alexander FrochRemoving hard-coded labels from train_tools for UMAMI and DL1rThis MR adds removes the hard-coded labels completely now from the `train_tools` also from UMAMI and DL1r. Unit tests are provided. MR is related to &1.
a step towards #58This MR adds removes the hard-coded labels completely now from the `train_tools` also from UMAMI and DL1r. Unit tests are provided. MR is related to &1.
a step towards #58Preprocessing rewriteAlexander FrochAlexander Frochhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/155Remove hard-coded labels for DIPS2021-08-17T18:03:22+02:00Alexander FrochRemove hard-coded labels for DIPSThis MR will remove the hard-coded labels in the training process for DIPS. I did this only for DIPS in one go, because the MR is already very large (Sorry for that...) but I wasn't able to make it smaller. But the next MRs concerning th...This MR will remove the hard-coded labels in the training process for DIPS. I did this only for DIPS in one go, because the MR is already very large (Sorry for that...) but I wasn't able to make it smaller. But the next MRs concerning the rewriting of the DL1r and UMAMI tools should be less.Preprocessing rewriteAlexander FrochAlexander Frochhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/522renaming variables to match snake_case style2022-04-19T17:27:42+02:00Manuel Guthrenaming variables to match snake_case style## Summary
This MR introduces the following changes
* Renaming a bunch of functions and variables to match snake_case style
Relates to the following issues
* first step towards https://gitlab.cern.ch/atlas-flavor-tagging-tools/algori...## Summary
This MR introduces the following changes
* Renaming a bunch of functions and variables to match snake_case style
Relates to the following issues
* first step towards https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/issues/167
## Conformity
- [x] [Changelog entry](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/changelog.md)
- [x] [Documentation](https://umami-docs.web.cern.ch)
- [x] [Development guidelines](https://umami-docs.web.cern.ch/setup/development/)
- [x] [Style guides](https://umami-docs.web.cern.ch/setup/development/good-practices/)https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/516Fix validation values json in continue_training2022-04-20T16:24:23+02:00Alexander FrochFix validation values json in continue_training## Summary
This MR introduces the following changes
* Fixes an issue where when you use the `continue_training` option, the validation json (where the rejections are saved per epoch and so on) is overwritten when the new training start...## Summary
This MR introduces the following changes
* Fixes an issue where when you use the `continue_training` option, the validation json (where the rejections are saved per epoch and so on) is overwritten when the new training starts.
## Conformity
- [x] [Changelog entry](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/changelog.md)
- [x] [Documentation](https://umami-docs.web.cern.ch)
- [x] [Development guidelines](https://umami-docs.web.cern.ch/setup/development/)
- [x] [Style guides](https://umami-docs.web.cern.ch/setup/development/good-practices/)Alexander FrochAlexander Frochhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/500Adding continue_training option2022-04-05T16:49:15+02:00Alexander FrochAdding continue_training option## Summary
This MR introduces the following changes
* Adding `continue_training` option for all models. With that, you can continue a broken training. For example, if your Job was killed due to time constrains, set this option to `True...## Summary
This MR introduces the following changes
* Adding `continue_training` option for all models. With that, you can continue a broken training. For example, if your Job was killed due to time constrains, set this option to `True` and the latest model is loaded with weights, optimiser state and epoch number. This is then used for continuing the training.
## Conformity
- [X] [Changelog entry](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/changelog.md)
- [X] [Documentation](https://umami-docs.web.cern.ch)
- [X] [Development guidelines](https://umami-docs.web.cern.ch/setup/development/)
- [X] [Style guides](https://umami-docs.web.cern.ch/setup/development/good-practices/)Alexander FrochAlexander Frochhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/444Update TFRecord reader/writer + Adding support for CADS and Umami Cond Att2022-02-28T15:59:10+01:00Alexander FrochUpdate TFRecord reader/writer + Adding support for CADS and Umami Cond AttThis MR adds the following:
* Adding multiple track collections in TFRecords is now available. Closes #109
* Adapting the reader/writer of TFRecords so the correct track collection, conditional info etc. is loaded.
* Adding possibility...This MR adds the following:
* Adding multiple track collections in TFRecords is now available. Closes #109
* Adapting the reader/writer of TFRecords so the correct track collection, conditional info etc. is loaded.
* Adding possibility to train CADS with TFRecords.
* Adapting unit tests of the TFRecords reader/writer.
* Adding Integration tests for CADS with TFRecords, Umami Cond Att and Umami Cond Att with TFRecords. Related to #142.
* Fixing issues in the evaluation chain of Umami Cond Att. Closes #143
* Update documentation of preprocessing config section `convert_to_tfrecord`.Alexander FrochAlexander Frochhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/436Add TFRecords support for CADS2022-02-22T17:36:37+01:00Alexander FrochAdd TFRecords support for CADSThis MR adds TFRecords support for CADSThis MR adds TFRecords support for CADSAlexander FrochAlexander Frochhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/435Fixing LRR issue and define validation n_jets and WP2022-02-18T11:54:43+01:00Alexander FrochFixing LRR issue and define validation n_jets and WPThis MR adds the following:
* Adding a validation specific `n_jets` and `WP` to the train configs (It is backward compatible). Documentation is added.
* Fixing an issue when disabling the Learning Rate Reducer (is fixed with a list wher...This MR adds the following:
* Adding a validation specific `n_jets` and `WP` to the train configs (It is backward compatible). Documentation is added.
* Fixing an issue when disabling the Learning Rate Reducer (is fixed with a list where the callbacks are appended to when they are initialized).
* Adapting the `plotting_epoch_performance.py` script. Per default, the validation json which is created with the `MyCallback` function is used when no argument (except the `-c`) is given (based on the parameters inside the train config).
* The `--dict` option in `plotting_epoch_performance.py` is still available. With this, you can use a validation json which not fits (name-wise).
* Added the `--recalculate` flag to `plotting_epoch_performance.py` which is now the command to recalculate the validation json.Alexander FrochAlexander Frochhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/430Fixing all issues of darglint and removing unused functions2022-02-16T16:50:23+01:00Alexander FrochFixing all issues of darglint and removing unused functionsThis MR fixes all `darglint` issues and removes the `MakePlots` and `Plot_vars` fuctions (outdated). Related to #61 and #145
The file `umami/train_tools/Plotting.py` is now excluded from darglint due to issues of darglint with the sphi...This MR fixes all `darglint` issues and removes the `MakePlots` and `Plot_vars` fuctions (outdated). Related to #61 and #145
The file `umami/train_tools/Plotting.py` is now excluded from darglint due to issues of darglint with the sphinx documentation style in the docstrings
closes https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/issues/97 #145Alexander FrochAlexander Frochhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/422CI improvements2022-02-15T18:51:20+01:00Manuel GuthCI improvementsThis MR
- fixes pylint, which started to fail after !418
- updates until test CI definition making it easier
- updates CI for pushing to hub removing unnecessary duplicationThis MR
- fixes pylint, which started to fail after !418
- updates until test CI definition making it easier
- updates CI for pushing to hub removing unnecessary duplicationhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/421Updating doc strings for yaml tools2022-02-15T21:21:58+01:00Manuel GuthUpdating doc strings for yaml toolshttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/379Renaming DIPS Cond Att to CADS + Adding option to load optimiser2022-01-28T16:33:38+01:00Alexander FrochRenaming DIPS Cond Att to CADS + Adding option to load optimiserThis MR does the following:
* Renaming all Dips Cond att things to CADS.
* If a model is now loaded (for continuing the training), the optimizer is now also loaded correctly and not freshly compiled.
* Adding the `load_optimiser` option...This MR does the following:
* Renaming all Dips Cond att things to CADS.
* If a model is now loaded (for continuing the training), the optimizer is now also loaded correctly and not freshly compiled.
* Adding the `load_optimiser` option (per default `True`). When loading a model, you can decide if the optimiser is loaded from the model file (for continuing a training) or a new optimiser is initialised (use the model as start point for a new training).
* Documentation for the new `load_optimiser` option is added.Alexander FrochAlexander Frochhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/374Merge master in protected branch2022-01-27T11:13:37+01:00Alexander FrochMerge master in protected branchThis MR adds the current master to `112-flexible-validation-test-file-definition`This MR adds the current master to `112-flexible-validation-test-file-definition`Alexander FrochAlexander Frochhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/351Merge Master in Validation remake branch2022-01-19T11:46:37+01:00Alexander FrochMerge Master in Validation remake branchAlexander FrochAlexander Frochhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/350Merging Master2022-01-19T11:30:39+01:00Alexander FrochMerging MasterAlexander FrochAlexander Frochhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/349Generalise validation files2022-01-25T17:52:01+01:00Joschka BirkGeneralise validation filesThe current changes work for DIPS.
Umami will be added in future commits (the pipeline will crash for sure at the moment).
Changes:
- [x] Configuration.py adjusted to new structure
- [x] Change plotting and validation calculation for DI...The current changes work for DIPS.
Umami will be added in future commits (the pipeline will crash for sure at the moment).
Changes:
- [x] Configuration.py adjusted to new structure
- [x] Change plotting and validation calculation for DIPS
- [x] Update unit tests for DIPS
- [x] Update integration tests for DIPS
- [x] Change plotting and validation calculation for Umami
- [x] Update unit tests for Umami
- [x] Update integration tests for Umami
- [x] Update documentation for validation files part
- [x] Remove the `validation_file` syntax everywhere in the `test_NN_tools.py` file
- [x] Update code for dl1r
- [x] Update plotting for Umami
- [x] Remove `get_variable_cuts` function and corresponding tests
Docstring style in `Plotting.py` will be updated in !360 to avoid even more changed lines in this MR.Joschka BirkJoschka Birkhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/346Pylint improvements2022-01-19T11:12:19+01:00Manuel GuthPylint improvementsadresses #105adresses #105