Umami merge requestshttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests2021-08-04T15:16:03+02:00https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/132Adding calc_bins function to evaluation tools2021-08-04T15:16:03+02:00Alexander FrochAdding calc_bins function to evaluation toolsThis MR does the following things:
1: Adding functions `calc_bins` and `calc_ratio` which calculates the bins and their unc and also the ratio steps and their unc.
2: Adding the `flavour_colors`, `flavour_labels` and `hist_err_sty...This MR does the following things:
1: Adding functions `calc_bins` and `calc_ratio` which calculates the bins and their unc and also the ratio steps and their unc.
2: Adding the `flavour_colors`, `flavour_labels` and `hist_err_style` to the `global_config`.
3: Adding Ratio Uncertainties.
4: Adding small warning in the `plot_input_vars`.Alexander FrochAlexander Frochhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/339Adding more flexibility for validation and test file usage2022-01-27T19:28:09+01:00Manuel GuthAdding more flexibility for validation and test file usageThis is a kind of epic MR collecting intermediate changes towards the resolution of the issue
Closes #112This is a kind of epic MR collecting intermediate changes towards the resolution of the issue
Closes #112Manuel GuthManuel Guthhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/171Adding new tf_tools and new conditional deep sets model2021-10-19T11:47:28+02:00Alexander FrochAdding new tf_tools and new conditional deep sets modelThis MR restructures the definition of the neural network models. All tensorflow/keras related functions (model definitions, custom layers, generators) are now in a separate folder `umami/tf_tools.
Also the new conditional deep sets...This MR restructures the definition of the neural network models. All tensorflow/keras related functions (model definitions, custom layers, generators) are now in a separate folder `umami/tf_tools.
Also the new conditional deep sets model + all needed custom layers from @mguth and @jraine will be added in this step.Preprocessing rewriteAlexander FrochAlexander Frochhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/151Adding resampling class2021-08-25T15:33:25+02:00Manuel GuthAdding resampling classReimplementation of the resampling.
Using a Baseclass to be used with all sampling methods
related to &1
closes https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/issues/16
introduces new yaml tool where the loader...Reimplementation of the resampling.
Using a Baseclass to be used with all sampling methods
related to &1
closes https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/issues/16
introduces new yaml tool where the loader can deal with file includes
after this MR there are some follow ups which need to be done (wanted to make this available first that people can start with other implementations of resampling methods): #60 #61 #62Preprocessing rewritehttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/177Adding scaling to preprocessing and re-activate CI2021-10-13T15:49:06+02:00Alexander FrochAdding scaling to preprocessing and re-activate CIThis MR adds the Scaling and Write classes for the preprocessing. The iterations are still not supported. I will open an issue for that.
Also, the integration tests are adapted and activated.
Merge after !170
Closes #66
Closes #58This MR adds the Scaling and Write classes for the preprocessing. The iterations are still not supported. I will open an issue for that.
Also, the integration tests are adapted and activated.
Merge after !170
Closes #66
Closes #58Preprocessing rewriteAlexander FrochAlexander Frochhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/192Add weighting class2021-12-01T11:32:57+01:00Frederic RennerAdd weighting classThis adds the Weighting class to calculate sample_weights as a preprocessing method. I also did some general refactoring from the ```Undersampling``` class into the ```Resampling``` base class to avoid code duplication. I would like to h...This adds the Weighting class to calculate sample_weights as a preprocessing method. I also did some general refactoring from the ```Undersampling``` class into the ```Resampling``` base class to avoid code duplication. I would like to hear your opinion on it.
Closes #64https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/721Bug fixing and restructure/simplifying resampling code2023-04-17T17:31:04+02:00Ivan OleksiyukBug fixing and restructure/simplifying resampling code## Summary
current resampling code still have code duplicates where some of the older code is still bug prone. Such code variations should be merged to fix the bugs persistent in the old code whereas they are fixed in the new.
E.g. when...## Summary
current resampling code still have code duplicates where some of the older code is still bug prone. Such code variations should be merged to fix the bugs persistent in the old code whereas they are fixed in the new.
E.g. when chunk_size<len(indices) resampling_generator cuts off the last non-full chunk whereas the sampling_generator append the rest to a previous chunk (which is also a bad practice, as intuitively chunk_size=max_chunk_size so that one is sure not to run out of memory although factor that is less then 2 plays role really seldom)
This MR introduces the following changes
* fix a BUG in resampling_base:Resampling:resampling_generator that throws away the last not full bath (chunk) ...
* ... by merging it with resampling_base:sampling_generator that does not have such a bug
* Bug in resampling_base:write_file made writing the last jet batch doubled into the output file which might have lead to a lot of problems, now it is fixed
## Conformity
- [x] [Changelog entry](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/changelog.md)
- [x] [Documentation](https://umami-docs.web.cern.ch)
- [x] [Development guidelines](https://umami-docs.web.cern.ch/setup/development/)
- [x] [Style guides](https://umami-docs.web.cern.ch/setup/development/good-practices/)Ivan OleksiyukIvan Oleksiyukhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/149Implement PDF sampling class + tests2021-08-10T13:45:46+02:00Alexander FrochImplement PDF sampling class + testsThis MR adds the PDF Sampling class with the needed functions for the resampling. Also unit tests for the class are provided.
This MR is related to &1This MR adds the PDF Sampling class with the needed functions for the resampling. Also unit tests for the class are provided.
This MR is related to &1Preprocessing rewriteAlexander FrochAlexander Frochhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/740Integration of upp pre-processing package into umami framework2023-11-29T11:36:38+01:00Ivan OleksiyukIntegration of upp pre-processing package into umami framework## Summary
This MR introduces the following changes
* possibility to comfortably use upp preprocessing in umami framework,
basically one can now do all the same pre-processing steps but with using upp config file (with some tweaks)!
*...## Summary
This MR introduces the following changes
* possibility to comfortably use upp preprocessing in umami framework,
basically one can now do all the same pre-processing steps but with using upp config file (with some tweaks)!
* examples on how to utilise upp config file in preprocessing
Relates to the following issues
* https://github.com/umami-hep/umami-preprocessing/pull/30
* https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/issues/240
* https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/issues/233
* https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/issues/234
* https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/issues/231
## Conformity
- [x] [Changelog entry](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/changelog.md)
- [x] [Documentation](https://umami-docs.web.cern.ch)
- [x] [Development guidelines](https://umami-docs.web.cern.ch/setup/development/)
- [x] [Style guides](https://umami-docs.web.cern.ch/setup/development/good-practices/)Rewrite PreprocessingIvan OleksiyukIvan Oleksiyukhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/194Merging Preprocessing-Remake in Master2021-10-19T14:05:21+02:00Alexander FrochMerging Preprocessing-Remake in MasterPreprocessing rewriteAlexander FrochAlexander Frochhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/302move weights calculation to resampling2021-12-21T18:29:58+01:00Frederic Rennermove weights calculation to resamplingRemove weight initialization from prepare and move the weights calculation to the write step. Also moved the options for the weighting procedure into the sampling options of the preprocess config.
Closes #96Remove weight initialization from prepare and move the weights calculation to the write step. Also moved the options for the weighting procedure into the sampling options of the preprocess config.
Closes #96Frederic RennerFrederic Rennerhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/162Remove hard-coded labels for DIPS2021-08-18T15:58:22+02:00Alexander FrochRemove hard-coded labels for DIPSThis MR will remove the hard-coded labels in the training process for DIPS. I did this only for DIPS in one go, because the MR is already very large (Sorry for that...) but I wasn't able to make it smaller. But the next MRs concerning th...This MR will remove the hard-coded labels in the training process for DIPS. I did this only for DIPS in one go, because the MR is already very large (Sorry for that...) but I wasn't able to make it smaller. But the next MRs concerning the rewriting of the DL1r and UMAMI tools should be less.
This is a copy of the old branch. The old was buggy and needed to be killed. All the changes and comments from the old branch/MR are added here. For the comments look at !155Preprocessing rewriteAlexander FrochAlexander Frochhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/164Remove hardcoded labels from input vars tools2021-08-25T16:57:05+02:00Alexander FrochRemove hardcoded labels from input vars toolsThis MR removes the hard-coded labels from the input variable plotting tools and update them. I cannot add a MR dependency here, but the changes from !161 are already implemented here. Merge !161 first.
This MR is related to &1This MR removes the hard-coded labels from the input variable plotting tools and update them. I cannot add a MR dependency here, but the changes from !161 are already implemented here. Merge !161 first.
This MR is related to &1Preprocessing rewriteAlexander FrochAlexander Frochhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/170Removing hard-coded labels from evaluation tools2021-09-15T16:27:48+02:00Alexander FrochRemoving hard-coded labels from evaluation toolsThis MR removes the hard-coded labels from the evaluation tools and cleans up the code concerning the evaluation.
Unit tests for the new/rewritten functions are provided.
Also, I tried running the integration tests for the preproce...This MR removes the hard-coded labels from the evaluation tools and cleans up the code concerning the evaluation.
Unit tests for the new/rewritten functions are provided.
Also, I tried running the integration tests for the preprocessing and the training. I updated the concerning test files according to the new behavior of the functions and calls.
In addition to the callbacks, the training metrics are now saved in an extra json file in the corresponding model folder with the name `history.json`. The handling of this file, when it comes to plotting, is providedPreprocessing rewriteAlexander FrochAlexander Frochhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/161Removing hard-coded labels from train_tools for UMAMI and DL1r2021-08-24T15:12:27+02:00Alexander FrochRemoving hard-coded labels from train_tools for UMAMI and DL1rThis MR adds removes the hard-coded labels completely now from the `train_tools` also from UMAMI and DL1r. Unit tests are provided. MR is related to &1.
a step towards #58This MR adds removes the hard-coded labels completely now from the `train_tools` also from UMAMI and DL1r. Unit tests are provided. MR is related to &1.
a step towards #58Preprocessing rewriteAlexander FrochAlexander Frochhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/675Rewriting train config Configuration2022-12-06T17:03:27+01:00Alexander FrochRewriting train config Configuration## Summary
This MR introduces the following changes
* Adding dataclasses for the train_config.
Relates to the following issues
* Closes #208
* #181 for the train configs
## Conformity
- [x] [Changelog entry](https://gitlab.cern.ch...## Summary
This MR introduces the following changes
* Adding dataclasses for the train_config.
Relates to the following issues
* Closes #208
* #181 for the train configs
## Conformity
- [x] [Changelog entry](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/changelog.md)
- [X] [Documentation](https://umami-docs.web.cern.ch)
- [X] [Development guidelines](https://umami-docs.web.cern.ch/setup/development/)
- [X] [Style guides](https://umami-docs.web.cern.ch/setup/development/good-practices/)Alexander FrochAlexander Frochhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/150Updated preprocessing: sample preparation using new PrepareSample class2021-08-25T10:37:01+02:00Manuel GuthUpdated preprocessing: sample preparation using new PrepareSample classThis MR implements the first version of the new `PrepareSamples` class, making everything independent of hardcoded flavours.
Furthermore, it removes the merging step. The output h5 files of the sample preparation are now written out inc...This MR implements the first version of the new `PrepareSamples` class, making everything independent of hardcoded flavours.
Furthermore, it removes the merging step. The output h5 files of the sample preparation are now written out incrementally.
Finally, the MR carries over latest changes in the master branch.
related to &1Preprocessing rewritePhilipp GadowManuel GuthPhilipp Gadow