Umami merge requestshttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests2022-05-04T09:25:52+02:00https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/531Update Sphinx theme2022-05-04T09:25:52+02:00Joschka BirkUpdate Sphinx theme## Summary
This MR introduces the following changes
* updates the theme used in the Sphinx documentation## Summary
This MR introduces the following changes
* updates the theme used in the Sphinx documentationJoschka BirkJoschka Birkhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/536Sphinx-docs: Change to pydata theme2022-05-10T11:03:22+02:00Joschka BirkSphinx-docs: Change to pydata theme## Summary
This MR introduces the following changes
* Change the Sphinx theme once again. [This theme is the one used by pydata, i.e. `numpy`, `scipy`, `pandas`, ...](https://pydata-sphinx-theme.readthedocs.io/en/stable/)## Summary
This MR introduces the following changes
* Change the Sphinx theme once again. [This theme is the one used by pydata, i.e. `numpy`, `scipy`, `pandas`, ...](https://pydata-sphinx-theme.readthedocs.io/en/stable/)https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/538Fix integration test artifacts2022-05-16T10:34:41+02:00Alexander FrochFix integration test artifacts## Summary
This MR introduces the following changes
* Fixing artifact issues in the integration tests chain. The names of the output directories were not correct and therefore not added to the artifacts.
## Conformity
- [X] [Changelog...## Summary
This MR introduces the following changes
* Fixing artifact issues in the integration tests chain. The names of the output directories were not correct and therefore not added to the artifacts.
## Conformity
- [X] [Changelog entry](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/changelog.md)
- [X] [Documentation](https://umami-docs.web.cern.ch)
- [X] [Development guidelines](https://umami-docs.web.cern.ch/setup/development/)
- [X] [Style guides](https://umami-docs.web.cern.ch/setup/development/good-practices/)Alexander FrochAlexander Frochhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/539Change to librep2022-05-16T11:45:16+02:00Joschka BirkChange to librep## Summary
This MR introduces the following changes
* moves the code of the line-block replacement that we are using in the markdown docs to a [separate repository on GitHub](https://github.com/umami-hep/librep)
## Conformity
- [x] [C...## Summary
This MR introduces the following changes
* moves the code of the line-block replacement that we are using in the markdown docs to a [separate repository on GitHub](https://github.com/umami-hep/librep)
## Conformity
- [x] [Changelog entry](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/changelog.md)
- [x] [Documentation](https://umami-docs.web.cern.ch)
- [x] ~[Development guidelines](https://umami-docs.web.cern.ch/setup/development/)~
- [x] ~[Style guides](https://umami-docs.web.cern.ch/setup/development/good-practices/)~Joschka BirkJoschka Birkhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/540Moving to puma package2022-05-18T14:30:13+02:00Manuel GuthMoving to puma package## Summary
This MR introduces the following changes
* switching to [puma package](https://github.com/umami-hep/puma) as plotting API
Relates to the following issues
* relates to https://gitlab.cern.ch/atlas-flavor-tagging-tools/algo...## Summary
This MR introduces the following changes
* switching to [puma package](https://github.com/umami-hep/puma) as plotting API
Relates to the following issues
* relates to https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/issues/168
*
## Conformity
- [x] [Changelog entry](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/changelog.md)
- [x] [Documentation](https://umami-docs.web.cern.ch)
- [x] [Development guidelines](https://umami-docs.web.cern.ch/setup/development/)
- [x] [Style guides](https://umami-docs.web.cern.ch/setup/development/good-practices/)https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/546Adding unit test for prepare_model and minor bug fixes2022-05-20T09:29:30+02:00Alexander FrochAdding unit test for prepare_model and minor bug fixes## Summary
This MR introduces the following changes
* Adding a unit test for the `prepare_model` function.
* Adding a `-t` option to xargs to print all the commands executed.
* Moving a logger.warning to a debug which prints the `plot_...## Summary
This MR introduces the following changes
* Adding a unit test for the `prepare_model` function.
* Adding a `-t` option to xargs to print all the commands executed.
* Moving a logger.warning to a debug which prints the `plot_args`.
* Adding `trained_tagger` to excluded variable which are not added to `plot_args`.
## Conformity
- [X] [Changelog entry](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/changelog.md)
- [X] [Documentation](https://umami-docs.web.cern.ch)
- [X] [Development guidelines](https://umami-docs.web.cern.ch/setup/development/)
- [X] [Style guides](https://umami-docs.web.cern.ch/setup/development/good-practices/)Alexander FrochAlexander Frochhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/547Fixing GPU images2022-05-18T17:30:24+02:00Manuel GuthFixing GPU images## Summary
This MR introduces the following changes
* updating tensorflow to version `2.9.0`
* updating PyTorch to version `1.11.0-cuda11.3-cudnn8-runtime`
* short cutting the MR docker image build to start directly
## Conformity
- [...## Summary
This MR introduces the following changes
* updating tensorflow to version `2.9.0`
* updating PyTorch to version `1.11.0-cuda11.3-cudnn8-runtime`
* short cutting the MR docker image build to start directly
## Conformity
- [x] [Changelog entry](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/changelog.md)https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/548Transition to puma package2022-05-20T18:19:09+02:00Joschka BirkTransition to puma package## Summary
This MR introduces the following changes
* follow-up MR for the transition `umami.plotting` -> `puma`
* remove metrics code that is now supported in `puma`
## Conformity
- [x] [Changelog entry](https://gitlab.cern.ch/atlas-...## Summary
This MR introduces the following changes
* follow-up MR for the transition `umami.plotting` -> `puma`
* remove metrics code that is now supported in `puma`
## Conformity
- [x] [Changelog entry](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/changelog.md)
- ~[ ] [Documentation](https://umami-docs.web.cern.ch)~
- [x] [Development guidelines](https://umami-docs.web.cern.ch/setup/development/)
- [x] [Style guides](https://umami-docs.web.cern.ch/setup/development/good-practices/)
- [x] fix pipeline
- [x] remove unnecessary leftovers from umami/plotting/utils.py
- [x] remove metrics code that is ported to pumaJoschka BirkJoschka Birkhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/554Moving Plotting Files in one folder2022-06-07T17:26:45+02:00Alexander FrochMoving Plotting Files in one folder## Summary
This MR introduces the following changes
* Moving all plotting functions of umami (the eval_plots, train_plots and preprocessing_plots) in one module `plotting_tools`.
* Renaming plotting functions so they comply with the PE...## Summary
This MR introduces the following changes
* Moving all plotting functions of umami (the eval_plots, train_plots and preprocessing_plots) in one module `plotting_tools`.
* Renaming plotting functions so they comply with the PEP8 guidelines.
Relates to the following issues
* Closes #186
## Conformity
- [x] [Changelog entry](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/changelog.md)
- [X] [Documentation](https://umami-docs.web.cern.ch)
- [X] [Development guidelines](https://umami-docs.web.cern.ch/setup/development/)
- [X] [Style guides](https://umami-docs.web.cern.ch/setup/development/good-practices/)Alexander FrochAlexander Frochhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/571fixing docker CI stage jump2022-06-22T17:36:26+02:00Manuel Guthfixing docker CI stage jump## Summary
This MR introduces the following changes
* Fixes the order of the CI docker building jobs
This implementation should still allow the base image build to start immediately such that the CI is faster in general, not including...## Summary
This MR introduces the following changes
* Fixes the order of the CI docker building jobs
This implementation should still allow the base image build to start immediately such that the CI is faster in general, not including dependencies from previous jobs (image size does not blow up which was fixed in https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/issues/166) while preserving the stage dependencies for the image builds which depend on the base images.
Relates to the following issues
* Closes #192
## Conformity
- [x] [Changelog entry](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/changelog.md)https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/573Fixing memory leak in Callback functions2022-06-24T10:08:59+02:00Alexander FrochFixing memory leak in Callback functions## Summary
This MR introduces the following changes
* Fixing a memory leak issue in the `model.predict()` and `model.evaluate` functions in the callback.
* Updating `tensorflow` to new `2.9.1`
## Conformity
- [X] [Changelog entry](htt...## Summary
This MR introduces the following changes
* Fixing a memory leak issue in the `model.predict()` and `model.evaluate` functions in the callback.
* Updating `tensorflow` to new `2.9.1`
## Conformity
- [X] [Changelog entry](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/changelog.md)
- [X] [Documentation](https://umami-docs.web.cern.ch)
- [X] [Development guidelines](https://umami-docs.web.cern.ch/setup/development/)
- [X] [Style guides](https://umami-docs.web.cern.ch/setup/development/good-practices/)Alexander FrochAlexander Frochhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/574Fixing image issue in pylint2022-06-24T14:23:30+02:00Alexander FrochFixing image issue in pylint## Summary
This MR introduces the following changes
* Fixing image used for pylint
## Conformity
- [X] [Changelog entry](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/changelog.md)
- [X] [Documentat...## Summary
This MR introduces the following changes
* Fixing image used for pylint
## Conformity
- [X] [Changelog entry](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/changelog.md)
- [X] [Documentation](https://umami-docs.web.cern.ch)
- [X] [Development guidelines](https://umami-docs.web.cern.ch/setup/development/)
- [X] [Style guides](https://umami-docs.web.cern.ch/setup/development/good-practices/)Alexander FrochAlexander Frochhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/576Updating docs and renaming hybrids folder2022-07-06T14:58:27+02:00Alexander FrochUpdating docs and renaming hybrids folder## Summary
This MR introduces the following changes
* Updating the docs for the preprocessing and the training
* Renaming the `hybrids` folder. There are no hybrids inside, only the prepared samples.
* Removing hardcoded `DL1` as namin...## Summary
This MR introduces the following changes
* Updating the docs for the preprocessing and the training
* Renaming the `hybrids` folder. There are no hybrids inside, only the prepared samples.
* Removing hardcoded `DL1` as naming from the `evaluate_model.py`
* Adapt plotting config accordingly
## Conformity
- [X] [Changelog entry](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/changelog.md)
- [X] [Documentation](https://umami-docs.web.cern.ch)
- [X] [Development guidelines](https://umami-docs.web.cern.ch/setup/development/)
- [X] [Style guides](https://umami-docs.web.cern.ch/setup/development/good-practices/)Alexander FrochAlexander Frochhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/577Fixing CI syntax with `cobertura`2022-07-05T10:33:23+02:00Alexander FrochFixing CI syntax with `cobertura`## Summary
This MR introduces the following changes
* Fixing issue with the Gitlab version and the current syntax of the coverage check `cobertura`.
## Conformity
- [x] [Changelog entry](https://gitlab.cern.ch/atlas-flavor-tagging-too...## Summary
This MR introduces the following changes
* Fixing issue with the Gitlab version and the current syntax of the coverage check `cobertura`.
## Conformity
- [x] [Changelog entry](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/changelog.md)
- [X] [Documentation](https://umami-docs.web.cern.ch)
- [X] [Development guidelines](https://umami-docs.web.cern.ch/setup/development/)
- [X] [Style guides](https://umami-docs.web.cern.ch/setup/development/good-practices/)Alexander FrochAlexander Frochhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/579fixing coverage report2022-07-05T17:21:14+02:00Manuel Guthfixing coverage report## Summary
This MR introduces the following changes
* fixes coverage report as described in https://docs.gitlab.com/ee/ci/pipelines/settings.html#add-test-coverage-results-using-coverage-keyword
## Conformity
- [ ] [Changelog entry]...## Summary
This MR introduces the following changes
* fixes coverage report as described in https://docs.gitlab.com/ee/ci/pipelines/settings.html#add-test-coverage-results-using-coverage-keyword
## Conformity
- [ ] [Changelog entry](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/changelog.md)Manuel GuthManuel Guthhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/583Adding support for hits2022-07-25T15:00:52+02:00Martino TanasiniAdding support for hits## Summary
This MR introduces the following changes
* Allowing for having track collections with different shapes
* Hits can now be used the same way as tracks
* Added integration test for the preprocessing of hits
* Fixing small issue...## Summary
This MR introduces the following changes
* Allowing for having track collections with different shapes
* Hits can now be used the same way as tracks
* Added integration test for the preprocessing of hits
* Fixing small issues in the scaling when some parts of the variables are not in the variables_dict
Relates to the following issues
* Closes #199
## Conformity
- [X] [Changelog entry](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/changelog.md)
- [X] [Documentation](https://umami-docs.web.cern.ch)
- [X] [Development guidelines](https://umami-docs.web.cern.ch/setup/development/)
- [X] [Style guides](https://umami-docs.web.cern.ch/setup/development/good_practices_code/)Alexander FrochAlexander Frochhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/586Reducing artifact size for the preprocessing integration tests2022-07-22T12:50:10+02:00Alexander FrochReducing artifact size for the preprocessing integration tests## Summary
This MR introduces the following changes
* Reducing the artifact size by removing all intermediate files from the preprocessing. When you run the test locally, the full folder with all intermediate files will be available. T...## Summary
This MR introduces the following changes
* Reducing the artifact size by removing all intermediate files from the preprocessing. When you run the test locally, the full folder with all intermediate files will be available. The new folder is the old name ending with `_full`.
* Set expiration date for unit test artifacts
Relates to the following issues
* Closes #200
## Conformity
- [X] [Changelog entry](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/changelog.md)
- [X] [Documentation](https://umami-docs.web.cern.ch)
- [X] [Development guidelines](https://umami-docs.web.cern.ch/setup/development/)
- [X] [Style guides](https://umami-docs.web.cern.ch/setup/development/good-practices/)Alexander FrochAlexander Frochhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/594Resolve "Checking if placeholder files are changed"2022-07-28T15:10:18+02:00Manuel GuthResolve "Checking if placeholder files are changed"## Summary
This MR introduces the following changes
* Makes bot comment on MR if file was changed which is used as placeholder in documentation
Relates to the following issues
* closes #187
## Conformity
- [x] [Changelog entry](htt...## Summary
This MR introduces the following changes
* Makes bot comment on MR if file was changed which is used as placeholder in documentation
Relates to the following issues
* closes #187
## Conformity
- [x] [Changelog entry](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/changelog.md)Manuel GuthManuel Guthhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/595Resolve "high level function to load scaled tracks and jets from h5 file"2022-08-03T15:03:43+02:00Manuel GuthResolve "high level function to load scaled tracks and jets from h5 file"## Summary
This MR introduces the following changes
* Adds script to apply scaling and shifting to a test sample and write it to disk (useful for taggers not trained within umami)
Relates to the following issues
* Closes #201
## Con...## Summary
This MR introduces the following changes
* Adds script to apply scaling and shifting to a test sample and write it to disk (useful for taggers not trained within umami)
Relates to the following issues
* Closes #201
## Conformity
- [x] [Changelog entry](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/changelog.md)
- [x] [Documentation](https://umami-docs.web.cern.ch)
- [x] [Development guidelines](https://umami-docs.web.cern.ch/setup/development/)
- [x] [Style guides](https://umami-docs.web.cern.ch/setup/development/good-practices/)Manuel GuthManuel Guthhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/603Draft: Fixing tensorflow import errors2022-08-05T11:33:44+02:00Alexander FrochDraft: Fixing tensorflow import errors## Summary
This MR introduces the following changes
* Fixing the import errors from `tensorflow`. The lazy import tensorflow uses is not accepted by the linters/interpreters correctly, so you get errors although this works fine. Adding...## Summary
This MR introduces the following changes
* Fixing the import errors from `tensorflow`. The lazy import tensorflow uses is not accepted by the linters/interpreters correctly, so you get errors although this works fine. Adding the `.python` fixes the issue for the linters/interpreters.
## Conformity
- [ ] [Changelog entry](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/changelog.md)
- [X] [Documentation](https://umami-docs.web.cern.ch)
- [X] [Development guidelines](https://umami-docs.web.cern.ch/setup/development/)
- [X] [Style guides](https://umami-docs.web.cern.ch/setup/development/good-practices/)Alexander FrochAlexander Froch