Umami merge requestshttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests2022-03-14T10:55:42+01:00https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/475dummy data generation2022-03-14T10:55:42+01:00Manuel Guthdummy data generation## Summary
This MR introduces the following changes
* ability to generate dummy data, especially for plotting tests
* reorganises plotting python API docs
* deactivates umami slim image creation in all branches
## Conformity
- [x] [C...## Summary
This MR introduces the following changes
* ability to generate dummy data, especially for plotting tests
* reorganises plotting python API docs
* deactivates umami slim image creation in all branches
## Conformity
- [x] [Changelog entry](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/changelog.md)
- [x] [Documentation](https://umami-docs.web.cern.ch)
- [x] [Development guidelines](https://umami-docs.web.cern.ch/setup/development/)
- [x] [Style guides](https://umami-docs.web.cern.ch/setup/development/good-practices/)https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/476Adding code blocks to docs directly from repo2022-04-05T14:25:06+02:00Joschka BirkAdding code blocks to docs directly from repo## Summary
This MR introduces the following changes
* Automated replacement of file-placeholders in documentation using a simple python script. This allows to get rid of code blocks in cases where we copy over whole files.
Relates to ...## Summary
This MR introduces the following changes
* Automated replacement of file-placeholders in documentation using a simple python script. This allows to get rid of code blocks in cases where we copy over whole files.
Relates to the following issues
* First step towards #111
### Example
The file placeholders of the form `§§§<filename>:<start>:<end>§§§` in the markdown file on the left will be replaced with the content of the actual files (from line `<start>` to line `<end>`), resulting in the docs page on the right. This all happens in the CI pipeline, such that the actual files of the repo are untouched.
![image](/uploads/f73b87361a151e70561f8a40a3ab0126/image.png)
## ToDos
- [x] make script more general such that we don't have to modify the ci config every time a new script needs replacements (probably the script should just go throught all documentation files)
- [x] add support for line numbers
- [x] change to syntax `§§§<filename>:start:end§§§`
## Conformity
- [x] [Changelog entry](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/changelog.md)
- [x] [Documentation](https://umami-docs.web.cern.ch)
- [x] [Development guidelines](https://umami-docs.web.cern.ch/setup/development/)
- [x] [Style guides](https://umami-docs.web.cern.ch/setup/development/good-practices/)Joschka BirkJoschka Birkhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/477linting on full project2022-03-14T14:24:39+01:00Manuel Guthlinting on full project## Summary
This MR introduces the following changes
* add linting for full project, not only for `./umami` folder
## Conformity
- [x] [Changelog entry](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/c...## Summary
This MR introduces the following changes
* add linting for full project, not only for `./umami` folder
## Conformity
- [x] [Changelog entry](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/changelog.md)
- [x] [Documentation](https://umami-docs.web.cern.ch)
- [x] [Development guidelines](https://umami-docs.web.cern.ch/setup/development/)
- [x] [Style guides](https://umami-docs.web.cern.ch/setup/development/good-practices/)https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/478Fix branch unit tests2022-03-14T15:37:53+01:00Alexander FrochFix branch unit tests## Summary
This MR introduces the following changes
* Applies the parallel processing of the unit tests also to the branches. This fixes the issue that at some unit tests the `matplotlib` style is changed and therefore all plots proces...## Summary
This MR introduces the following changes
* Applies the parallel processing of the unit tests also to the branches. This fixes the issue that at some unit tests the `matplotlib` style is changed and therefore all plots processed after that are also changed and therefore fail.
## Conformity
- [X] [Changelog entry](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/changelog.md)
- [X] [Documentation](https://umami-docs.web.cern.ch)
- [X] [Development guidelines](https://umami-docs.web.cern.ch/setup/development/)
- [X] [Style guides](https://umami-docs.web.cern.ch/setup/development/good-practices/)Alexander FrochAlexander Frochhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/480Add example plots to docs2022-03-16T15:18:47+01:00Joschka BirkAdd example plots to docs## Summary
This MR introduces the following changes
* adds a job to the CI pipeline which runs the example scripts in `examples/plotting` and hands the resulting plots as artifacts to the job which creates the docs
* updates in the doc...## Summary
This MR introduces the following changes
* adds a job to the CI pipeline which runs the example scripts in `examples/plotting` and hands the resulting plots as artifacts to the job which creates the docs
* updates in the docs, adding the above mentioned plots
* adds docs section for pt_vs_eff plot
* adds optional `add_pt` argument to dummy data
* fixing small bug in `replace_placeholders_in_md.py` which resulted in one line missing at the end
Relates to the following issues
* first step towards #159
The result is that we can add the output plot of an example script directly to the docs, which then also corresponds to the most recent version of the code.
<img src="/uploads/4ad17c915dd654bd2c15dbb7bd840225/image.png" height="800">
## Conformity
- [x] [Changelog entry](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/changelog.md)
- ~~[x] [Documentation](https://umami-docs.web.cern.ch)~~
- [x] [Development guidelines](https://umami-docs.web.cern.ch/setup/development/)
- [x] [Style guides](https://umami-docs.web.cern.ch/setup/development/good-practices/)Joschka BirkJoschka Birkhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/481CI and packaging updates2022-03-15T16:28:22+01:00Manuel GuthCI and packaging updates## Summary
This MR introduces the following changes
* updating the python packaging moving most things from `setup.py` to `setup.cfg`
* moving `classification_tools` into `helper_tools`
* updating labelling in MR and adding `Tools` lab...## Summary
This MR introduces the following changes
* updating the python packaging moving most things from `setup.py` to `setup.cfg`
* moving `classification_tools` into `helper_tools`
* updating labelling in MR and adding `Tools` label for not yet covered cases
## Conformity
- [x] [Changelog entry](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/changelog.md)
- [x] [Documentation](https://umami-docs.web.cern.ch)
- [x] [Development guidelines](https://umami-docs.web.cern.ch/setup/development/)
- [x] [Style guides](https://umami-docs.web.cern.ch/setup/development/good-practices/)https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/482slim image update2022-03-16T08:55:55+01:00Manuel Guthslim image update## Summary
This MR introduces the following changes
* updates the umami slim image
Relates to the following issues
* closes #151
## Conformity
- [x] [Changelog entry](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/uma...## Summary
This MR introduces the following changes
* updates the umami slim image
Relates to the following issues
* closes #151
## Conformity
- [x] [Changelog entry](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/changelog.md)
- [x] [Documentation](https://umami-docs.web.cern.ch)
- [x] [Development guidelines](https://umami-docs.web.cern.ch/setup/development/)
- [x] [Style guides](https://umami-docs.web.cern.ch/setup/development/good-practices/)https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/485fix unittests on branches2022-03-18T17:21:11+01:00Manuel Guthfix unittests on branches## Summary
This MR introduces the following changes
* using the latest umami image for the unit tests ran on branches
## Conformity
- [x] [Changelog entry](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/mas...## Summary
This MR introduces the following changes
* using the latest umami image for the unit tests ran on branches
## Conformity
- [x] [Changelog entry](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/changelog.md)
- [x] [Documentation](https://umami-docs.web.cern.ch)
- [x] [Development guidelines](https://umami-docs.web.cern.ch/setup/development/)
- [x] [Style guides](https://umami-docs.web.cern.ch/setup/development/good-practices/)https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/488Use plotting python API in input var plots (track variables)2022-03-30T13:52:48+02:00Joschka BirkUse plotting python API in input var plots (track variables)## Summary
This MR introduces the following changes
* Using the plotting python API for the **track** input var plots (jet variable plots will be updated in another MR)
* reduced amount of code by using the plotting python API
...## Summary
This MR introduces the following changes
* Using the plotting python API for the **track** input var plots (jet variable plots will be updated in another MR)
* reduced amount of code by using the plotting python API
* made `plot_input_vars_trks` also cover the case of multiple input datasets --> removed `plot_input_vars_trks_comparison`
Relates to the following issues
* First step towards #147
## Notes:
* The control plots for jet variables also show up in the diff, but this is only due to the change to only legend column (new default value). The histograms themselves are untouched for jet variables.
## Conformity
- [x] [Changelog entry](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/changelog.md)
- ~~[ ] [Documentation](https://umami-docs.web.cern.ch)~~
- [x] [Development guidelines](https://umami-docs.web.cern.ch/setup/development/)
- [x] [Style guides](https://umami-docs.web.cern.ch/setup/development/good-practices/)Joschka BirkJoschka Birkhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/492Resolve "mkdocs job crashes"2022-03-25T13:27:26+01:00Manuel GuthResolve "mkdocs job crashes"## Summary
This MR introduces the following changes
* updating medics material version to `8.2.7`
Related issues
* closes #161
## Conformity
- [x] [Changelog entry](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umam...## Summary
This MR introduces the following changes
* updating medics material version to `8.2.7`
Related issues
* closes #161
## Conformity
- [x] [Changelog entry](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/changelog.md)
- [x] [Documentation](https://umami-docs.web.cern.ch)
- [x] [Development guidelines](https://umami-docs.web.cern.ch/setup/development/)
- [x] [Style guides](https://umami-docs.web.cern.ch/setup/development/good-practices/)
Closes #161Manuel GuthManuel Guthhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/493Fixing failing unit test2022-03-25T14:16:57+01:00Joschka BirkFixing failing unit test## Summary
This MR introduces the following changes
* fix a unit test which failed from time to time (this was a problem since the usage of sets was introduced in !414)
* adds the changes from !492
* closes #161
## Conformity
- ~~[ ]...## Summary
This MR introduces the following changes
* fix a unit test which failed from time to time (this was a problem since the usage of sets was introduced in !414)
* adds the changes from !492
* closes #161
## Conformity
- ~~[ ] [Changelog entry](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/changelog.md)~~
- ~~[ ] [Documentation](https://umami-docs.web.cern.ch)~~
- [x] [Development guidelines](https://umami-docs.web.cern.ch/setup/development/)
- [x] [Style guides](https://umami-docs.web.cern.ch/setup/development/good-practices/)Joschka BirkJoschka Birkhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/497Update black version2022-03-29T12:25:06+02:00Joschka BirkUpdate black version## Summary
This MR introduces the following changes
* updates `black` to be compatible with newer `click` version (which is in the image `python:3.8-slim`) (see Github issue [here](https://github.com/psf/black/issues/2964))
* new `blac...## Summary
This MR introduces the following changes
* updates `black` to be compatible with newer `click` version (which is in the image `python:3.8-slim`) (see Github issue [here](https://github.com/psf/black/issues/2964))
* new `black` version we use will be 22.3.0 (see Releases [here](https://github.com/psf/black/releases))
Relates to the following issues
* closes #163
## Conformity
- ~~[ ] [Changelog entry](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/changelog.md)~~
- ~~[ ] [Documentation](https://umami-docs.web.cern.ch)~~
- ~~[ ] [Development guidelines](https://umami-docs.web.cern.ch/setup/development/)~~
- ~~[ ] [Style guides](https://umami-docs.web.cern.ch/setup/development/good-practices/)~~Joschka BirkJoschka Birkhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/501Extend flexibility in input var plotting functions2022-04-05T18:36:33+02:00Joschka BirkExtend flexibility in input var plotting functions## Summary
This MR introduces the following changes
* using `**kwargs` in the plotting functions, allowing to specify any parameter that is understood in the plotting API (in this case the `histogram_plot` class) in the config file for...## Summary
This MR introduces the following changes
* using `**kwargs` in the plotting functions, allowing to specify any parameter that is understood in the plotting API (in this case the `histogram_plot` class) in the config file for plotting
* removed the `check_kwargs_var_plots` function
* moved the "kwargs translation" (for backwards compatibility) to the high-level functions in `plot_input_vars.py` (--> kwargs are translated to new naming right away, such that the new naming is used consistently afterwards)
* making `histogram_plot` functions less verbose (`logger.info` -> `logger.debug`), to reduce stdout in large plotting scripts
* adding a script that converts the "Parameters" section of certain docstrings to markdown tables. The resulting files can then be used in the docs (using our placeholder syntax we introduced in !476)
## Example
With this change, you can now e.g. remove the ATLAS branding from the input var plots by simply adding `atlas_brand: ""` to the `plot_settings` in the config:
<img src="/uploads/668c189f5dbb218c5ed3d2d028d6f55c/nTracks_per_Jet_All.png" height=500>
-->
<img src=/uploads/24e13d84e1c13e6e42998a4d1ba206b2/image.png height=200>
-->
<img src=/uploads/f1bf9920b2a34750feca933089d62dcc/nTracks_per_Jet_All.png height=500>
Relates to the following issues
* #147
## Conformity
- [x] [Changelog entry](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/changelog.md)
- [x] [Documentation](https://umami-docs.web.cern.ch)
- [x] [Development guidelines](https://umami-docs.web.cern.ch/setup/development/)
- [x] [Style guides](https://umami-docs.web.cern.ch/setup/development/good-practices/)Joschka BirkJoschka Birkhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/502Merge resampling classes2022-05-03T17:13:00+02:00Victor Hugo Ruelas RiveraMerge resampling classes## Summary
This MR introduces the following changes
* Updates docs for importance sampling without replacement method
* Abstracts code into own module
Relates to the following issues
*
*
## Conformity
- [x] [Changelog entry](https:/...## Summary
This MR introduces the following changes
* Updates docs for importance sampling without replacement method
* Abstracts code into own module
Relates to the following issues
*
*
## Conformity
- [x] [Changelog entry](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/changelog.md)
- [x] [Documentation](https://umami-docs.web.cern.ch)
- [x] [Development guidelines](https://umami-docs.web.cern.ch/setup/development/)
- [x] [Style guides](https://umami-docs.web.cern.ch/setup/development/good-practices/)Victor Hugo Ruelas RiveraVictor Hugo Ruelas Riverahttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/503Adding prepare_docs stage to CI2022-04-06T14:26:02+02:00Joschka BirkAdding prepare_docs stage to CI## Summary
This MR introduces the following changes
* Adds a stage `prepare_docs` to the CI pipeline, since we now have multiple jobs which should be run before building the docs
* Move placeholder rendering to separate job
## Conform...## Summary
This MR introduces the following changes
* Adds a stage `prepare_docs` to the CI pipeline, since we now have multiple jobs which should be run before building the docs
* Move placeholder rendering to separate job
## Conformity
- [x] [Changelog entry](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/changelog.md)
- ~~[ ] [Documentation](https://umami-docs.web.cern.ch)~~
- ~~[ ] [Development guidelines](https://umami-docs.web.cern.ch/setup/development/)~~
- ~~[ ] [Style guides](https://umami-docs.web.cern.ch/setup/development/good-practices/)~~Joschka BirkJoschka Birkhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/511Adding URL support to placeholder syntax2022-04-11T16:45:45+02:00Joschka BirkAdding URL support to placeholder syntax## Summary
This MR introduces the following changes
* adding support for specifying a URL in the markdown placeholders
## Example
It works just like the placeholders we already use, but for a url the syntax changes from `§§§<filename>...## Summary
This MR introduces the following changes
* adding support for specifying a URL in the markdown placeholders
## Example
It works just like the placeholders we already use, but for a url the syntax changes from `§§§<filename>:<start>:<end>§§§` to `§§§url="<url>":<start>:<end>§§§`
The left side shows a placeholder with a URL specified. On the right side you see the inserted content of the file living at this URL after rendering the md file.
<img src=/uploads/a1ca8bd3ead9d2dc0e137fbeacd4b107/image.png width=1500>
## Conformity
- ~[ ] [Changelog entry](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/changelog.md)~
- [x] [Documentation](https://umami-docs.web.cern.ch)
- [x] [Development guidelines](https://umami-docs.web.cern.ch/setup/development/)
- ~[ ] [Style guides](https://umami-docs.web.cern.ch/setup/development/good-practices/)~Joschka BirkJoschka Birkhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/519Adding error message if file in placeholder does not exist2022-04-14T15:25:24+02:00Joschka BirkAdding error message if file in placeholder does not exist## Summary
This MR introduces the following changes
* adds that the md-placeholder script raises an error if the file specified in a placeholder is not found (to avoid moving files without changing the path in the placeholders)
* fixin...## Summary
This MR introduces the following changes
* adds that the md-placeholder script raises an error if the file specified in a placeholder is not found (to avoid moving files without changing the path in the placeholders)
* fixing wrong path in markdown placeholder
## Example
The invalid placeholder below (fixed in this MR) did not raise an error so far.
Now it makes the job fail (yay)
<img src=/uploads/27fd4da9fecdf8fe664ffa10f0d8c7ff/image.png width=1200>
## Conformity
- [x] [Changelog entry](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/changelog.md)
- ~[ ] [Documentation](https://umami-docs.web.cern.ch)~
- [x] [Development guidelines](https://umami-docs.web.cern.ch/setup/development/)
- [x] [Style guides](https://umami-docs.web.cern.ch/setup/development/good-practices/)Joschka BirkJoschka Birkhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/520Make docs titles clickable (for links)2022-04-14T17:31:14+02:00Joschka BirkMake docs titles clickable (for links)## Summary
This MR introduces the following changes
* adding links to the headings in the docs
* adding minimal `needs` to jobs in `prepare_docs` stage of our pipeline, to make them start right away after triggering the pipeline. I thi...## Summary
This MR introduces the following changes
* adding links to the headings in the docs
* adding minimal `needs` to jobs in `prepare_docs` stage of our pipeline, to make them start right away after triggering the pipeline. I think if not specified, the jobs wait until all previous stages are completed
## Example
If you hover with your mouse over the area right next to a heading, a hashtag will appear. If you click it, the url in your browser will change to a link right to that heading.
![image](/uploads/fe2786836ffd46c887e8fd9cdd730fa2/image.png)Joschka BirkJoschka Birkhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/522renaming variables to match snake_case style2022-04-19T17:27:42+02:00Manuel Guthrenaming variables to match snake_case style## Summary
This MR introduces the following changes
* Renaming a bunch of functions and variables to match snake_case style
Relates to the following issues
* first step towards https://gitlab.cern.ch/atlas-flavor-tagging-tools/algori...## Summary
This MR introduces the following changes
* Renaming a bunch of functions and variables to match snake_case style
Relates to the following issues
* first step towards https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/issues/167
## Conformity
- [x] [Changelog entry](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/changelog.md)
- [x] [Documentation](https://umami-docs.web.cern.ch)
- [x] [Development guidelines](https://umami-docs.web.cern.ch/setup/development/)
- [x] [Style guides](https://umami-docs.web.cern.ch/setup/development/good-practices/)https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/523Draft: ci improvements2022-05-10T10:50:08+02:00Manuel GuthDraft: ci improvements## Summary
This MR introduces the following changes
* simplifying some CI jobs## Summary
This MR introduces the following changes
* simplifying some CI jobs