Umami merge requestshttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests2023-10-30T12:07:44+01:00https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/742Upp version update2023-10-30T12:07:44+01:00Ivan OleksiyukUpp version update## Summary
This MR introduces the following changes
* Updates upp version to 0.0.3 that is required for upp integration
## Conformity
- [ ] [Changelog entry](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/m...## Summary
This MR introduces the following changes
* Updates upp version to 0.0.3 that is required for upp integration
## Conformity
- [ ] [Changelog entry](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/changelog.md)
- [ ] [Documentation](https://umami-docs.web.cern.ch)
- [ ] [Development guidelines](https://umami-docs.web.cern.ch/setup/development/)
- [ ] [Style guides](https://umami-docs.web.cern.ch/setup/development/good-practices/)https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/715Draft: Merge current Preprocessing Rewrite2023-03-28T12:15:49+02:00Alexander FrochDraft: Merge current Preprocessing Rewrite## Summary
This MR introduces the following changes
* Merge the current status of the preprocessing rewrite into master
Relates to the following issues
* Closes #232 #235 #237
* Touches #60 #152 #233 #234
## Conformity
- [ ] [Chang...## Summary
This MR introduces the following changes
* Merge the current status of the preprocessing rewrite into master
Relates to the following issues
* Closes #232 #235 #237
* Touches #60 #152 #233 #234
## Conformity
- [ ] [Changelog entry](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/changelog.md)
- [X] [Documentation](https://umami-docs.web.cern.ch)
- [X] [Development guidelines](https://umami-docs.web.cern.ch/setup/development/)
- [X] [Style guides](https://umami-docs.web.cern.ch/setup/development/good-practices/)Alexander FrochAlexander Frochhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/704Draft: Sweep Master into Preprocessing-Rewrite2023-02-27T11:07:48+01:00Alexander FrochDraft: Sweep Master into Preprocessing-Rewrite## Summary
This MR introduces the following changes
* Sweeping master into preprocessing-rewrite## Summary
This MR introduces the following changes
* Sweeping master into preprocessing-rewriteAlexander FrochAlexander Frochhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/362Merging master in protected branch2022-01-21T14:44:14+01:00Alexander FrochMerging master in protected branchhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/350Merging Master2022-01-19T11:30:39+01:00Alexander FrochMerging MasterAlexander FrochAlexander Frochhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/282Reducing chunksize in train file writing2021-11-30T15:08:54+01:00Alexander FrochReducing chunksize in train file writingThis MR reduces the `chunk_size` in `Write_Train_file` due to RAM issues. With the new number, it works fine.This MR reduces the `chunk_size` in `Write_Train_file` due to RAM issues. With the new number, it works fine.Alexander FrochAlexander Frochhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/169WIP: [preprocessing] move scaling operations to class2021-09-15T16:47:43+02:00Philipp GadowWIP: [preprocessing] move scaling operations to classWIPWIPhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/154WIP: Framework implementations for the PDF Sampling2021-09-13T16:02:07+02:00Alexander FrochWIP: Framework implementations for the PDF SamplingThis MR adds needed framework implementations and configs for the new PDF Sampling method. Also documentation and tests are added.This MR adds needed framework implementations and configs for the new PDF Sampling method. Also documentation and tests are added.Preprocessing rewriteAlexander FrochAlexander Frochhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/144initiating pdf sampling2021-08-06T14:48:12+02:00Manuel Guthinitiating pdf samplingAn initial commit to start the anticipated structure for the new pdf sampling
related to &1An initial commit to start the anticipated structure for the new pdf sampling
related to &1Preprocessing rewritehttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/746Update paper branch2023-11-22T10:04:08+01:00Philipp GadowUpdate paper branch## Summary
This MR introduces the following changes
* adds a "how to contribute" section to the documentation
* adds a tutorial based on the JetClass dataset
Relates to the following issues
* https://github.com/umami-hep/umami/issues...## Summary
This MR introduces the following changes
* adds a "how to contribute" section to the documentation
* adds a tutorial based on the JetClass dataset
Relates to the following issues
* https://github.com/umami-hep/umami/issues/4
* https://github.com/umami-hep/umami/issues/2
* https://github.com/umami-hep/umami/issues/1
## Conformity
- [ ] [Changelog entry](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/changelog.md)
- [x] [Documentation](https://umami-docs.web.cern.ch)
- [x] [Development guidelines](https://umami-docs.web.cern.ch/setup/development/)
- [x] [Style guides](https://umami-docs.web.cern.ch/setup/development/good-practices/)Philipp GadowPhilipp Gadowhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/741Black mismatch2023-10-16T18:55:41+02:00Ivan OleksiyukBlack mismatch## Summary
This MR introduces the following changes
* Changes to black==23.3 in CI as it is stated in dev-requirements
## Conformity
- [ ] [Changelog entry](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/ma...## Summary
This MR introduces the following changes
* Changes to black==23.3 in CI as it is stated in dev-requirements
## Conformity
- [ ] [Changelog entry](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/changelog.md)
- [x] [Documentation](https://umami-docs.web.cern.ch)
- [x] [Development guidelines](https://umami-docs.web.cern.ch/setup/development/)
- [x] [Style guides](https://umami-docs.web.cern.ch/setup/development/good-practices/)Ivan OleksiyukIvan Oleksiyukhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/740Integration of upp pre-processing package into umami framework2023-11-29T11:36:38+01:00Ivan OleksiyukIntegration of upp pre-processing package into umami framework## Summary
This MR introduces the following changes
* possibility to comfortably use upp preprocessing in umami framework,
basically one can now do all the same pre-processing steps but with using upp config file (with some tweaks)!
*...## Summary
This MR introduces the following changes
* possibility to comfortably use upp preprocessing in umami framework,
basically one can now do all the same pre-processing steps but with using upp config file (with some tweaks)!
* examples on how to utilise upp config file in preprocessing
Relates to the following issues
* https://github.com/umami-hep/umami-preprocessing/pull/30
* https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/issues/240
* https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/issues/233
* https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/issues/234
* https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/issues/231
## Conformity
- [x] [Changelog entry](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/changelog.md)
- [x] [Documentation](https://umami-docs.web.cern.ch)
- [x] [Development guidelines](https://umami-docs.web.cern.ch/setup/development/)
- [x] [Style guides](https://umami-docs.web.cern.ch/setup/development/good-practices/)Rewrite PreprocessingIvan OleksiyukIvan Oleksiyukhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/739Change truth label names2023-10-30T15:37:02+01:00Philipp GadowChange truth label names## Summary
This MR introduces the following changes
* Change truth label names to account for changes in DAODs (see salt MR here: https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/salt/-/merge_requests/142/diffs)
## Conform...## Summary
This MR introduces the following changes
* Change truth label names to account for changes in DAODs (see salt MR here: https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/salt/-/merge_requests/142/diffs)
## Conformity
- [x] [Changelog entry](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/changelog.md)
- [x] [Documentation](https://umami-docs.web.cern.ch)
- [x] [Development guidelines](https://umami-docs.web.cern.ch/setup/development/)
- [x] [Style guides](https://umami-docs.web.cern.ch/setup/development/good-practices/)Philipp GadowPhilipp Gadowhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/732Update PUMA and atlas-ftag-tools2023-06-14T15:46:18+02:00Alexander FrochUpdate PUMA and atlas-ftag-tools## Summary
This MR introduces the following changes
* Updating Puma (to v0.2.6)
* Updating atlas-ftag-tools (to v0.1.4)
* Updating black (to 23.3.0)
* Updating the global_config to include bc-jets correctly
Tagging @nekumari
## Con...## Summary
This MR introduces the following changes
* Updating Puma (to v0.2.6)
* Updating atlas-ftag-tools (to v0.1.4)
* Updating black (to 23.3.0)
* Updating the global_config to include bc-jets correctly
Tagging @nekumari
## Conformity
- [X] [Changelog entry](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/changelog.md)
- [x] [Documentation](https://umami-docs.web.cern.ch)
- [x] [Development guidelines](https://umami-docs.web.cern.ch/setup/development/)
- [x] [Style guides](https://umami-docs.web.cern.ch/setup/development/good-practices/)Alexander FrochAlexander Frochhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/731Support PUMA variables in general preprocessing config for preprocessing plots2023-06-01T11:21:35+02:00Alexander FrochSupport PUMA variables in general preprocessing config for preprocessing plots## Summary
This MR introduces the following changes
* Enables all PUMA variables in the preprocessing config to configure the preprocessing plots.
* Fixing some typos
## Conformity
- [X] [Changelog entry](https://gitlab.cern.ch/atlas-...## Summary
This MR introduces the following changes
* Enables all PUMA variables in the preprocessing config to configure the preprocessing plots.
* Fixing some typos
## Conformity
- [X] [Changelog entry](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/changelog.md)
- [X] [Documentation](https://umami-docs.web.cern.ch)
- [X] [Development guidelines](https://umami-docs.web.cern.ch/setup/development/)
- [X] [Style guides](https://umami-docs.web.cern.ch/setup/development/good-practices/)Alexander FrochAlexander Frochhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/730Add option to use tdd file format to train in umami2023-07-19T11:41:50+02:00Ivan OleksiyukAdd option to use tdd file format to train in umami## Summary
This MR introduces the following changes
* Adds a new generator base class that is able to read file from tdd format rescale variables and structure them such that they are suited for training taggers in umami
* Add all the ...## Summary
This MR introduces the following changes
* Adds a new generator base class that is able to read file from tdd format rescale variables and structure them such that they are suited for training taggers in umami
* Add all the child classes for each model
* For all models add a switch to change to this new generator
* Fixed an issue with the "writer" from the old pre-processing which was saving "weight" of a jet e.g. from sampler as one of the NN's inputs (Therefore if anyone would use the weights for training it will give one discriminative feature to the network which actually likely shouldn`t be there)
* Some simplifications with the data generator in models overall (models now get necessary parameters like data dimensions from data generator itself)
* Attempted optimising speed of the new tddgenerator but it is still lacking somewhat compared to the original generator
Solves following issues:
* #242
## Conformity
- [x] [Changelog entry](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/changelog.md)
- [x] [Documentation](https://umami-docs.web.cern.ch)
- [x] [Development guidelines](https://umami-docs.web.cern.ch/setup/development/)
- [x] [Style guides](https://umami-docs.web.cern.ch/setup/development/good-practices/)Rewrite PreprocessingIvan OleksiyukIvan Oleksiyukhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/727lowercase flavour categories for boosted tagging2023-04-26T13:53:04+02:00Osama Karkouto.karkout@nikhef.nllowercase flavour categories for boosted tagging## Summary
This MR introduces the following changes:
* (Hbb, Hcc, QCD) changed to (hbb, hcc, qcd)
## Conformity
- [X] [Changelog entry](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/changelog.md)
- [...## Summary
This MR introduces the following changes:
* (Hbb, Hcc, QCD) changed to (hbb, hcc, qcd)
## Conformity
- [X] [Changelog entry](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/changelog.md)
- [X] [Documentation](https://umami-docs.web.cern.ch)
- [X] [Development guidelines](https://umami-docs.web.cern.ch/setup/development/)
- [X] [Style guides](https://umami-docs.web.cern.ch/setup/development/good-practices/)Osama Karkouto.karkout@nikhef.nlOsama Karkouto.karkout@nikhef.nlhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/724Removing unused preprocessing UnderSamplingProp + Fixing coverage issue2023-04-18T14:00:47+02:00Alexander FrochRemoving unused preprocessing UnderSamplingProp + Fixing coverage issue## Summary
This MR introduces the following changes
* Removes the unused `UnderSamplingProp` method from resampling methods.
* Fixing a bug in the artefacts so that some coverage files were not correctly saved.
## Conformity
- [x] [Ch...## Summary
This MR introduces the following changes
* Removes the unused `UnderSamplingProp` method from resampling methods.
* Fixing a bug in the artefacts so that some coverage files were not correctly saved.
## Conformity
- [x] [Changelog entry](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/changelog.md)
- [X] [Documentation](https://umami-docs.web.cern.ch)
- [X] [Development guidelines](https://umami-docs.web.cern.ch/setup/development/)
- [X] [Style guides](https://umami-docs.web.cern.ch/setup/development/good-practices/)Alexander FrochAlexander Frochhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/722Adding plot_type to Preprocessing configs2023-04-17T13:17:25+02:00Alexander FrochAdding plot_type to Preprocessing configs## Summary
This MR introduces the following changes
* Adding the option `plot_type` to the preprocessing config which sets the plot type of all plots which are created during preprocessing. This is optional and backward compatible. The...## Summary
This MR introduces the following changes
* Adding the option `plot_type` to the preprocessing config which sets the plot type of all plots which are created during preprocessing. This is optional and backward compatible. The default (if `plot_type` is not given in the config) is pdf.
## Conformity
- [x] [Changelog entry](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/changelog.md)
- [X] [Documentation](https://umami-docs.web.cern.ch)
- [X] [Development guidelines](https://umami-docs.web.cern.ch/setup/development/)
- [X] [Style guides](https://umami-docs.web.cern.ch/setup/development/good-practices/)Alexander FrochAlexander Frochhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/721Bug fixing and restructure/simplifying resampling code2023-04-17T17:31:04+02:00Ivan OleksiyukBug fixing and restructure/simplifying resampling code## Summary
current resampling code still have code duplicates where some of the older code is still bug prone. Such code variations should be merged to fix the bugs persistent in the old code whereas they are fixed in the new.
E.g. when...## Summary
current resampling code still have code duplicates where some of the older code is still bug prone. Such code variations should be merged to fix the bugs persistent in the old code whereas they are fixed in the new.
E.g. when chunk_size<len(indices) resampling_generator cuts off the last non-full chunk whereas the sampling_generator append the rest to a previous chunk (which is also a bad practice, as intuitively chunk_size=max_chunk_size so that one is sure not to run out of memory although factor that is less then 2 plays role really seldom)
This MR introduces the following changes
* fix a BUG in resampling_base:Resampling:resampling_generator that throws away the last not full bath (chunk) ...
* ... by merging it with resampling_base:sampling_generator that does not have such a bug
* Bug in resampling_base:write_file made writing the last jet batch doubled into the output file which might have lead to a lot of problems, now it is fixed
## Conformity
- [x] [Changelog entry](https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/blob/master/changelog.md)
- [x] [Documentation](https://umami-docs.web.cern.ch)
- [x] [Development guidelines](https://umami-docs.web.cern.ch/setup/development/)
- [x] [Style guides](https://umami-docs.web.cern.ch/setup/development/good-practices/)Ivan OleksiyukIvan Oleksiyuk