Umami merge requestshttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests2021-10-29T09:52:54+02:00https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/217adding more labels in MR check2021-10-29T09:52:54+02:00Manuel Guthadding more labels in MR checkhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/218docu ci skip2021-10-29T11:21:45+02:00Manuel Guthdocu ci skipmost of the CI is being skipped if only the documentation is changedmost of the CI is being skipped if only the documentation is changedhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/229Update tests and general bug fixes2021-11-11T10:09:02+01:00Alexander FrochUpdate tests and general bug fixesThis MR adds the following things:
- Adding `-s` option to integration tests, so the stdout of the tests is shown while they are running.
- Removing `Configuration` of the `evaluation_tools` -> Never used.
- Removing not-implemented fun...This MR adds the following things:
- Adding `-s` option to integration tests, so the stdout of the tests is shown while they are running.
- Removing `Configuration` of the `evaluation_tools` -> Never used.
- Removing not-implemented function from `evaluation_tools` (is covered by Maxence MR !190).
- Created new integration tests for the pdf sampling method. It runs in the `test_preprocessing` stage and is not further used till now -> Maybe add more integration tests for the taggers with these preprocessed samples later.
- Fixed long runtime of the `test_train_dl1r` -> Shapley still had default values which were way to big for testing purposes.
- Added unit tests for `tf_tools` and a small one for `evaluation_tools`.
- Moving the transformation of the `np.arrays` to `tf.tensors` to the `NN_tools` -> The `tf.tensors` fixed the memory issue for DIPS and Umami.
- Fixing the number of jets used in training (the `nJets_train` from the config were overwritten by the full size of the training file. One of the reasons the integration tests needed so much. The integration tests for training the taggers should be way faster now (from 17 Minutes to 3 Minutes).
- Overall improvement of the time needed for the whole pipeline: 30 minutes -> 10 minutes.Alexander FrochAlexander Frochhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/219Rearranging pipelines in umami2021-11-19T15:08:07+01:00Manuel GuthRearranging pipelines in umamiThis MR cleans up a bit the CI, fixes some small issues when the bot is approving MR which only concern documentation
closes https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/issues/80
small docs adaptationsThis MR cleans up a bit the CI, fixes some small issues when the bot is approving MR which only concern documentation
closes https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/issues/80
small docs adaptationshttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/254Adding conditional attention2021-11-24T09:52:39+01:00Alexander FrochAdding conditional attentionThis MR adds the following:
- Adding new train script for DIPS Conditional Attention + example train config.
- Adding support for new DIPS Conditional Attention model in the validation/evaluation chain.
- Adding integration test for tra...This MR adds the following:
- Adding new train script for DIPS Conditional Attention + example train config.
- Adding support for new DIPS Conditional Attention model in the validation/evaluation chain.
- Adding integration test for train/validation/evaluation of DIPS Conditional Attention.
- Fixing some issues in the `tf_tools/models` with the masking (also adapted the unit test).
- Adding new generator for DIPS Conditional Attention.
- Adding some flexibility to the loading of the `loading_validation_data` functions.
- Adding compatibility of the `evaluate_model.py` for DIPS and DIPS Conditional Attention.
- Adding DIPS Conditional Attention to possible models for `plotting_epoch_performance.py`
- Adding DIPS Conditional Attention train script to the `setup.py`.
- Make the calculation of the Saliency maps steerable in the train config of the DIPS models.Alexander FrochAlexander Frochhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/235Convert h5 to tf2021-11-25T10:57:54+01:00Tomke SchroerConvert h5 to tfScript for Convention of h5 file to tf Record and reload tf Record fileScript for Convention of h5 file to tf Record and reload tf Record filehttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/277Fixing labels set by the CI2021-11-29T16:48:30+01:00Alexander FrochFixing labels set by the CIThis MR fixes a bug with the automatic labels. Since we moved the model files, we need to also change it in the automated label setting of the CI. This is done nowThis MR fixes a bug with the automatic labels. Since we moved the model files, we need to also change it in the automated label setting of the CI. This is done nowAlexander FrochAlexander Frochhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/192Add weighting class2021-12-01T11:32:57+01:00Frederic RennerAdd weighting classThis adds the Weighting class to calculate sample_weights as a preprocessing method. I also did some general refactoring from the ```Undersampling``` class into the ```Resampling``` base class to avoid code duplication. I would like to h...This adds the Weighting class to calculate sample_weights as a preprocessing method. I also did some general refactoring from the ```Undersampling``` class into the ```Resampling``` base class to avoid code duplication. I would like to hear your opinion on it.
Closes #64https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/290running black on umami2021-12-06T14:12:51+01:00Frederic Rennerrunning black on umamihttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/291adding doc string linting2021-12-07T14:04:56+01:00Manuel Guthadding doc string lintingfirst step towards #97
So far the doc string check test is only performed if there was at all any doc string in the function and the CI stage is allowed to fail for now
This also means the CI will from now on display that it ended wit...first step towards #97
So far the doc string check test is only performed if there was at all any doc string in the function and the CI stage is allowed to fail for now
This also means the CI will from now on display that it ended with a warning until all the docstring errors are fixedhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/303Updating to new tf version 2.7.02021-12-09T15:52:05+01:00Manuel GuthUpdating to new tf version 2.7.0this version now has finally python 3.8this version now has finally python 3.8https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/309moving functions into separate histogram tools2021-12-15T13:42:00+01:00Manuel Guthmoving functions into separate histogram toolsadding separate histogram tools in new module `umami.helper_tools`
extending testing of these functionsadding separate histogram tools in new module `umami.helper_tools`
extending testing of these functionshttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/310replacing .format with fstrings2021-12-15T15:24:11+01:00Manuel Guthreplacing .format with fstringshttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/314Fixing unit tests to run at once2021-12-15T17:45:18+01:00Manuel GuthFixing unit tests to run at oncecloses #94closes #94https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/316Updating doc strings2021-12-16T12:38:13+01:00Manuel GuthUpdating doc stringsfirst step towards #97first step towards #97https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/321pylint init2021-12-17T10:51:58+01:00Manuel Guthpylint initadding pylint in the CI without `allow_failure`
a first pick of disabled errors was made, still lots of warnings and errors are raised
fixing black line length
step towards #105adding pylint in the CI without `allow_failure`
a first pick of disabled errors was made, still lots of warnings and errors are raised
fixing black line length
step towards #105https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/320Adding tfrecords for DIPS2021-12-17T11:39:15+01:00Alexander FrochAdding tfrecords for DIPSAlexander FrochAlexander Frochhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/322Kaniko Docker Builder2021-12-17T14:16:36+01:00Jackson BarrKaniko Docker BuilderChanging docker building to use the Kaniko builder, addressing #87.
closes #87Changing docker building to use the Kaniko builder, addressing #87.
closes #87https://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/324Splitting up requirements2021-12-21T07:09:01+01:00Manuel GuthSplitting up requirementsSplitting requirements and adding a new base image `umamibase-plus` with additional packages not necessarily needed for umami itselfSplitting requirements and adding a new base image `umamibase-plus` with additional packages not necessarily needed for umami itselfhttps://gitlab.cern.ch/atlas-flavor-tagging-tools/algorithms/umami/-/merge_requests/326quick fix such that pipeline master runs2021-12-20T11:18:12+01:00Manuel Guthquick fix such that pipeline master runs