Skip to content

add tensorboard and comet logging

Minh Tuan Pham requested to merge tp_add_tensorboard into dev

Add tensorboard and comet logging as alternative to CSV and WandB. WandB is very finicky sometimes and is the cause of failure of many slurm job, especially those that undergo automatic resumption. CSV is simple and safe but not convenient for visualization. New features in this MR include:

  • addition of a get_logger method in core_utils.py that returns the appropriate logger based on the "logger" configuration in the yaml file, default to CSVLogger. The "log_wandb" config no longer has any effect since we have more than 2 possible loggers.
  • Some small changes to train_stage.py to handle the WandB hyperparameter sweep.
  • Set rid of complicated slurm requirements on setting the default root directory and the wandb run name

Merge request reports