Skip to content

Overhaul dataloading

Samuel Van Stroud requested to merge svanstro/salt:svanstro/v2-data into main

This MR overhauls dataloading in salt as discussed in #18 (closed).

We now use a single standardised training and testing file format which is the same format as the TDD output.

The "umami test file" (e.g. *-hybrid-resampled_scaled_shuffled.h5) is no longer supported, but you can still train from the umami resampled file (e.g. *-hybrid-resampled.h5). Input normalisation happens on the fly as the data is loaded.

Also some improvements for #13.

closes #18 (closed)

Todo

  • Upload samples to eos
Edited by Samuel Van Stroud

Merge request reports