Discussion around precision

!608 (merged) switched from float16 to float32 for the writing of the final train samples. It's clear now from the algs meeting that this doesn't affect training performance, so we should switch back to float16 to save space and improve dataloading times during training.

In a separate problem, dumping from the TDD at half precision results in reduced tagger performance. This suggests something is going wrong in the umami preprocessing (probably in the scaling). To try and mitigate, we should make sure all operations are performed at full precision, but only ever write out at half precision (since this will approx 2x i/o speeds).

@mguth @alfroch @dguest