Make BandwidthTestHandler (a bit) more robust

Goes with lhcb-datapkg/PRConfig!349 (merged) and lhcb/Moore!2651 (merged).

This MR makes BandwidthTestHandler a bit more robust by:

• Not hard-coding/assuming any of the file names output by the bandwidth test. We can just upload all the files with a certain set of file extensions (we already were pretty-much uploading them all anyway), except for index.html, which of course must be present.
• Use a single function to build the comparison tables per stream and per line. Previously there was a hard-coded rates-all-lines.csv that had to be there; now all we do in assert that the per-line table has to have a Line column, and the others a Stream column. I think that's a fairly good assumption, rather that the hard-coded paths.
• Move a few blocks of duplicated code into helper functions. Move the mattermost feedback into its own function to try to reduce the size of the collectResultsExt function.

assigned to @rjhunter

changed title from Draft: Make BandwidthTestHandler more robust to Draft: Make BandwidthTestHandler (a bit) more robust

mentioned in merge request lhcb-datapkg/PRConfig!349 (merged)

added 1 commit

• 471cfe28 - Move comparison block around a lil to hopefully make slightly more comprehensible

Compare with previous version

mentioned in merge request lhcb/Moore!2651 (merged)

marked this merge request as ready

added 1 commit

• 7b05fc76 - Fix a couple of small errors identified in testing

Compare with previous version

added 2 commits

• 1737cf55 - Fix two small handler bugs
• 23ec9218 - Reduce table size for all lines table

Compare with previous version

I was able to run a local BW test from the Moore and PRConfig branches and test the handler from that output. After the last few commits, the handler should work fine (the only things I can't test are wherever it actually interacts with the dashboard, but I was able to replace whatever it gets from there with some dummy output for testing).

I was seeing that some of the pages are missing in the last CI tests here because I had changed the names. From the log of the test I can see now what it would upload (if it was allowed), and scping those files that it would upload to a different directory, I can confirm that all the html pages that the BW test expects are in that different directory, so all the pages should be there after this MR.

Ready to merge with/after the Moore and PRConfig dependencies.

added 1 commit

• 79bcd16c - Put logging in a more useful place

Compare with previous version

mentioned in commit 37446554

merged

mentioned in merge request !261 (merged)

mentioned in commit 0ab27d2e

mentioned in commit 06b3873e