Fix grafana monitoring of CI throughput (!642) · Merge requests · LHCb / Allen · GitLab

Snippets Groups Projects

Merged Dorothea Vom Bruch requested to merge dovombru_fix_telegraf into master 3 years ago

The throughput measured in the CI used to be sent to telegraf for monitoring in a grafana plot. This has been broken for a long time (see for example #235 (closed)). With this MR, the script to send data to telegraf is refactored, taking as input the CSV file of throughput and device name.

The grafana plot can be found here.

to do:

Ensure that all devices can be selected in grafana plot (currently only the A40 and MI100 show up)

Edited 3 years ago by Rainer Schwemmer

Activity

Dorothea Vom Bruch added bug fix label 3 years ago

added bug fix label
Dorothea Vom Bruch requested review from @rschwemm and @roneil 3 years ago

requested review from @rschwemm and @roneil
Dorothea Vom Bruch added only GitLab CI label 3 years ago

added only GitLab CI label
D

Dorothea Vom Bruch @dovombru · 3 years ago

Author Maintainer

@rschwemm this posts values to AllenCIPerformance_v3, named CHEP AllenPerformance. Is it okay to re-use this grafana plot? And if yes, can I remove the CHEP in the title?
Dorothea Vom Bruch added 1 commit 3 years ago
added 1 commit

7dc6e5e5 - Fixed formatting

Compare with previous version
Dorothea Vom Bruch changed title from Fix grafana monitoring of throughput over time to Fix grafana monitoring of CI throughput 3 years ago

changed title from Fix grafana monitoring of throughput over time to Fix grafana monitoring of CI throughput
Dorothea Vom Bruch marked this merge request as draft 3 years ago

marked this merge request as draft
D

Dorothea Vom Bruch @dovombru · 3 years ago

Author Maintainer

Resolved 3 years ago by Dorothea Vom Bruch

Sending from the CI failed with the following message: ConnectionRefusedError: [Errno 111] Connection refused (see here) This did work when I ran the python script locally on one of the CI GPU servers. @rschwemm do you have an idea what could be the problem here?

Last reply by Dorothea Vom Bruch 3 years ago
Rainer Schwemmer added 1 commit 3 years ago
added 1 commit

fc66b4cf - Updated telegraf target ip address to go through central server, which should...

Compare with previous version
Dorothea Vom Bruch added 2 commits 3 years ago
added 2 commits

d8102c85 - add quotes around device name

61cb62f4 - Merge branch 'dovombru_fix_telegraf' of ssh://gitlab.cern.ch:7999/lhcb/Allen...

Compare with previous version
Dorothea Vom Bruch resolved all threads 3 years ago

resolved all threads
Dorothea Vom Bruch added 1 commit 3 years ago
added 1 commit

a01001e5 - try \_ as whitespace in telegraf string

Compare with previous version
Dorothea Vom Bruch added 1 commit 3 years ago
added 1 commit

3b32cc70 - Fixed formatting

Compare with previous version
Rainer Schwemmer added 1 commit 3 years ago
added 1 commit

aea8d4f6 - Fixed bug which prevented devices with spaces in their names to show up in telegraf (hopefully)

Compare with previous version
Rainer Schwemmer added 1 commit 3 years ago
added 1 commit

fe8d8c2e - The inevitable bug - part 1

Compare with previous version
Dorothea Vom Bruch marked the checklist item Ensure that all devices can be selected in grafana plot (currently only the A40 and MI100 show up) as completed 3 years ago

marked the checklist item Ensure that all devices can be selected in grafana plot (currently only the A40 and MI100 show up) as completed
Dorothea Vom Bruch marked this merge request as ready 3 years ago

marked this merge request as ready
D

Dorothea Vom Bruch @dovombru · 3 years ago

Author Maintainer

Thanks to @rschwemm for fixing the device names, this is ready for review now.
Dorothea Vom Bruch assigned to @acorread 3 years ago

assigned to @acorread
Dorothea Vom Bruch removed review request for @rschwemm and @roneil 3 years ago

removed review request for @rschwemm and @roneil

Please register or sign in to reply