Configure CI with as many runners as GPUs on servers
It is possible to configure gitlab-runner with several runners on each server (https://docs.gitlab.com/runner/). This would allow running on multiple GPUs in a server at once.
Some tasks in order to enable this:
- Make sure each runner runs a separate GPU. Use variables (perhaps runner names) to choose the GPU slot.
- Balance properly the compilation / execution of GPU jobs. It may not be ideal to ie. issue three compilations and a run at the same, which may compromise the throughput measurement.