Protect CI runners from disk space over consumption
This job was regularly killing kubernetes CI runners because cta-taped
process kept generating large core dump files until the runner disk was full.
Some protections need to be put in place to prevent this disk explosion
as after this the runner is not usable anymore with the current setup:
# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 4.0M 0 4.0M 0% /dev
tmpfs 7.1G 88K 7.1G 1% /dev/shm
tmpfs 2.9G 219M 2.6G 8% /run
/dev/vda1 80G 79G 633M 100% /
/dev/vda15 544M 7.1M 537M 2% /boot/efi
tmpfs 1.5G 0 1.5G 0% /run/user/0
shm 63M 84K 63M 1% /var/lib/containers/storage/overlay-containers/fb8037469e478f6ed46f8f18cfbafffa128e2c213d241f66d1f47a98e5d1445a/userdata/shm
overlay 80G 79G 633M 100% /var/lib/containers/storage/overlay/0006bb421c887e8ecf1f374d6b1de61fea1d1dfcf7a1c9ca718a21e9ef3042d0/merged
In order to protect the runners we should make sure that /var/lib/containers
is not growing too large and/or put quotas on kubernetes
managed volumes (problem is that this could cause trouble for stress tests).
Partitionning a runner/dev machines in openstack
Create openstack large VM
<VM NAME>
- alma9 x86_64
- large (preferably xlarge)
- Configuration: Browse -> userdata4multi.txt
- Metadata:
- lanDB InternetConnectivity -> true
- lanDB IPv6Ready -> true
- lanDB Alias -> whatEverYouNeed