Protect CI runners from disk space over consumption

This job was regularly killing kubernetes CI runners because cta-taped process kept generating large core dump files until the runner disk was full.

Some protections need to be put in place to prevent this disk explosion as after this the runner is not usable anymore with the current setup:

# df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        4.0M     0  4.0M   0% /dev
tmpfs           7.1G   88K  7.1G   1% /dev/shm
tmpfs           2.9G  219M  2.6G   8% /run
/dev/vda1        80G   79G  633M 100% /
/dev/vda15      544M  7.1M  537M   2% /boot/efi
tmpfs           1.5G     0  1.5G   0% /run/user/0
shm              63M   84K   63M   1% /var/lib/containers/storage/overlay-containers/fb8037469e478f6ed46f8f18cfbafffa128e2c213d241f66d1f47a98e5d1445a/userdata/shm
overlay          80G   79G  633M 100% /var/lib/containers/storage/overlay/0006bb421c887e8ecf1f374d6b1de61fea1d1dfcf7a1c9ca718a21e9ef3042d0/merged

In order to protect the runners we should make sure that /var/lib/containers is not growing too large and/or put quotas on kubernetes managed volumes (problem is that this could cause trouble for stress tests).

Partitionning a runner/dev machines in openstack

Create openstack large VM

• <VM NAME>
• alma9 x86_64
• large (preferably xlarge)
• Configuration: Browse -> userdata4multi.txt
• Metadata:
  • lanDB InternetConnectivity -> true
  • lanDB IPv6Ready -> true
  • lanDB Alias -> whatEverYouNeed