increase docker.io proxy cache quota to 4TB

DockerHub is implementing new limits on the image pulls from 1st March, 2025, restricting unauthenticated users to 10 pulls per hour and authenticated users with free accounts to 40 pulls per hour. Organizations require a paid subscription to exceed these limits.
We need to audit our pull-through cache usage to ensure we won't exceed these rate limits.

added A-registry p2 labels

changed iteration to Kubernetes Sprints Feb 19, 2025 - Mar 4, 2025

@akothiwa i'm not sure we can ensure not to exceed the limits. What we can try is to mitigate the issue... some points from the discussion in mattermost:

It does have an impact in the pull through cache for docker.io. There are a couple options to consider to mitigate, we're following it up:

* Increase the cache storage (this will slow the cache rotation and increase the hit rate)
* Add authentication to docker.io, we already do it for other registry caches (this will not be enough we have peaks of more than 100 misses in 1h)
* Alternatively increase the number of nodes serving this purpose - which helps since the usage limits are per IP when unauthenticated, but is no guarantee

Plus some additional options we'll consider. In any case we're tracking this

The additional options would be to pay for a user/account per year, so we drop these limits for the pull through cache.

This could also be an issue for the image replication workflows that we have if we need to add more than 10 images in a MR

assigned to @rbritoda

At the moment docker.io usage is at 95% (2032053186029 / 2147483648000). Let's 200GB (10% increase)? It's stuck at 95% for some time which looks suspicious.

cc @rbritoda

spyros and i are currently being rate limited in registry-next after restarting / rescheduling most of the pods on to fresh nodes - this suggests that we are likely going to run into issues next month if we are struggling to pull the images for our own infra

i realise we are already aware of this, but thought it was worth documenting in the ticket

also it is the 1st of april not march that this change comes in effect from docker

Things to be done:

• Increase the cache quota to 4TB (it was 2TB), needs the bucket quota to be checked that that works. Reshaping this ticket to become this task
• Agree on a paid user/account with dockerhub (being done with the security team)

The "registry" bucket has a quota of 15TB of which around 9TB is currently being used. We can increase the docker cache size to 4TB.

Just to highlight, this bucket is currently in "Cloud Infra Services" and will be moved to "Kubernetes" in the upcoming intervention. So, we can have similar quota available in the new bucket.

This is a very good point. What's the S3 quota in 'Kubernetes'?

Seems we have 20TB there already?

changed title from Check our pull-through cache usage to increase docker.io proxy cache quota to 4TB

assigned to @akothiwa and unassigned @rbritoda

changed iteration to Kubernetes Sprints Mar 5, 2025 - Mar 18, 2025

mentioned in issue #124 (closed)

Increased the quota to 4TB.

closed