LE registry.cern.ch certificate not renewing automatically

We got a notification email from LE mentioning it will expire in 19 days.

Figure this out asap! @digaponc @rvasek we can apply the required change next week but if you could already have a quick look of why it's failing (the cert-manager logs should be useful).

added A-registry p1 labels

changed iteration to Kubernetes Sprints Nov 1, 2023 - Nov 14, 2023

assigned to @digaponc and @rvasek

"message"="Renewing certificate as renewal was scheduled at 2023-10-29 04:38:29 +0000 UTC"

Operation cannot be fulfilled on certificaterequests.cert-manager.io \"harbor-harbor-ingress-vswn8\": the object has been modified; please apply your changes to the latest version and try again"

It was supposed to be renewed on October 29th, it tried 5 times with the error above.

The existing certificate is 50 days old, there is a 10 days old request for the a new one, that is Approved but not Ready:

$ kubectl get certificaterequests.cert-manager.io -n registry-prod 
NAME                          APPROVED   DENIED   READY   ISSUER        REQUESTOR                                        AGE
harbor-harbor-ingress-vswn8   True                False   letsencrypt   system:serviceaccount:kube-system:cert-manager   10d
$ kubectl get certificates.cert-manager.io -n registry-prod 
NAME                    READY   SECRET                  AGE
harbor-harbor-ingress   True    harbor-harbor-ingress   50d

The issue is that somehow the cluster issuers are gone:

$ kubectl get clusterissuers.cert-manager.io -A              
No resources found

The cluster is created with cert-manager disabled and then cert-manager is installed as a separate chart:

helm ls -A                                 
NAME        	NAMESPACE  	REVISION	UPDATED                                	STATUS  	CHART              	APP VERSION
cern-magnum 	kube-system	1       	2023-08-25 10:07:45.464379991 +0000 UTC	deployed	cern-magnum-0.12.2 	           
cert-manager	kube-system	1       	2023-09-19 13:08:38.2041699 +0200 CEST 	deployed	cert-manager-v1.7.1	v1.7.1  

Annotation on challenges.acme.cert-manager.io CRD is wrong:

metadata:
  annotations:
    cert-manager.io/inject-ca-from-secret: kube-system/cern-magnum-cert-manager-webhook-ca

Actually on all cert-manager CRDs.

We always deploy CRDs from dependencies, regardless of the enabled: true in cern-magnum's values.yaml. This then causes conflict when it's installed manually.

The secret is called cert-manager-webhook-ca instead of cern-magnum-cert-manager-webhook-ca. It's the same setup on staging.

• Does this mean the certificate creation was never working on this clusters? The certificate was copied from cci-infra-002 as part of tls-secret-sync job) and the renewal was not checked?

Do we have to patch all cert manager crds? Not sure what is the best solution.

I think indeed the cert was copied between the previous and new clusters.

We need to review and clean this up.

Also the cert-manager's webhook CA secret kube-system/cern-magnum-cert-manager-webhook-ca is missing (Pod cert-manager-cainjector-c8657d84f-zb9mc):

E1108 06:17:35.176009       1 sources.go:201] cert-manager/secret/customresourcedefinition/generic-inject-reconciler "msg"="unable to fetch associated secret" "error"="Secret \"cern-magnum-cert-manager-webhook-ca\" not found" "resource_kind"="CustomResourceDefinition" "resource_name"="challenges.acme.cert-manager.io" "resource_namespace"="" "resource_version"="v1" "secret"={"Namespace":"kube-system","Name":"cern-magnum-cert-manager-webhook-ca"}

This is being spammed a lot there.

What happens

So after investigating with Robert, this is our findings:

• It seems that Diogo was decoupling cert-manager from cern-magnum. Cern-magnum is installed without cert-manager, as a result there are no cluster issuers. From his ticket for harbor upgrade:

• The issuers should be installed as part of registry chart: https://gitlab.cern.ch/kubernetes/automation/releases/kops-registry/-/blob/master/charts/kops-registry/templates/cert-manager-issuer.yaml?ref_type=heads.
  • There is a check for {{- if .Capabilities.APIVersions.Has "cert-manager.io/v1" }} and I will check in ArgoCD

How to solve

So, on ArgoCD we hardcode the monitoring capability, not the cert-manager one:

helm template $ARGOCD_APP_NAME --api-versions=monitoring.coreos.com/v1
    --include-crds -n $ARGOCD_APP_NAMESPACE ${ARGOCD_ENV_HELM_ARGS} .

See this MR for more context: https://gitlab.cern.ch/kubernetes/automation/releases/kops/-/merge_requests/35

If registry relies on this capability, we need to provide it in the HELM_ARGS.

Let's drop that check, it's not useful since we know we always need it. Test on registry-staging and we update next week.

Okay, I'll open an MR

Once this is fixed, we still have to figure out the crds issue: #43 (comment 7285426).

!47 (merged)

changed iteration to Kubernetes Sprints Nov 15, 2023 - Nov 28, 2023

The issuers were applied manually and the certificate created:

The fix will come with the restructuring of the repository: #44 (closed)

closed