Skip to content

Some crds can break cluster creation when creating cluster from branch

Context

When we create a cluster from branch, we use the following script:

for d in \$(cat cern-magnum-\${branch}/Chart.yaml | yq '.dependencies[].name'); do if tar zxf cern-magnum-\${branch}/charts/\${d}*tgz; then cat \${d}/crds/*yaml >> cern-magnum-\${branch}/crds/generated.yaml 2>/dev/null || true; fi; rm -rf \${d}; done

which means for each dependency, we search for the crds folder, and copy all found crds to generated.yaml

Many charts (for example traefik, velero), they start the crds with ---, example:

---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  labels:
    component: velero
  annotations:

And if we just paste al the found crds to a common file, they will be separated by ---, and everything will work without issues.

Problem

There are some charts that don't start their crds with ---, and if we paste all of them to the same file, only the last crd will be created.

This is the case with gatekeeper. If we take a look at the generated.yaml, all the crds for gatekeeper are not separated:

                description: URL is the url for the provider. URL is prefixed with https://.
                type: string
            type: object
        type: object
    served: true
    storage: true
# <- here we need a separator
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.14.0
  labels:
    gatekeeper.sh/system: "yes"
  name: syncsets.syncset.gatekeeper.sh
spec:
  group: syncset.gatekeeper.sh

As a result, from ALL the gatekeeper crds, only syncsets.syncset.gatekeeper.sh will be created, as it is the last one in this unseparated blob of yaml.

The bigger issue is that this gatekeeper, by not having a separator, also takes into the unseparated yaml blob, the last non gatekeeper crd. For the feat-gatekeeper branch, the last crd before gatekeeper is volumesnapshots.snapshot.storage.k8s.io, as a result this crd is not created, and breaks the snapshotter controller pod, and as a result breaks the cluster creation (it will never succeed).