Some crds can break cluster creation when creating cluster from branch
Context
When we create a cluster from branch, we use the following script:
for d in \$(cat cern-magnum-\${branch}/Chart.yaml | yq '.dependencies[].name'); do if tar zxf cern-magnum-\${branch}/charts/\${d}*tgz; then cat \${d}/crds/*yaml >> cern-magnum-\${branch}/crds/generated.yaml 2>/dev/null || true; fi; rm -rf \${d}; done
which means for each dependency, we search for the crds folder, and copy all found crds to generated.yaml
Many charts (for example traefik, velero), they start the crds with ---
, example:
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
labels:
component: velero
annotations:
And if we just paste al the found crds to a common file, they will be separated by ---
, and everything will work without issues.
Problem
There are some charts that don't start their crds with ---
, and if we paste all of them to the same file, only the last crd will be created.
This is the case with gatekeeper. If we take a look at the generated.yaml, all the crds for gatekeeper are not separated:
description: URL is the url for the provider. URL is prefixed with https://.
type: string
type: object
type: object
served: true
storage: true
# <- here we need a separator
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
annotations:
controller-gen.kubebuilder.io/version: v0.14.0
labels:
gatekeeper.sh/system: "yes"
name: syncsets.syncset.gatekeeper.sh
spec:
group: syncset.gatekeeper.sh
As a result, from ALL the gatekeeper crds, only syncsets.syncset.gatekeeper.sh
will be created, as it is the last one in this unseparated blob of yaml.
The bigger issue is that this gatekeeper, by not having a separator, also takes into the unseparated yaml blob, the last non gatekeeper crd. For the feat-gatekeeper
branch, the last crd before gatekeeper is volumesnapshots.snapshot.storage.k8s.io
, as a result this crd is not created, and breaks the snapshotter controller pod, and as a result breaks the cluster creation (it will never succeed).