GitOps Getting Started

Note: This is an updated version of the guide with helm3 and SOPS - checkout this branch for the old version using helm2 and the helm secrets plugin

An example of an application deployment and configuration managed using GitOps and the tool Flux.

The goal is to have a central git repo hosting all the configuration and deployments for an application, including its dependencies. Permissions set in this repo also define who is allowed to update the configuration.

This example has two main components:

• a frontend web application based on wordpress
• a backend MySQL database

It should be enough to give you hints on how to manage your own deployments in the same way.

Pre-Requisites

• A Kubernetes cluster up and running, instructions here
• Helm v3, instructions here

Flux Overview

Flux has two main components:

• Flux itself, which is responsible for syncing with the remote git repository. We rely on the Helm integration, so Flux will instantiate in the cluster the HelmRelease custom resources defined in the releases directory
• Helm Operator, which watches for the HelmRelease custom resources and applies them in the cluster - basically by doing helm install/upgrade/...

Check the helm releases defined in this repo under the releases directory to see what goes in these resources.

Structure and Releases

The structure of this repo is as follows:

• chart: The umbrella chart. This is where all the default configuration for the dependencies is stored, inside the values.yaml file. You can also add custom manifests under the templates directory
• namespaces: Flux expects all the required namespaces to be defined here. In this example we have both prod and stg
• releases: The different deployments of the application (staging, production), each with overrides for the configuration as required - completing the default values. These are Flux HelmRelease resources

Deployment

You need to deploy both the helm operator and flux itself:

$ helm upgrade -i helm-operator fluxcd/helm-operator --namespace flux \
    --version 1.1.0 --values helm-operator-values.yaml

$ helm upgrade -i flux fluxcd/flux --namespace flux \
    --version 1.3.0 \
    --set git.url=https://gitlab.cern.ch/helm/releases/gitops-getting-started \
    --values flux-values.yaml

The files helm-operator-values.yaml and flux-values.yaml contain the configuration for each of the components. You can change these as required.

Check the logs to see the multiple releases are being picked up:

$ kubectl -n flux logs -f deployment.apps/helm-operator
$ kubectl -n flux logs -f deployment.apps/flux

All going well you should see them in helm:

$ helm ls -A
NAME                        NAMESPACE   REVISION    UPDATED                                     STATUS      CHART                           APP VERSION
flux                        flux        1           2020-06-16 14:54:59.842375787 +0200 CEST    deployed    flux-1.3.0                      1.19.0     
gitops-getting-started-prod prod        2           2020-06-16 20:19:55.789758883 +0000 UTC     deployed    gitops-getting-started-0.1.0               
gitops-getting-started-stg  stg         2           2020-06-16 20:18:04.590517379 +0000 UTC     deployed    gitops-getting-started-0.1.0               
helm-operator               flux        1           2020-06-16 14:54:50.404127842 +0200 CEST    deployed    helm-operator-1.1.0             1.1.0      

Secrets

This is not about handling Kubernetes Secret resources, that's handled by the different helm charts. This is a solution to handle sensitive information stored in the helm yaml files (that often populate the Kubernetes Secrets).

We rely on sops and its OpenStack Barbican plugin to manage the encrypted content in the helm values files - Flux has built-in sops support meaning it will be able to decrypt the values when deploying.

Note: barbican support is still under review in upstream sops, please use our fork for now

Setup

First install the sops client (we need our custom build for now to get barbican support):

wget https://gitlab.cern.ch/cloud/sops/-/jobs/8834328/artifacts/raw/sops?inline=false

Then make sure you have your OpenStack environment setup. If you're relying on kerberos authentication, you'll need to fetch a token first:

export OS_TOKEN=$(openstack token issue -c id -f value)

Important: The environment above must match the project owning the Kubernetes cluster where Flux is being deployed

Encrypting

sops requires a master key that will be used to encrypt the different data encryption keys. Only the master key needs to be in barbican.

The first step is to generate a master key and store it in barbican:

$ export KEY="$(openssl rand -base64 32)\n$(openssl rand -base64 12)"
$ openstack secret store -s symmetric -p "$(echo -e $KEY)" -n gitops-getting-started

Check the secret href of the generated secret and pass that to sops:

sops --barbican https://openstack.cern.ch:9311/v1/secrets/SOMEID \
    releases/prod/values.yaml

By default sops will encrypt all values in the yaml file, which is not ideal. You can pass a regex matching the keys that should be encrypted, here's an example to encrypt all entries named 'password' or ending with 'key':

sops --barbican https://openstack.cern.ch:9311/v1/secrets/SOMEID \
    --encrypted-regex '^(password)|(.*key)$' \
    releases/prod/values.yaml

These parameters are only required when first creating the file - they are stored as metadata inside the file along the contents. To later edit the file you just need:

sops releases/prod/values.yaml

And to display the decrypted content:

sops -d releases/prod/values.yaml

TIP: Instead of passing the secret href and regex as params, use a .sops.yaml file which can be safely committed to the git repo. This is especially useful if you plan to handle multiple values file and want to use a single master key.

$ cat .sops.yaml
creation_rules:
  - encrypted_regex: '^(password)|(.*key)$'
    barbican_secret_href: "https://openstack.cern.ch:9311/v1/secrets/SOMEID"

When relying on this config file you don't need to pass any params to sops even when creating/encrypting a new file.

FAQ

What's the best practice to update the release values?

It depends which releases you want to affect.

• If you want the change to impact all releases (prod, staging, ...), change the values in the chart/application name/values.yaml file. These are the defaults for all releases
• If you want the change to impact only one release, change the corresponding yaml file in the releases/release-to-be-changed directory

Flux will detect the change and apply it locally on the cluster(s) where it is deployed.

How do i force Flux to sync with the remote repository?

Flux will sync periodically, following the value in the git.pollInterval parameter. If you've set this value conservatively and want to get Flux to sync immediately, try:

$ export FLUX_FORWARD_NAMESPACE=flux
$ fluxctl sync

How do i add custom manifests not part of any existing Helm chart?

You can add these additional manifests in the same way you would do for a normal chart. Put them under charts/application-name/templates and they will be picked up during the installation (and updates). You can use values as usual as well.

How do i restrict the deployment to only one release?

To restrict the deployment in this cluster to a single release, pass the allowed namespace parameter when deploying the helm operator:

$ helm install fluxcd/helm-operator --namespace flux --name helm-operator \
  --values helm-operator-values.yaml \
  --set allowNamespace=stg

Can i deploy the application in multiple clusters?

Absolutely. The repository knows nothing about the clusters the application will be deployed in, which is an advantage as you don't have to set the credentials anywhere outside the cluster(s).

Deploying multiple times in different clusters will also help you reduce blast radius, it's a good way to perform upgrades by redeploying the full setup and gradually moving resources to the new instance, and it's one of the possible alternatives to achieve high availability.