Add post on ingress usage at CERN

content/post/a-journey-with-ingress.md

+---
+title: "A Journey with Ingress"
+date: 2019-10-08T09:00:00+02:00
+author: Ricardo Rocha
+tags: ["kubernetes", "ingress", "traefik", "nginx", "haproxy"]
+---
+
+{{% figure src="../../img/post/ingress-diagram.png"
+     caption="Two of the Basic Ingress Use Cases"
+     class="caption"
+%}}
+
+There's a lot of ongoing work to check what a better Ingress in Kubernetes
+might look like. Ingress v1 is easy to use and does the job for many use cases,
+but falls short on others. I highly recommend you listen to [Tim Hockin's
+episode](https://kubernetespodcast.com/episode/041-ingress/) on this topic in
+the Kubernetes Podcast by Google, or read about the reasoning behind the
+[Ingress Route](https://blog.heptio.com/improving-the-multi-team-kubernetes-ingress-experience-with-heptio-contour-0-6-55ae0c0cadef)
+CRD from Contour.
+
+In this post we document our own journey in Ingress and how we tried to
+accomodate the different use cases from our users.
+
+## A Quick Digression: Resources and Drivers
+
+In Kubernetes there are many built-in resources that will behave similarly no
+matter the environment, such as Deployments, ReplicaSets, DaemonSets, etc.
+Others are exposed via a common API, but rely on the cluster administrator
+setting up additional components to be able to work. These include among others
+the container network interface (CNI), container storage interface (CSI) and
+also Ingress.
+
+How much is in the *common* API part varies greatly. As an example, the CSI
+seems to (at least up to now) expose the level of abstraction required for our
+use cases. We can have a cluster operator define the appropriate Storage Class
+for our on-prem clusters:
+
+```yaml
+apiVersion: storage.k8s.io/v1
+kind: StorageClass
+metadata:
+  name: fast
+  provisioner: manila-provisioner
+  parameters:
+   backend: csi-cephfs
+    csi-driver: csi-cephfsplugin
+    osSecretName: os-trustee
+    osSecretNamespace: kube-system
+    protocol: CEPHFS
+```
+
+And let our users set their persistent volume claims (PVC) appropriately:
+```yaml
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: my-volume
+spec:
+  storageClassName: fast
+  accessModes:
+    - ReadWriteOnce
+  resources:
+    requests:
+      storage: 10Gi
+```
+
+This separation of roles allows the operator to redefine the Storage
+Class for our clusters running on public clouds, while the end user can still
+rely on the same unchanged volume claim definition:
+```yaml
+apiVersion: storage.k8s.io/v1
+kind: StorageClass
+metadata:
+  name: fast
+provisioner: kubernetes.io/gce-pd
+parameters:
+  type: pd-ssd
+```
+
+The Ingress v1 resource also needs a controller behind, and there are multiple
+options - Traefik, Nginx, HAProxy and several service mesh backed solutions.
+
+In our and many others operators experience, the Ingress is very easy to use
+and for the basic use cases is enough. But the common part of its
+definition seems under specified which leads to a wild west of annotations.
+
+Read along for details.
+
+## Level 1: Basic Ingress with Traefik
+
+We rely mostly on HAProxy for our non-containerized environments. But when
+looking at this service the first time, we chose Traefik as the *cloud native*
+solution. It worked great out of the box:
+```yaml
+apiVersion: extensions/v1beta1
+kind: Ingress
+metadata:
+  name: nginx-ingress-tls
+  namespace: default
+  annotations:
+    kubernetes.io/ingress.class: traefik
+    ingress.kubernetes.io/ssl-redirect: true
+    traefik.ingress.kubernetes.io/frontend-entry-points: https
+spec:
+  rules:
+  - host: myclusterdns.cern.ch
+    http:
+      paths:
+      - path: /
+        backend:
+          serviceName: nginx-service
+          servicePort: 8080
+  tls:
+    - secretName: mycluster-tls-cert
+```
+
+This is all you need for a SSL terminated endpoint, with multiple backend
+replicas. Updates are almost instantaneous, which is a huge difference
+compared to our existing puppet based deployments.
+
+But the required annotations indicated that we were exposing details of the
+infrastructure implementation to the end user. Even for basic things like which
+protocols to offer or configuring redirection.
+```yaml
+  annotations:
+    kubernetes.io/ingress.class: traefik
+    ingress.kubernetes.io/ssl-redirect: true
+    traefik.ingress.kubernetes.io/frontend-entry-points: https
+```
+
+It works, but is not ideal as it reduces our flexibility to change things on
+the infrastructure side without impacting the end user.
+
+## Level 2: Passthrough SSL and Nginx
+
+By this time users with basic use cases were happy, and so were we. The day
+quickly came when someone asked for features that Traefik did not
+support at the time: TCP redirection for an IMAP backend, and SSL passthrough to
+support the not so widely used [X509 proxy
+certificates](https://tools.ietf.org/html/rfc3820).
+
+So we introduced Nginx as an additionally supported controller:
+```yaml
+apiVersion: extensions/v1beta1
+kind: Ingress
+metadata:
+  name: nginx-ingress
+  annotations:
+    kubernetes.io/ingress.class: nginx
+    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
+    nginx.ingress.kubernetes.io/ssl-redirect: "true"
+spec:
+  rules:
+  - host: myclusterdns.cern.ch
+    http:
+      paths:
+      - path: /
+        backend:
+          serviceName: nginx-service
+          servicePort: 443
+  tls:
+  - hosts:
+    - myclusterdns.cern.ch
+```
+
+And this also meant documenting a new set of annotations for our users, even
+for basic tasks like http->https redirection. In addition to now supporting two
+controllers on the infrastructure side.
+
+## Level 3: Header Based Redirection and Public Clouds
+
+It didn't stop there and one of our users came asking to redirect to one of
+a set of services based on HTTP headers. There's no way to express this in the
+Ingress resource, and there doesn't seem to be a way to do this with Traefik or
+Nginx. The only option seems to be using HAProxy, and even there the controller
+does not seem to expose this even with annotations.
+
+And then we also have deployments in public clouds, where we need another set
+of annotations as required by their own Ingress controller. Example with the
+Google Cloud:
+```yaml
+apiVersion: extensions/v1beta1
+kind: Ingress
+metadata:
+  name: my-ingress-2
+  annotations:
+    kubernetes.io/ingress.allow-http: "false"
+```
+
+## What Next
+
+Ingress is one of the most popular features of Kubernetes as it significantly
+simplifies the otherwise complex setup of L4 and L7 load balancers. The
+existing Ingress resource works very well for the basic use cases, and when a
+single controller is capable of offering all the required functionality.
+
+But unlike other pluggable parts of Kubernetes it doesn't completely hide the
+details of the infrastructure, which leads to the end user being aware of
+configuration details that are specific to the chosen controller.
+
+It will be interesting to see what the community comes up with to tackle these
+issues and improve this area of Kubernetes. Ingress v1 seems to be going GA so
+that people can focus on what the next Ingress (in whatever form) might look like.
+
+Skipping the service mesh options which we don't understand well yet, we see
+some nice efforts:
+
+* The IngressRoute CRD, as defined by
+  [Contour](https://github.com/projectcontour/contour/blob/master/design/ingressroute-design.md)
+  and [Traefik](https://docs.traefik.io/providers/kubernetes-crd/). It doesn't
+  look like they follow the same definition though, and our last use case for
+  header based redirection is not covered by the built-in definitions
+* Something newer like [HTTPProxy](https://github.com/projectcontour/contour/blob/master/design/routing-design.md)
+  which seems to cover all our use cases
+