deploy: configure falco with audit logging monitoring and alerts forwarding disabled by default

Integrates falco and falcosidekick (alert forwarding provider) inside of cern-magnum: falco is enabled by default for the basic falco rule sets with alerts being logged to stdout, users need to configure falcosidekick themselves (via values.yaml) for their alerting method of choice.

A subset of the default falco rules need to be patched for cern-magnum to allow list certain behaviours. Rule sets can be managed as OCI artifacts or directly specified in the helm values.yaml under customRules: {}. I have setup the management using OCI artifacts so that we do not need to duplicate config between the kops and cern-magnum setup, which also keeps the values.yaml smaller.

Management of these rules is in a new repo: https://gitlab.cern.ch/kubernetes/security/falco-rules. Please can this be also given a quick review as I have had to push to master to ensure all the CI is configured properly.

Audit Logging

Falco provides integration to read directly from native kubernetes audit logs, however as this is not setup already in mangum I have manually modified the master node to configure. Updates to magnum will need to be made as discussed.

Add the following files to master node:

# /etc/kubernetes/audit-policies/policy.yaml
# ref: https://kubernetes.io/docs/tasks/debug/debug-cluster/audit/
apiVersion: audit.k8s.io/v1
kind: Policy
omitStages:
  - "RequestReceived"
rules:
  - level: RequestResponse
    resources:
    - group: ""
      resources: ["pods"]
  - level: Metadata
    resources:
    - group: ""
      resources: ["pods/log", "pods/status"]

  - level: None
    resources:
    - group: ""
      resources: ["configmaps"]
      resourceNames: ["controller-leader"]

  - level: None
    users: ["system:kube-proxy"]
    verbs: ["watch"]
    resources:
    - group: "" # core API group
      resources: ["endpoints", "services"]

  - level: None
    userGroups: ["system:authenticated"]
    nonResourceURLs:
    - "/api*" # Wildcard matching.
    - "/version"

  - level: Request
    resources:
    - group: "" # core API group
      resources: ["configmaps"]
    namespaces: ["kube-system"]

  - level: Metadata
    resources:
    - group: "" # core API group
      resources: ["secrets", "configmaps"]

  - level: Request
    resources:
    - group: "" # core API group
    - group: "extensions" # Version of group should NOT be included.

  - level: Metadata
    # Long-running requests like watches that fall under this rule will not
    # generate an audit event in RequestReceived.
    omitStages:
      - "RequestReceived"
---
# /etc/kubernetes/webhook-config.yaml
apiVersion: v1
kind: Config
clusters:
  - name: k8s-audit
    cluster:
      # certificate-authority: /path/to/ca.crt # for https
      server: http://localhost:30007/k8s-audit
contexts:
  - context:
      cluster: k8s-audit
      user: ""
    name: default-context
current-context: default-context
preferences: {}
users: []

Add the following flags kube-apiserver config:

-audit-log-path=-
--audit-policy-file=/etc/kubernetes/audit-policies/policy.yaml
--audit-webhook-config-file=/etc/kubernetes/webhook-config.yaml

Follow up PRs:

• Document the process of enabling alert forwarding via falco sidekick to mattermost.
• Updates to magnum to configure audit logging policy and kube-apiserver flags to enable.

Closes #95 (closed).