Skip to content
Snippets Groups Projects
Commit dd793583 authored by Nacho Barrientos's avatar Nacho Barrientos
Browse files

[MONIT-4187] Add option to only send logs from a list of namespaces

parent 1bc29e4f
No related branches found
No related tags found
1 merge request!61[MONIT-4187] Allow specifing a namespace filter for logs
......@@ -43,6 +43,7 @@ See [getting started](docs/getting_started.md).
| fluentbit.image.repository | string | `"registry.cern.ch/monit/cern-it-monitoring-fluent-bit"` | image repository applied to all Fluent Bit instances |
| fluentbit.image.tag | string | `"3.2.6"` | image tag applied to all Fluent Bit instances |
| kubernetes.clusterName | string | `""` | name of the kubernetes cluster to monitor. This value will be appended to very metric and log via k8s_cluster_name label. This bit is required if fluentbit is enabled (default) |
| logs.collectedNamespaces | list | `["default"]` | restrict log sending from pods in these namespaces. you can provide literals or regular expressions. |
| logs.enabled | bool | `false` | indicates if logs metrics components should be enabled or not. If set to false no logs component will be installed nor configured |
| logs.fluentbit.customParsers | string | `""` | |
| logs.fluentbit.enabled | bool | `false` | indicates if fluentbit logs component should be installed or not |
......
......@@ -12,12 +12,22 @@ To **enable** or **disable** logs collection:
logs:
enabled: true # Set to 'false' to disable logs collection
```
If `logs.enabled` is set to `false`, no logging components (such as Fluentbit) will be installed or configured.
## 2. Base Logs Collected
By default, the chart deploys **Fluentbit** as a DaemonSet to gather logs from Kubernetes containers. Fluentbit is highly efficient and lightweight but can generate substantial API requests in large deployments (100+ nodes) due to the Kubernetes filter.
Without further user-provided configuration, only logs generated by pods in the `default` namespace are collected. To augment the set of mined namespaces, please configured the desired list of namespaces via the `logs.collectedNamespaces` setting, for example:
```yaml
logs:
collectedNamespaces:
- myapp
- user.+
```
### Fluentbit Configuration
Fluentbit collects logs from the `/var/log/containers/` directory and applies a set of filters before forwarding them to the central monitoring system via the OpenTelemetry protocol.
......
......@@ -49,3 +49,41 @@ tests:
path: data["custom_parsers.conf"]
- exists:
path: data["fluent-bit.conf"]
- it: should filter in the default namespace by default
set:
tenant.name: test
tenant.password: test
kubernetes.clusterName: test
logs.enabled: true
logs.fluentbit.enabled: true
asserts:
- matchRegex:
path: data["fluent-bit.conf"]
pattern: "(?m).*Regex namespace \\(default\\).*"
- it: should add a filter to filter in namespaces if configured (1)
set:
tenant.name: test
tenant.password: test
kubernetes.clusterName: test
logs.enabled: true
logs.fluentbit.enabled: true
logs.collectedNamespaces:
- test
asserts:
- matchRegex:
path: data["fluent-bit.conf"]
pattern: "(?m).*Regex namespace \\(test\\).*"
- it: should add a filter to filter in namespaces if configured (1+)
set:
tenant.name: test
tenant.password: test
kubernetes.clusterName: test
logs.enabled: true
logs.fluentbit.enabled: true
logs.collectedNamespaces:
- test2
- test3
asserts:
- matchRegex:
path: data["fluent-bit.conf"]
pattern: "(?m).*Regex namespace \\(test2\\|test3\\).*"
......@@ -344,6 +344,11 @@ logs:
# -- indicates if logs metrics components should be enabled or not. If set to false no logs component will be installed nor configured
enabled: false
# -- restrict log sending from pods in these namespaces.
# you can provide literals or regular expressions.
collectedNamespaces:
- default
# Fluentbit is deployed as a daemon set to gather logs. Be careful on big
# deployments (100+) as the fluentbit kubernetes filter will produce requests
# to the kubernetes API.
......@@ -441,6 +446,13 @@ logs:
Rename kubernetes_pod pod
Rename kubernetes_container_name container
{{ if .Values.logs.collectedNamespaces -}}
[FILTER]
Name grep
Match kube.*
Regex namespace ({{ join "|" .Values.logs.collectedNamespaces }})
{{- end }}
[FILTER]
Name lua
Match *
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment