Skip to content
Snippets Groups Projects
Commit 56d65cd2 authored by Alex Iribarren's avatar Alex Iribarren
Browse files

Merge branch 'monit' into 'master'

update monit docs

See merge request !129
parents 44b266ee 49a47709
Branches
No related tags found
1 merge request!129update monit docs
Pipeline #10364527 passed
# Log monitoring
The Linux team is responsible for 3 main services:
- The main distribution server: linuxsoft.cern.ch
- The installation infrastructure servers: aims.cern.ch
- The RPM build service: koji.cern.ch
Each service is composed by test and prod virtual machines configured via Puppet.
......@@ -20,14 +23,21 @@ In our case the destination of the logs is [OpenSearch](https://opensearch.org/)
I recommend you read the [Fluentbit documentation](https://docs.fluentbit.io/manual), but in summary, we define which logs are we looking for, using the input configuration. Then we can modify them using the filters and send them to Opensearch using the output configuration.
We are using the `http` output plugin to send the logs to monit-logs.cern.ch. The MONIT team generated an http_password that can be found here: `tbag show --hg lsb monit_logs_tenant_lsb_password`.
The [fluentbit module](https://gitlab.cern.ch/ai/it-puppet-module-fluentbit) is maintained by MONIT and contains the fluentbit installation and all the different inputs, parsers, filters and outputs that can be used.
The [linux-monitoring module](https://gitlab.cern.ch/ai/it-puppet-module-linux_monitoring) is common structure to:
- Define the fluent-bit configuration constants.
- Instantiate a fluent-bit service with the hostgroup name.
- Create the required configuration files and the service instance.
- Enable the debug in `/var/log/messages` using [stdout filter](https://docs.fluentbit.io/manual/pipeline/filters/standard-output).
- Define a way to receive more than one input, filters and outputs and handle its properties.
This module uses the fluentbit module.
All the fluentbit configuration needed in each hostgroup will be made by calling `linux-monitoring` module.
......@@ -37,6 +47,9 @@ One of mandatory fields is the `agent_name` that is used to create the fluentbit
The other mandatory fields are `input_plugins`, `output_plugins` and `filters`.
So, let's take a look at each main service and understand which logs are we interested in and how this is being processed by Fluentbit.
- [LXSOFT](./lxsoft.md)
- [AIMS](./aims.md)
- [KOJI](./lsb.md)
\ No newline at end of file
......@@ -100,8 +100,6 @@ Setting the `debug_log_files` to true will enable the `stdout` filter and send a
Logs can be sent to the monit-logs HTTP endpoint.
> **_NOTE:_** The endpoint http://monit-logs.cern.ch:10012 is getting deprecated.
We need to send the logs to https://monit-logs.cern.ch:10013/\<producer\>.
More information in [MONIT documentation](https://monit-docs.web.cern.ch/logs/http/#sending-data).
......@@ -125,3 +123,13 @@ hg_lsb::output_plugins:
```
The http_password that can be found here: `tbag show --hg lsb monit_logs_tenant_lsb_password`.
### 6) Create a Index template and a Index patterns in Opensearch
Once we have defined the fields that will search in Opensearch, two things needs to be created:
- Index Management -> Templates -> Create Template
- Dashboards Management -> Index patterns -> Create a Index pattern
> **_NOTE:_** Every time that you change the fields, you need to recreate the Index Template and delete the index in question from `Index Management -> Templates` (it will appear again when new data arrives). Also, please refresh the Index pattern.
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment