diff --git a/docs/logs/introduction.md b/docs/logs/introduction.md index 53df0fbb8d57e6d1f4ab0688d9efc2b7548feda3..b84b11f6ddb3f10395833635c07916fa9267f490 100644 --- a/docs/logs/introduction.md +++ b/docs/logs/introduction.md @@ -1,8 +1,11 @@ # Log monitoring The Linux team is responsible for 3 main services: + - The main distribution server: linuxsoft.cern.ch + - The installation infrastructure servers: aims.cern.ch + - The RPM build service: koji.cern.ch Each service is composed by test and prod virtual machines configured via Puppet. @@ -20,14 +23,21 @@ In our case the destination of the logs is [OpenSearch](https://opensearch.org/) I recommend you read the [Fluentbit documentation](https://docs.fluentbit.io/manual), but in summary, we define which logs are we looking for, using the input configuration. Then we can modify them using the filters and send them to Opensearch using the output configuration. We are using the `http` output plugin to send the logs to monit-logs.cern.ch. The MONIT team generated an http_password that can be found here: `tbag show --hg lsb monit_logs_tenant_lsb_password`. + The [fluentbit module](https://gitlab.cern.ch/ai/it-puppet-module-fluentbit) is maintained by MONIT and contains the fluentbit installation and all the different inputs, parsers, filters and outputs that can be used. The [linux-monitoring module](https://gitlab.cern.ch/ai/it-puppet-module-linux_monitoring) is common structure to: + - Define the fluent-bit configuration constants. + - Instantiate a fluent-bit service with the hostgroup name. + - Create the required configuration files and the service instance. + - Enable the debug in `/var/log/messages` using [stdout filter](https://docs.fluentbit.io/manual/pipeline/filters/standard-output). + - Define a way to receive more than one input, filters and outputs and handle its properties. + This module uses the fluentbit module. All the fluentbit configuration needed in each hostgroup will be made by calling `linux-monitoring` module. @@ -37,6 +47,9 @@ One of mandatory fields is the `agent_name` that is used to create the fluentbit The other mandatory fields are `input_plugins`, `output_plugins` and `filters`. So, let's take a look at each main service and understand which logs are we interested in and how this is being processed by Fluentbit. + - [LXSOFT](./lxsoft.md) + - [AIMS](./aims.md) + - [KOJI](./lsb.md) \ No newline at end of file diff --git a/docs/logs/procedures.md b/docs/logs/procedures.md index aff8fc338c29f2dfeb7916d5a85c621a2ab69c18..c747de26d60961e3ef5a43dc95f627c59bea860c 100644 --- a/docs/logs/procedures.md +++ b/docs/logs/procedures.md @@ -100,8 +100,6 @@ Setting the `debug_log_files` to true will enable the `stdout` filter and send a Logs can be sent to the monit-logs HTTP endpoint. -> **_NOTE:_** The endpoint http://monit-logs.cern.ch:10012 is getting deprecated. - We need to send the logs to https://monit-logs.cern.ch:10013/\<producer\>. More information in [MONIT documentation](https://monit-docs.web.cern.ch/logs/http/#sending-data). @@ -125,3 +123,13 @@ hg_lsb::output_plugins: ``` The http_password that can be found here: `tbag show --hg lsb monit_logs_tenant_lsb_password`. + +### 6) Create a Index template and a Index patterns in Opensearch + +Once we have defined the fields that will search in Opensearch, two things needs to be created: + +- Index Management -> Templates -> Create Template + +- Dashboards Management -> Index patterns -> Create a Index pattern + +> **_NOTE:_** Every time that you change the fields, you need to recreate the Index Template and delete the index in question from `Index Management -> Templates` (it will appear again when new data arrives). Also, please refresh the Index pattern.