Skip to content

NXCALS-927 Logstash ansible role and Grafana dashboard for pipeline metrics

Grigorios Avgitidis requested to merge feature-NXCALS-741-install-logstash into develop

#Merge request template @acc-logging-team - please have a look for approval.

This merge requests adds a very useful feature to our environment and it's logging propagation mechanisms.

Thanks to Logstash, we can now get profit of it's queue based pipelines. For our case, it will work like logging data sink/concentrator. It collects log data from our filebeats that are scattered to all the NXCALS nodes, and publishes them to our ES instance (provided by IT) via bulk insertion.

This reduces potential failing write attempts cause it will be the sole component for data handover to ES, instead of the previous, push-by-each filebeat strategy.

Furthermore, via the Prometheus data source in our Grafana instance, we can acquire some useful graphs that describe each node's performance at the corresponding target environment where NXCALS ecosystem lives.

This role has as target the machine hostgroup [logstashers], with nxcals-{{username}}4.cern.ch as sole member.

Edited by Grigorios Avgitidis

Merge request reports