NXCALS-1058 Enhance Filebeat - Logastash communication
#Merge request template @acc-logging-team - please have a look for approval.
This MR applies 2 small configuration additions on the Logstash side.
Moreover, naming changes have been applied to the node exporter's alerts, in order to achieve more descriptive notifications, about the incidents found and reported by Prometheus. (This does not change any of the current actual functionality).
Logstash additions:
- On the pipeline: There is reported from Prometheus Alerts that the communication experiences some request timeouts. By enlarging the client inactivity timeout, we can force the filebeat-logstash communication to stay longer active, rather than closing it, and forcing filebeat to re-establish it's communication. This could cause the following issue presented on the screenshot, taken from a dev grafana instance that reflects/confirms also the incidents on production.
- On the logstash JVM options, we can enforce the source socket to favor IPv4 stack, in order to avoid hostname resolution problems. This is done by setting the java.net.preferIPv4Stack to true.
The following changes applied to my dev environment and left to run for approx 2 days.
No other incidents like the above were experienced on the system after those changes took effect.