Re-enable HPA on the fluentd-aggregator
Initially in 4.5 we tried to deploy the fluentd-aggregator with a HPD unfortunately due to a RBAC bug this was not working properly. Bug was https://bugzilla.redhat.com/show_bug.cgi?id=1836087 it should be fixed on 4.6.1 so after the upgrade it would be a good idea to revisit this. I tried creating the RBAC manually just to see what would happen and it worked but it lead to another problem this time:
5m Warning FailedGetResourceMetric horizontalpodautoscaler/fluentd-aggregator unable to get metrics for resource memory: no metrics returned from resource metrics API
5m Warning FailedGetResourceMetric horizontalpodautoscaler/fluentd-aggregator unable to get metrics for resource cpu: no metrics returned from resource metrics API
However, this it might just be missing configuration from my part. I think we might be missing a servicemonitor
(check last link)
Relevant links to check out while trying to solve this:
- https://www.openshift.com/blog/enabling-monitoring-and-scaling-of-your-own-services-application
- https://docs.openshift.com/container-platform/4.5/monitoring/exposing-custom-application-metrics-for-autoscaling.html
- https://github.com/helm/charts/blob/master/stable/fluentd/templates/servicemonitor.yaml
/cc @alossent