fluentd pods fail to start silently if connecting to rubygems fails
ERROR: Could not find a valid gem 'fluent-plugin-concat' (>= 0), here is why:
Unable to download data from https://rubygems.org/ - timed out (https://api.rubygems.org/specs.4.8.gz)
The fluentd daemonset downloads ruby plugins on startup. If the pod fails to connect to the ruby registry, it fails silently to download the plugin and rejects log records with error (to stdout).
The issue with connecting to rubygems was due to ipv6 as disabling ipv6 solved the problem.
It would still be a good idea to install the plugins into the image to avoid missing logs in production in case the upstream package registry has an outage and the fluentd pods are restarted.