Skip to content

Investigation of unhealthy pods after latest release

After the release performed for the non-critical prod sites on 2/10, see !251 (merged), some cases of sites with pods in a crash loop back-off status have been identified:

NAMESPACE                                          NAME                                                            READY   STATUS                   RESTARTS         AGE
apc-test                                           apc-test-8687d5866b-2q6mx                                       5/6     CrashLoopBackOff         6 (4m56s ago)    10m
be-dep-ea                                          ea-d10-test1-bdb6d6fcf-9tw6x                                    5/6     CrashLoopBackOff         6 (4m47s ago)    10m
cafein                                             cafein-546987dbbc-9vq5b                                         5/6     CrashLoopBackOff         6 (4m39s ago)    10m
cafein                                             cafein-web-team-79685ddb4d-dp26j                                6/7     CrashLoopBackOff         6 (4m33s ago)    10m
fluka2019                                          test2-d10-fluka2019-7c49d858b6-95ccp                            5/6     CrashLoopBackOff         6 (3m35s ago)    9m43s
lhc-training                                       lhc-training-5fc6c8cc9d-s9lbx                                   5/6     CrashLoopBackOff         6 (3m4s ago)     9m13s
na-cons-project                                    nacons-project-57dc754d4f-w2whx                                 5/6     CrashLoopBackOff         6 (3m7s ago)     9m3s
staff-association                                  staff-asociation-b4bb7769f-9q8kv                                5/6     CrashLoopBackOff         6 (2m30s ago)    8m31s
staff-association                                  staff-association-test-modules-6ccd48c4b8-hbs76                 6/7     CrashLoopBackOff         6 (2m30s ago)    8m30s
ut-lhcb                                            ut-lhcb-677fcf47f5-v98xl                                        5/6     CrashLoopBackOff         6 (117s ago)     8m3s
vito                                               vito-mtest-7b84457888-cp97j                                     5/6     CrashLoopBackOff         6 (115s ago)     8m3s

The issue is onphp-fpm container and the error logged is:

In Container.php line 147:
Circular reference detected for service "token", path: "token -> cache_tags
.invalidator -> plugin.manager.block -> logger.channel.default -> logger.fa
ctory -> logger.filelog".

Let's use this issue for investigation purposes and follow-up on these websites.

fyi @crdeoliv

Edited by Christina Petala