Investigation of unhealthy pods after latest release
After the release performed for the non-critical prod sites on 2/10, see !251 (merged), some cases of sites with pods in a crash loop back-off status have been identified:
NAMESPACE NAME READY STATUS RESTARTS AGE
apc-test apc-test-8687d5866b-2q6mx 5/6 CrashLoopBackOff 6 (4m56s ago) 10m
be-dep-ea ea-d10-test1-bdb6d6fcf-9tw6x 5/6 CrashLoopBackOff 6 (4m47s ago) 10m
cafein cafein-546987dbbc-9vq5b 5/6 CrashLoopBackOff 6 (4m39s ago) 10m
cafein cafein-web-team-79685ddb4d-dp26j 6/7 CrashLoopBackOff 6 (4m33s ago) 10m
fluka2019 test2-d10-fluka2019-7c49d858b6-95ccp 5/6 CrashLoopBackOff 6 (3m35s ago) 9m43s
lhc-training lhc-training-5fc6c8cc9d-s9lbx 5/6 CrashLoopBackOff 6 (3m4s ago) 9m13s
na-cons-project nacons-project-57dc754d4f-w2whx 5/6 CrashLoopBackOff 6 (3m7s ago) 9m3s
staff-association staff-asociation-b4bb7769f-9q8kv 5/6 CrashLoopBackOff 6 (2m30s ago) 8m31s
staff-association staff-association-test-modules-6ccd48c4b8-hbs76 6/7 CrashLoopBackOff 6 (2m30s ago) 8m30s
ut-lhcb ut-lhcb-677fcf47f5-v98xl 5/6 CrashLoopBackOff 6 (117s ago) 8m3s
vito vito-mtest-7b84457888-cp97j 5/6 CrashLoopBackOff 6 (115s ago) 8m3s
The issue is onphp-fpm container and the error logged is:
In Container.php line 147:
Circular reference detected for service "token", path: "token -> cache_tags
.invalidator -> plugin.manager.block -> logger.channel.default -> logger.fa
ctory -> logger.filelog".
Let's use this issue for investigation purposes and follow-up on these websites.
fyi @crdeoliv
Edited by Christina Petala