OIDC registration issue
During migration of sites to the webeos prod environment, some sites were't properly registered in the Application Portal:
berghaus
bfontana
camendol
campore
The error log for them is the same and contains the following:
oc -n openshift-cern-authz-operator logs authz-operator-79544d584f-6qzh5 --timestamps | grep berghau
2020-10-21T14:45:44.612950557Z {"level":"info","ts":1603291544.6128783,"logger":"controller_applicationregistration","msg":"Initializing AppReg Spec","Request.Namespace":"berghaus","Request.Name":"berghaus"}
2020-10-21T14:46:30.362688749Z {"level":"info","ts":1603291590.362631,"logger":"controller_applicationregistration","msg":"Initializing AppReg Status","Request.Namespace":"berghaus","Request.Name":"berghaus"}
2020-10-21T14:47:17.787344101Z {"level":"info","ts":1603291637.7872784,"logger":"controller_applicationregistration","msg":"Ensuring a matching Application in the AuthzAPI","Request.Namespace":"berghaus","Request.Name":"berghaus"}
2020-10-21T14:48:27.553790096Z {"level":"info","ts":1603291707.5537074,"logger":"controller_applicationregistration","msg":"Ensuring OIDC registration in the AuthzAPI","Request.Namespace":"berghaus","Request.Name":"berghaus"}
2020-10-21T14:48:31.908747905Z {"level":"error","ts":1603291711.9084227,"logger":"controller_applicationregistration","msg":"AuthzAPIError retrying to ensure OIDC registration in the AuthzAPI","Request.Namespace":"berghaus","Request.Name":"berghaus","error":"POST /api/v1.0/Registration/08d875d0-35c9-4521-8ca0-ab5da3c619f5/f0000000-0000-0000-0000-000000000051 \ndata: {\"consentRequired\":false,\"implicitFlowEnabled\":false,\"redirectUris\":[]}\nResponse 500: POST 'api/v1.0/client/openid': InternalServerError (500).","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/gitlab.cern.ch/paas-tools/operators/authz-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngitlab.cern.ch/paas-tools/operators/authz-operator/pkg/controller/applicationregistration.(*ReconcileApplicationRegistration).Reconcile.func1\n\t/go/src/gitlab.cern.ch/paas-tools/operators/authz-operator/pkg/controller/applicationregistration/applicationregistration_controller.go:123\ngitlab.cern.ch/paas-tools/operators/authz-operator/pkg/controller/applicationregistration.(*ReconcileApplicationRegistration).Reconcile\n\t/go/src/gitlab.cern.ch/paas-tools/operators/authz-operator/pkg/controller/applicationregistration/applicationregistration_controller.go:162\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/src/gitlab.cern.ch/paas-tools/operators/authz-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:256\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/gitlab.cern.ch/paas-tools/operators/authz-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/src/gitlab.cern.ch/paas-tools/operators/authz-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/gitlab.cern.ch/paas-tools/operators/authz-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/gitlab.cern.ch/paas-tools/operators/authz-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/gitlab.cern.ch/paas-tools/operators/authz-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
2020-10-21T14:48:31.920876823Z {"level":"error","ts":1603291711.9207504,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"applicationregistration-controller","request":"berghaus/berghaus","error":"POST /api/v1.0/Registration/08d875d0-35c9-4521-8ca0-ab5da3c619f5/f0000000-0000-0000-0000-000000000051 \ndata: {\"consentRequired\":false,\"implicitFlowEnabled\":false,\"redirectUris\":[]}\nResponse 500: POST 'api/v1.0/client/openid': InternalServerError (500).","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/gitlab.cern.ch/paas-tools/operators/authz-operator/vendor/github.com/go-logr/zapr/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/src/gitlab.cern.ch/paas-tools/operators/authz-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:258\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/gitlab.cern.ch/paas-tools/operators/authz-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/src/gitlab.cern.ch/paas-tools/operators/authz-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/gitlab.cern.ch/paas-tools/operators/authz-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/gitlab.cern.ch/paas-tools/operators/authz-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/gitlab.cern.ch/paas-tools/operators/authz-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
2020-10-21T14:50:00.393926647Z {"level":"info","ts":1603291800.3938565,"logger":"controller_applicationregistration","msg":"Ensuring OIDC registration in the AuthzAPI","Request.Namespace":"berghaus","Request.Name":"berghaus"}
2020-10-21T14:50:00.612507217Z {"level":"error","ts":1603291800.6122878,"logger":"controller_applicationregistration","msg":"Permanent error marked as transient!","Request.Namespace":"berghaus","Request.Name":"berghaus","error":"Failed to create initial roles: POST /api/v1.0/Application/08d875d0-35c9-4521-8ca0-ab5da3c619f5/roles \ndata: {\"applicationId\":\"08d875d0-35c9-4521-8ca0-ab5da3c619f5\",\"applyToAllUsers\":true,\"description\":\"Users must be from CERN or eduGAIN to have access\",\"displayName\":\"Default Allowed Users\",\"minimumLoaId\":\"f0000000-0000-0000-0000-0000000000b4\",\"name\":\"default-role\",\"required\":true}\nResponse 400: Duplicate entry '08d875d0-35c9-4521-8ca0-ab5da3c619f5-default-role' for key 'role_unique_name_for_app_key_ix'","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/gitlab.cern.ch/paas-tools/operators/authz-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngitlab.cern.ch/paas-tools/operators/authz-operator/pkg/controller/applicationregistration.(*ReconcileApplicationRegistration).Reconcile.func1\n\t/go/src/gitlab.cern.ch/paas-tools/operators/authz-operator/pkg/controller/applicationregistration/applicationregistration_controller.go:128\ngitlab.cern.ch/paas-tools/operators/authz-operator/pkg/controller/applicationregistration.(*ReconcileApplicationRegistration).Reconcile\n\t/go/src/gitlab.cern.ch/paas-tools/operators/authz-operator/pkg/controller/applicationregistration/applicationregistration_controller.go:162\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/src/gitlab.cern.ch/paas-tools/operators/authz-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:256\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/gitlab.cern.ch/paas-tools/operators/authz-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/src/gitlab.cern.ch/paas-tools/operators/authz-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/gitlab.cern.ch/paas-tools/operators/authz-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/gitlab.cern.ch/paas-tools/operators/authz-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/gitlab.cern.ch/paas-tools/operators/authz-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
2020-10-21T16:22:14.422252834Z {"level":"info","ts":1603297334.4219375,"logger":"controller_applicationregistration","msg":"Ensuring OIDC registration in the AuthzAPI","Request.Namespace":"berghaus","Request.Name":"berghaus"}
2020-10-21T16:22:14.558215424Z {"level":"error","ts":1603297334.5579538,"logger":"controller_applicationregistration","msg":"Permanent error marked as transient!","Request.Namespace":"berghaus","Request.Name":"berghaus","error":"Failed to create initial roles: POST /api/v1.0/Application/08d875d0-35c9-4521-8ca0-ab5da3c619f5/roles \ndata: {\"applicationId\":\"08d875d0-35c9-4521-8ca0-ab5da3c619f5\",\"applyToAllUsers\":true,\"description\":\"Users must be from CERN or eduGAIN to have access\",\"displayName\":\"Default Allowed Users\",\"minimumLoaId\":\"f0000000-0000-0000-0000-0000000000b4\",\"name\":\"default-role\",\"required\":true}\nResponse 400: Duplicate entry '08d875d0-35c9-4521-8ca0-ab5da3c619f5-default-role' for key 'role_unique_name_for_app_key_ix'","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/gitlab.cern.ch/paas-tools/operators/authz-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngitlab.cern.ch/paas-tools/operators/authz-operator/pkg/controller/applicationregistration.(*ReconcileApplicationRegistration).Reconcile.func1\n\t/go/src/gitlab.cern.ch/paas-tools/operators/authz-operator/pkg/controller/applicationregistration/applicationregistration_controller.go:128\ngitlab.cern.ch/paas-tools/operators/authz-operator/pkg/controller/applicationregistration.(*ReconcileApplicationRegistration).Reconcile\n\t/go/src/gitlab.cern.ch/paas-tools/operators/authz-operator/pkg/controller/applicationregistration/applicationregistration_controller.go:162\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/src/gitlab.cern.ch/paas-tools/operators/authz-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:256\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/gitlab.cern.ch/paas-tools/operators/authz-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/src/gitlab.cern.ch/paas-tools/operators/authz-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/gitlab.cern.ch/paas-tools/operators/authz-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/gitlab.cern.ch/paas-tools/operators/authz-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/gitlab.cern.ch/paas-tools/operators/authz-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
And from then on the last error: "Permanent error marked as transient!" , ""error":"Failed to create initial roles: POST /api/v1.0/Application/08d875d0-35c9-4521-8ca0-ab5da3c619f5/roles \ndata: {"applicationId":"08d875d0-35c9-4521-8ca0-ab5da3c619f5","applyToAllUsers":true,"description":"Users must be from CERN or eduGAIN to have access","displayName":"Default Allowed Users","minimumLoaId":"f0000000-0000-0000-0000-0000000000b4","name":"default-role","required":true}\nResponse 400: Duplicate entry '08d875d0-35c9-4521-8ca0-ab5da3c619f5-default-role' for key 'role_unique_name_for_app_key_ix'", " continues.
/cc @alossent