Commit e7e83eb6 authored by Alexandre Lossent's avatar Alexandre Lossent
Browse files

Merge branch 'cern-4.8-dev' into 'cern-4.8'

[cern] Add env vars to deployment and annotations for LB service

See merge request !2
parents cb614126 f9e81ac5
Pipeline #3468811 passed with stage
in 2 minutes and 50 seconds
- project: 'paas-tools/infrastructure-ci'
file: 'docker-images-ci-templates/DockerImages.gitlab-ci.yml'
FROM AS builder
FROM AS builder
WORKDIR /ingress-operator
COPY . .
RUN make build
COPY --from=builder /ingress-operator/ingress-operator /usr/bin/
COPY manifests /manifests
ENTRYPOINT ["/usr/bin/ingress-operator"]
# CERN modifications
Openshift's ingress operator gives us very little flexibility for configuration the HAProxy router deployments and associated services.
One issue in particular made it necessary to fork the upstream component and add our own modifications: the ability to modify Loadbalancer services created by the operator (we need to set specific annotations for Openstack) and the ability to selectively enable the PROXY protocol for specific router shards.
Details can be found in [okd4-install!703](
The Gitlab repository is configured to automatically mirror the [upstream cluster-ingress-operator repository](
Use the following workflow to port our custom patches to a new release:
# Make sure your repo is up-to-date
git fetch origin
# Check out the pristine upstream release
git checkout origin/release-4.x
# Push a new branch with the pristine upstream release
git checkout -b cern-4.x
git push origin cern-4.x
# Create a new branch for development
git checkout -b cern-4.x-dev
Then, locate the latest patches that have been made by us.
This can be done by checking out an old release branch (e.g. `cern-4.7`) and looking at the commits prefixed with `[cern]`.
git log origin/cern_cloud-4.7
git cherry-pick [CERN_PATCH_REFS]
At this point, you might need to resolve the merge conflicts (*fingers crossed*).
Afterwards, commit your changes with a `[cern]` prefix and push them to the dev branch.
Finally, create a merge request from `cern-4.x-dev` to `cern-4.x` so someone else can review your changes.
# OpenShift Ingress Operator
Ingress Operator is an [OpenShift]( component which enables external access to cluster services by configuring Ingress Controllers, which route traffic as specified by OpenShift [Route]( and Kubernetes [Ingress]( resources.
......@@ -804,6 +804,11 @@ func IsProxyProtocolNeeded(ic *operatorv1.IngressController, platform *configv1.
switch ic.Status.EndpointPublishingStrategy.Type {
case operatorv1.LoadBalancerServiceStrategyType:
// We always want to enable the PROXY protocol when we are using services of type loadbalancer
if ic.Status.EndpointPublishingStrategy.Type == operatorv1.LoadBalancerServiceStrategyType {
return true, nil
// For now, check if we are on AWS. This can really be done for for any external
// [cloud] LBs that support the proxy protocol.
if platform.Type == configv1.AWSPlatformType {
......@@ -383,6 +383,39 @@ func desiredRouterDeployment(ci *operatorv1.IngressController, ingressController
{Name: "STATS_PASSWORD_FILE", Value: filepath.Join(statsVolumeMountPath, "statsPassword")},
// Enable IPv6 for HostNetwork router pods
// IPv6 is only enabled automatically when the entire cluster is running in Dual-Stack mode
// We don't do that, but our HAProxy routers should still listen on IPv6 addresses
// because they are running in HostNetwork mode on our ingress nodes (which have IPv6 addresses).
if ci.Status.EndpointPublishingStrategy.Type == operatorv1.HostNetworkStrategyType {
env = append(env, corev1.EnvVar{Name: "ROUTER_IP_V4_V6_MODE", Value: "v4v6"})
// Add the following environment variables only routers serving user applications
if ci.Name != "default" {
// Graceful shutdown (see
// amount of time router still accepts new connections when pod enters Terminating state
// we need to wait 2 minutes before we can expect clients to stop connecting to a router pod
// shutting down:
// - external-dns updates the DNS every 60 seconds
// - the DNS record TTL is 60 seconds
env = append(env, corev1.EnvVar{Name: "ROUTER_GRACEFUL_SHUTDOWN_DELAY", Value: "120s"})
// amount of time the router gives existing active connection to terminate gracefully after router
// stops accepting new connections. The router shuts down earlier if all connections terminated,
// or terminated remaining connections at the end of this delay.
// NB: this cannot be more than 1h because the ingress-operator hardcodes a terminationGracePeriod
// of 3600 seconds for router pods.
// unlike the others, this value must be a raw number of seconds (e.g. 30).
env = append(env, corev1.EnvVar{Name: "ROUTER_MAX_SHUTDOWN_TIMEOUT", Value: "900"})
// Some clients are slow to send the request, e.g. SVN.
// The OKD4 defaults are too aggressive.
// Values determined from years of operating Openshift 3
env = append(env, corev1.EnvVar{Name: "ROUTER_DEFAULT_CLIENT_TIMEOUT", Value: "60s"})
env = append(env, corev1.EnvVar{Name: "ROUTER_SLOWLORIS_TIMEOUT", Value: "60s"})
// Enable prometheus metrics
certsSecretName := fmt.Sprintf("router-metrics-certs-%s", ci.Name)
certsVolumeName := "metrics-certs"
......@@ -243,8 +243,13 @@ func desiredLoadBalancerService(ci *operatorv1.IngressController, deploymentRef
service.Annotations = map[string]string{}
// Add Openstack- and CERN-specific annotations to the Loadbalancer service
if proxyNeeded {
service.Annotations[awsLBProxyProtocolAnnotation] = "*"
// Enable the PROXY protocol on the side of the Openstack cloud loadbalancer
service.Annotations[""] = "true"
if platform != nil {
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment