Migrating from DrupalLB to LBaaS
Due to several reasons, we are now planning a smooth transition from the drupal loadbalancers to LBaaS.
The current setup has the follow:
- One static IPv4 and IPv6 (188.184.37.219 and 2001:1458:201:b0::100:1b)
- One machine handling the loadbalancing (Currently
drupal8p11
, but due to floating IP, a new machine can assume the work once this one fails the liveness probing)
Thus, the requirement for IPv6 and static IPs were a must when looking into LBaaS.
Since the requirements (Doc with initial discussion) are met, the following is a proposed set of actions to do a transparent migration to the new LBs.
Preparation and Requirements
-
Meet with the LBaaS team and assess the state of the service and load handling -
Prepare inventory of all domains and redirections served in DrupalLB domains-oldinfra.txt -
Prepare inventory of domains already served in paas and drupal -
Investigate if all redirections are supported on new infra. Plan implementation of redirections if needed -> Proposed solution for redirections: https://gitlab.cern.ch/webservices/web-redirector-v2/ -
Prepare plan for move of domains: configuration of domains and redirections manual or automated? Coordination of DNS with Veronique needed? Batches? Dates. Devise procedure to validate, monitoring. Devise steps to rollback in case of need, shorter TTLs needed during migration in case of rollback needed? -
Standardise TTL of .cern domains with Veronique: 5 minutes -
Configure new LBaaS for Drupal -
Create a new Octavia LoadBalancers (v2) in the critical area, to be used by all '.cern' projects and with custom domains
Steps described here: https://okd-internal.docs.cern.ch/components/cloud-controller/ -
Add it to the configuration Similar configuration required after: https://gitlab.cern.ch/paas-tools/okd4-install/-/blob/master/chart/values-paas.yaml?ref_type=heads#L338-L359 -
Create a new project using the new LBaaS instance -> URL: fborgesa-newlb.web.cern.ch -
Stress test the new project, confirm load capacity
-
-
Have a failover request while already handling some .cern traffic (before home.cern) and coordinate with the LBaaS team -
Test, document and validate the LBaaS failover mechanism -
Create OTG for dates decided -
Have home.cern and others with the annotation containing all necessary router labels (Updated due to new feature: OTG0149529) -
Update namespaces with apps-lb-crit
value once the change has been done fully -
Update documentation: https://gitlab.cern.ch/paas-tools/okd4-install/-/merge_requests/1314
*
Reason why we will have temporarily all
annotation of loadbalancing:
flowchart TD;
A[User] --> B[Old LB];
B --> C[Apps-Shard-1];
A --> D[Apps-lb-crit];
We currently have the flow on the left (through old LB infra), and we want the final state to be the one on the right (directly to apps-lb-crit). But due to DNS update time, for a while both flows will have to work, which means the namespace has to accept requests coming from both apps-shard-
and apps-lb-crit
. This is the point in time we use all
, so that during the transition period both flows will work for the users.
Procedure
Once the setup is ready, request change of DNS records for the multiple .cern
domains, preferably leaving home.cern
for last in case a rollback is required.
We should talk directly with Veronique to coordinate.
We use the move as an opportunity to standardize DNS TTL records and request lower times (currently it can be up to 12h to update DNS records globally)
During this period, all that is required is monitoring and validating that traffic is not affected.
Prepare project to receive traffic
# Allow namespace to receive traffic from new LB
oc label ns/<NAMESPACE> ingress-controller.okd.cern.ch/apps-lb-crit="true"
# Validate the URL is serving correctly now from the new LB
curl -H "Host: <DOMAIN>" http://drupal-apps-lb-crit.cern.ch -L -o /tmp/test.html --silent -w '%{url_effective} %{http_code}'
### If the code is 200 -> Works
### If the code is 404 -> Does not work
### Any other code, please report as it is not expected
.web.cern.ch
DNS records
Update .cern
DNS records are going to be updated upon request by our colleagues, but .web.cern.ch
are managed by us and require our action.
To have the route update an existing DNS record, the following command must be done for the project being migrated:
oc annotate route -n $PROJECT --all external-dns.alpha.kubernetes.io/target=drupal-apps-lb-crit.cern.ch --overwrite
Status
Excel sheet with all domains from the old infrastructure, and it's state/proposed state
Cleanup
Delete the following projects and their redirections:
- aliceinfo.cern.ch (Drupal-lb-1), was migrated to the LBaaS, using the old flavor LB
- cixp-redirect (paas), currently serving www.cixp.net, and migrate the DNS record to the new LBaaS
- clicdp.cern.ch (paas), was on Drupal LBs, currently on PaaS
- oqi-cern (drupal), currently handles oqi.cern redirection, WordPress instance and with cloudflare config
- www.ippog.org (None), this was handled by Drupal LBs, no longer registered
[More instances to be added, all mentioned on old Drupal LB covered]
Notes from discussion about design: https://docs.google.com/document/d/1lKyuafnnFKP5Yd_GQE8cx9QFamMa4g3IV27byKGhsCE/edit?usp=sharing
Timeline
Time | Step |
---|---|
February | Start LBaaS migration plan |
First half of March | Discuss with LBaaS team and communicate plan |
Second half of March | Provision new LB instance |
April | Start pointing cern domains onto it |
April-May | Full migration of dot cern domains |
May-June | Decomission of old infrastructur |