Test registry-next upgrade

Under Epic let's first test the upgrade with registry-next.cern.ch to v2.11.1.

• confirm backup and recovery after the upgrade
  • all the images and charts are available
• take DBoD backup and confirm that metadata is properly restored
  • projects, users, roles, replication policies, tag retention policies, scanners, charts, and images metadata

Here, we can also confirm the recovery of the registry.cern.ch s3 backup.

assigned to @akothiwa

added to epic &3 (closed)

mentioned in merge request !94 (closed)

changed iteration to Kubernetes Sprints Nov 13, 2024 - Nov 26, 2024

@akothiwa: This issue is awaiting triage, we'll update it soon.

added triageneeds-attention label

removed triageneeds-attention label

added A-registry p2 labels

mentioned in merge request !112 (closed)

mentioned in merge request !114 (merged)

mentioned in merge request !115 (closed)

Upgrade to 2.9.5 for harbor core

2024-11-21T13:47:59Z [INFO] [/common/dao/base.go:72]: Register database completed
2024-11-21T13:48:00Z [DEBUG] [/migration/migration.go:37]: current database schema version: 100
2024-11-21T13:48:00Z [INFO] [/common/dao/pgsql.go:135]: Upgrading schema for pgsql ...
2024-11-21T13:48:00Z [INFO] [/vendor/github.com/golang-migrate/migrate/v4/migrate.go:879]: Start buffering 110/u 2.8.0_schema
2024-11-21T13:48:00Z [INFO] [/vendor/github.com/golang-migrate/migrate/v4/migrate.go:879]: Start buffering 111/u 2.8.1_schema
2024-11-21T13:48:00Z [INFO] [/vendor/github.com/golang-migrate/migrate/v4/migrate.go:879]: Start buffering 120/u 2.9.0_schema
2024-11-21T13:48:00Z [INFO] [/vendor/github.com/golang-migrate/migrate/v4/migrate.go:745]: Read and execute 110/u 2.8.0_schema
2024-11-21T13:48:01Z [INFO] [/vendor/github.com/golang-migrate/migrate/v4/migrate.go:763]: Finished 110/u 2.8.0_schema (read 33.973093ms, ran 875.928951ms)
2024-11-21T13:48:01Z [INFO] [/vendor/github.com/golang-migrate/migrate/v4/migrate.go:745]: Read and execute 111/u 2.8.1_schema
2024-11-21T13:48:01Z [INFO] [/vendor/github.com/golang-migrate/migrate/v4/migrate.go:763]: Finished 111/u 2.8.1_schema (read 931.255487ms, ran 100.59561ms)
2024-11-21T13:48:01Z [INFO] [/vendor/github.com/golang-migrate/migrate/v4/migrate.go:745]: Read and execute 120/u 2.9.0_schema
2024-11-21T13:48:02Z [INFO] [/vendor/github.com/golang-migrate/migrate/v4/migrate.go:763]: Finished 120/u 2.9.0_schema (read 1.054286742s, ran 809.639794ms)
2024-11-21T13:48:02Z [INFO] [/vendor/github.com/golang-migrate/migrate/v4/migrate.go:199]: Closing source and database
2024-11-21T13:48:02Z [INFO] [/vendor/github.com/golang-migrate/migrate/v4/migrate.go:199]: Closing source and database
2024-11-21T13:48:02Z [INFO] [/core/main.go:197]: The database has been migrated successfully

Issues found:

1. We were not admisn in registry-next after the upgrade. (unclear if we were before)
2. docker.io/library/python was showing 23 pulls by 0 artifacts. We need to make sure that all proxy-cache artifacts will be preserved.

Notes:

1. If redis is moved to 2.9.5, then moving back to 2.7.x won't work since the schema of redis will change. (Do we need to move to another HA redis or maybe just remove peristence on redis?)

Download docker.io in the proxy cache: for i in memcached nginx httpd busybox alpine ubuntu redis postgres python node mongo mysql rabbitmq traefik docker hello-world mariadb openjdk golang registry ruby wordpress debian php centos ; do skopeo copy docker://registry-next.cern.ch/docker.io/library/$i oci://data/foo ; rm -rf /data/foo; done

for i in memcached nginx httpd busybox alpine ubuntu redis postgres python node mongo mysql rabbitmq traefik docker hello-world mariadb openjdk golang registry ruby wordpress debian php centos ; do docker pull registry-next.cern.ch/docker.io/library/$i ; docker system prune -a -f; done

We were not admin in registry-next after the upgrade. (unclear if we were before)

The issue here was not due to the upgrade. The user rules are directly populated from OIDC. I forgot to add the harbor-admin role during the OIDC creation and hence there were no user admin roles even before the upgrade. But before the upgrade, I manually added myself as admin and this was in place after the upgrade too. We can verify this when we upgrade it again.

assigned to @strigazi

changed iteration to Kubernetes Sprints Nov 27, 2024 - Dec 10, 2024

@rbritoda We got the permission to ready groups for registry next, but it looks like this is not the way to get admin access based on an egroup. Do you know how is it done?

https://cern.service-now.com/service-portal?id=ticket&table=u_request_fulfillment&n=RQF2930770 https://auth.docs.cern.ch/applications/group-based-permissions/

Things to verify after upgrading:

1. number of proxy cache artifacts in docker.io/python
   • take screenshot before upgrading
   • 
2. quota set to 99GB for the kuberntes project
3. kubernetes-developers is a maintainer in the kubernetes project
4. k8s robot accounts can push/pull in the private kubernetes-private project and push to the kubernetes public project.

• robot-kubernetes+k8s in kubernetes
• robot-kubernetes-private+k8s in kubernetes-private

5. chart gallery has the the releases/cern-magnum-0.13.2.tgz

[strigazi@aiadm08 s3cmd-profiles]$ helm  repo list  | grep releases
releases            	https://registry.cern.ch/chartrepo/releases       
[strigazi@aiadm08 s3cmd-profiles]$ helm  search repo cern-magnum
NAME                	CHART VERSION	APP VERSION	DESCRIPTION                                
releases/cern-magnum	0.13.2       	           	A Helm chart for the CERN Magnum deployment

6. artefacts in registry.cern.ch/kubernetes-private/hyperkube
7. vulns in registry.cern.ch/kubernetes-private/hyperkube
   • same as 6
8. Speed of opening the UI for docker.io/library/python
   • https://registry-next.cern.ch/api/v2.0/projects/docker.io/repositories/library%252Fpython/artifacts?with_tag=false&with_scan_overview=true&with_label=true&with_accessory=false&page_size=15&page=1
   • Not reproducible, when opening in the morning (no one was using registry-next at night), opens in 2.56s, yesterday aftenoon it was slower, in prod takes 38.47s again in the morning.
9. Verify replication event-based replication rule works from registry-next.cern.ch/kubernetes-next/ to registry-staging.cern.ch/kubernetes-next/

Replicating data with rclone to another bucket:

rclone sync s3-cloud-infra-services://registry-next s3-kubernetes-developers://registry-next-bucket \
--config rclone.conf \
--bwlimit=160M \
--transfers=4 \
--s3-chunk-size=64M \
--s3-upload-concurrency=4 \
--buffer-size=500M \
--checkers=16 \
--progress \
--stats=60s \
--retries=3 \
--inplace \
--fast-list \
--verbose

registry-next-dump.sh

#!/bin/bash

HOST="dbod-harbordb-next.cern.ch"
PORT="6610"
DATABASE="registry_next"
USER="admin"
DUMP_FILE="registry_next.sql"

export PGPASSWORD="$(tbag show --hg cloud_container  registry-next-dbod-admin-pass | jq .secret -r)"

pg_dump -h $HOST -p $PORT -U $USER -d $DATABASE  -f $DUMP_FILE
unset PGPASSWORD

registry-next-import.sh

#!/bin/bash

HOST="dbod-harbordb-next-clone-20241121112235.cern.ch"
PORT="6628"
DATABASE="registry_next"
USER="admin"
DUMP_FILE="registry_next.sql"

export PGPASSWORD="$(tbag show --hg cloud_container  registry-next-dbod-admin-pass | jq .secret -r)"

psql -h "$HOST" -p "$PORT" -U "$USER" -d "$DATABASE" -f "$DUMP_FILE"
unset PGPASSWORD

mentioned in merge request !119 (merged)

mentioned in merge request !120 (merged)

Checked everything from the comment above. Looks ok. Moving to 2.11.2

Things look ok for 2.11.2 as well. Speed of docker.io/library/python is under 2s, but these are new pods so it's not clear if there is really an effect.

Some things to check:

• next shows 'Login with keycloak', the current prod 'Login with OIDC provider'. Why the difference?
• can we drop the 'Login with Local DB' from the web interface? This was something diogo was asking for upstream long ago and security has asked us about it

Let's close this, next is in 2.11.2.

closed