Review CTA CI workflow

The current CI workflow is complex and not very flexible.
The Kubernetes environment is setup with cumbersome commands and not in a "standard" way.
The current CI does not allow to run parallels pipelines with different software environments.
By example it would be good to be able to test the same code with (most urgent first):

1. xrootd5
2. eos5
3. Alma9
4. Custom dev environment (by example with the new postgres library)

This is a prerequisite before adding new type of tests to the CI, so avoid doing them twice.

assigned to @rbachman

added Continuous integration in CTA-old label

assigned to @agounon

changed title from Review CTA CI worflow to Review CTA CI workflow

changed the description

added workflowAssigned in CTA-old label

After discussion some more info about the steps to follow:

1. First concentrate on kubernetes deployment:

• kubeadm - podman - calico?
• use base install from puppet class cta/dev/ci_mini.pp
• how to deal with firewalling (puppet and k8s both want to manage it)?

2. mhvtl deployment:

• not managed by puppet, but rather scripted (in what is currently called buildtree)
• can it be run in a container?

unassigned @agounon

added workflowAccepted in CTA-old label and removed workflowAssigned in CTA-old label

assigned to @rbachman

added workflowAssigned in CTA-old label and removed workflowAccepted in CTA-old label

added Continuous Integration + 1 deleted label

Looks like we need to import the branch associated to this issue from CTA-old repo: https://gitlab.cern.ch/cta/CTA-old/-/tree/ctaci

In addition there are some modifications on ctaci01 that have not been pushed yet to the old repository and are spread a bit on that VM: some in the local /root/CTA local clone of CTA-old repo and some in /root folder.

We need to sort out what is the most up to date information and add all this in the branch associated to this ticket.

Looks like the tools are converging toward kubeadm, with calico and CRIO, would be good to have an option to use podman too.

What about minikube for the full install of kubernetes?

Regarding mhvtl, we need more packages on our side: we have a specific version of mhvtl with removed fsync (check with @smurray) that we must be able to build in CI and distribute.

I have migrated Aurelien's branch to the new repo: https://gitlab.cern.ch/cta/CTA/-/tree/ctaci We can add new changes here.

added workflowIn Progress label and removed 1 deleted label

Some notes from what I tried yesterday:

Some notes

All package installation instructions are writing repo files for CentOS in /etc/yum.repos.d. On puppetized machines we use /etc/yum-puppet.repos.d but adding a repo file here means that puppet will just delete it. Repo files can stay in /etc/yum.repos.d, so that they are safe, then to install rpms from there using yum: just use a specific yum.conf that has the following line:

reposdir=/etc/yum.repos.d/

Just type yum -c yum.conf ...

Install cri-o

cri-o rpm is needed as well as cri-tools that are a requirement for kubeadm rpm. Installation instructions are interesting:

• cri-o rpm must have the same version as the kubernetes rpms: kubernetes 1.22-XX requires cri-o 1.22 (release number is not a big deal I guess).
• cri-tools does not seem to follow the same logic and I got 1.25 which was the latest greatest and kubeadm seems to be happy with that.

Here is a small snippet to install cri-o 1.22 on CentOS7:

OS=CentOS_7
VERSION=1.22 # kubernetes version
curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable.repo https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/devel:kubic:libcontainers:stable.repo
curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable:cri-o:$VERSION.repo https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable:cri-o:$VERSION/$OS/devel:kubic:libcontainers:stable:cri-o:$VERSION.repo
yum -c yum.conf install cri-o

It looks like cri-o is not yet available for all versions of kubernetes, therefore check available cri-o before looking for kubernetes.

I could not find cri-o-1.25 and wasted time installing kubernetes 1.25, so better check cri-o before.

Install kubernetes

Add the repos:

cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-\$basearch
enabled=1
gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kubelet kubeadm kubectl
EOF

We need to add --disableexcludes=kubernetes in all yum commands to see kubernetes packages...

Get the latest kubernetes version for your version of cri-o:

Where version is extracted from:

yum -c yum.conf search kubeadm --disableexcludes=kubernetes --showduplicates | grep 1.22 | tail -1
kubeadm-1.22.15-0.x86_64 : Command-line utility for administering a Kubernetes

Now you need to install kube rpms 1.22.15-0:

K8VERSION=1.22.15-0
yum install -c yum.conf --disableexcludes=kubernetes -y kubelet-$K8VERSION kubeadm-$K8VERSION kubectl-$K8VERSION

Looks like hostgroup cta/dev/ci_mini and environment cta_ci is working fine to install cri-o-1.22.5-2.3.el7.x86_64 along with kubernetes 1.22.6-0.

coredns issues

There is still one problem:

cat /etc/resolv.conf 
# Generated by NetworkManager
search cern.ch
nameserver 127.0.0.1

Caused an infinite loop in coredns containers. Checking that we have at least one non loopback valid DNS to forward requests is really needed.

mentioned in issue #171 (closed)