Buildtree replacement proposal

Problem to solve

buildtree was introduce to ease CTA development and testing in locally running CI. The idea is that the developers compile CTA code and buildtree will populate CTA binaries by creating plenty of symlinks to the developer generated binaries and libraries.

double buildtree was introduced later to allow the same symlink based workflow this time for EOS4 development. This second step doubled the complexity of the single buildtree workflow.

The current issue is that adapting double buildtree for EOS5 explodes again the complexity and this approach won't scale any further: think about possible restic developments or dCache.

In addition buildtree adds complexity in the CI containers definition: adding conditions in the container scripts that complicate CI code modifications and testing. Indeed CI developers are not likely to test buildtree and even less likely to test double buildtree environments...

Stakeholders

CTA developers, EOS developers developing some tape specific code they need to test, dcache developers...

Proposal

We need to remove the buildtree approach and replace it with something that is less intrusive for the main CI code.

One simple way to replicate this is to move developers workflow to one that is closer to CI:

1. change source and compile code
2. generate and publish rpms
3. instantiate a local CI kubernetes namespace that will consume these rpms

Here is a mock up of the proposal:

All this can run on a laptop or any openstack dev VM: the developer should only interact with the local folder with his favorite editor. Everything else should ideally run inside containers: CTA and EOS compilation are taking place in gitlab inside containers.

added Needs Discussion label

added priorityhigh label

added 1 deleted label

assigned to @jleduc and @jcamarer

added workflowIn Progress label and removed 1 deleted label

assigned to @rbachman

marked this issue as related to #364 (closed)

I'd suggest we try out an nginx-less solution first. With Nginx we risk ending up with developers having to debug an enginx deployment when something is going wrong with new rpm versions.

In k8s we can define host-local directories to mount as volumes in the pods:

volumes:
- name: data-volume
  hostPath:
    path: /code/bin/rpm
    type: Directory

These we could use both on local dev machines and in CI (where the rpms should be available as artifacts on the runner in the various CI stages).

In addition we could supply logic re. how to treat a local repo with recards to remote repos (priorities and such) by using configmaps. These can then be built on in future refinements of the CI.

mentioned in issue #364 (closed)

added Building and Packaging label

added Continuous Integration label

changed milestone to %CI update push

mentioned in issue #386 (closed)

marked this issue as related to #386 (closed)

removed Needs Discussion label

This is a big improvement on the buildtree. @jcamarer made an important improvement now that RPMs can be built without having to repeat the compilation step.

However, the question raised by @rdm yesterday should be addressed: how can developers attach a debugger to a process running inside a pod?

About the question, always you can enter in the pod, stop the main process and relaunch it using gdb. I think @afonso has some experience doing this.

closed

This issue has been completed in this MR !246 (merged)