Improve release cycle to allow for faster releases

As discussed, we should aim to do more frequent releases of CTA. To do this, we should have a look at the full process of creating a new CTA release and what we can do there to automate things. This ticket is meant as a general overview of the release process and the various improvements to be made to it to ensure the effort to create a new release is minimal.

Currently, the process is follows:

1. Update the changelog
   1. Manually Trigger the update-changelog job in the main branch
   2. Manually review the entries, fix mistakes and merge the MR created by the job
2. Create a tag
   1. Determine what the new version number should be
   2. Manually create a tag
3. Run the stress test on the image created by the tag
   1. Ssh into stress test machine
   2. Run the instructions on https://gitlab.cern.ch/cta/sandbox/ci_monitoring/-/tree/master?ref_type=heads
   3. Wait for the stress test to complete
   4. Compare monitoring charts
4. Publish the RPMs to a repo
   1. Trigger the release job in the tag pipeline
   2. Create a GitLab release
5. Create a community forum post

Overall there are still a lot of time-consuming steps that prevent us from making these releases more frequent. Ideally, we should first aim to automate this as much as possible. The person doing the release should (for now) only have to do:

1. Manually trigger the changelog update job
2. Review the changelog. There are some improvements to be made here to make this check faster. E.g. checks on each MR to ensure the commits follow the correct format
3. Determine the new version number. It would be good to have another commit trailer saying something like version: major/minor/patch/packaging. The new release version can then be automatically determined by these trailers and the developer only has to look at the latest changelog entry header.
4. Create a tag. This should automatically trigger the stress test and determine whether the performance is good enough. This requires additional automation; we could have e.g. a stress-test job in the CI and have this be triggered on tagged pipelines and automatically compare performance against baseline somehow (this is a bit tricky as we currently compare graphs; TBD). If the test passes, it should automatically push to the repo and create a GitLab release
5. Create a community forum post. Perhaps this step can also be automated, but that has less priority as the cost/benefit ratio might not be too high

In large parts, we can divide this work in three parts:

• Improve changelog generation
  • Make improvements based on comments in last release: #996
  • Add automated checks to commits before merging to ensure commit template correctness
  • Add commit trailer to automatically determine new CTA version number. Add clear guidelines on when to use which trailer
  • Make the MR for the changelog update extremely quick so that it doesn't have to run the full pipeline
  • It would be nice if this can be included in the tag pipeline, but I don't really see how as it requires adding another commit
• Automate stress test running
  • Register stress test as ci-runner with custom tag
  • Add stress-test job to CI and automatically trigger this on tag pipeline
  • Add automated check to see if performance was good enough
• Automate release publishing
  • Add new unstable repo
  • Automatically push to unstable repo if stress-test passes
  • Automatically create a GitLab release entry (mention unstable)
  • Generate template for community forum post. If possible, do this automatically via the API: https://docs.discourse.org/#tag/Posts/operation/createTopicPostPM

The goal is to have all of this happen with minimal input from a developer. If the above works, the developer only has to trigger the changelog job, review the changelog and create a tag. This would cost < 5 min meaning we can do more frequent releases.

added Continuous Integration ci: pipeline ci: runners typeignore labels

assigned to @nbugel

assigned to @afonso

added Needs Discussion label

Looking at what other companies are doing to manage their releases (e.g. GitLab), it seems that a separate release repository would be a really nice way to tackle the problem. Let's call it e.g. cta-release-tools

Essentially it allows us to fully decouple the release procedure from the development workflow. We can have a pipeline in the release repository that takes care of all the necessary steps:

1. Updating the changelog MR (will most likely still need manual verification)
2. Running the stress test
3. Tagging the commit to the provided version (as we are now outside of the original repo, we can nicely automate this)
4. Publishing the RPMs to our RPM repos and to GitLab

To create a new release, first, a manual decision has to be made on what the new version should be (this is unavoidable; we need to know and verify whether it is a patch, minor or major version bump). Once this has been determined, a pipeline in cta-release-tools can be triggered with two input variables: the commit to tag and the version to tag it with. The pipeline will look something like this:

flowchart LR
  UC["update-changelog"]
  STRESS["stress-test"]
  READY["release-ready (manual)"]
  TAG["tag-cta"]
  RELEASE_UNSTABLE["publish-unstable"]
  RELEASE_TESTING["publish-testing (manual)"]
  RELEASE_STABLE["publish-stable (manual)"]

  UC --> READY
  STRESS --> READY
  READY --> TAG
  TAG --> RELEASE_UNSTABLE
  TAG --> RELEASE_TESTING
  TAG --> RELEASE_STABLE

A few notes:

• All job dependencies as seen above will require the previous jobs to have been successful.
• update-changelog will require someone to review and merge the generated MR containing the changelog updates. This ensures that the changelog updates are correct.
• stress-test will require some to assess whether the stress test was successful or not after it has passed (at least until we have more reliable metric to assess performance with)
• release-ready will check that CHANGELOG.md on main contains the changelog entries of the desired version
• tag-cta will create a tag of the desired commit in the CTA repository
• The other jobs are for publishing the RPMs to the RPM repos
• If we want to take this a step further, we can add another job that creates a release on GitLab. Additionally, we could also add a job that creates a post on the community forum. The question is when do we do this (e.g. for all releases or only for stable releases)?

The only thing to consider is which CTA RPMs to use. E.g. tagging will update the spec file with a changelog entry so we would need to rebuild CTA for that to go into effect. Perhaps this is not an issue, but it means that the RPMs that we are releasing and the RPMs that were (stress) tested are not 100% identical.

Perhaps we also need to retrigger a normal pipeline first on the commit we want to tag to ensure we have the RPM artefacts (seeing as artefacts expire, so we wouldn't be able to easily tag old CTA versions).

Maybe this can be done with 3 steps:

1. First pipeline includes all the steps up to (and including) release-ready. By the end, a developer can approve the MR with the Changelog. The MR would be flagged as "release_ready" (tbd, but maybe using one of these gitlab CI variables)
2. The MR of the Changelog would trigger automatically a pipeline on main. By checking that this is a "release_ready" commit, the CI is able to automatically tag the new commit and create the artefacts with the proper names.
3. Finally, the developer can decide to promote manually these artefacts to unstable -> testing -> stable, and do any announcements (automated or not).

Could you elaborate a bit on which problem this is trying to solve? Because generating new artefacts can also be done by an extra job after release-ready in the release pipeline (by simply triggering a pipeline on said commit). That way we don't end up with a fragmented release pipeline. The problem then remains though; these are not the artefacts we used to run the stress tests.

We could do

update-changelog -> release-ready (manual) -> trigger CTA main pipeline for artefacts -> stress test -> tag (manual) -> publish`

but then we have 2 manual steps. I would also prefer it if we could run the stress test first and then do the changelog update. That way if we find issues, we can address them without having to touch the changelog (stress test failed means no release, so we need to revert the changelog update we did).

Maybe something like this would be better:

trigger CTA main pipeline for artefacts + image -> stress test -> update-changelog (manual) -> release-ready (manual) -> tag -> publish`

I think this might be the way to go as it should address all of these problems. Once we have telemetry, we should be able to automate the decision on the stress test performance and we are back to 1 manually triggered job in the middle of the pipeline (which I think is unavoidable)

In this case I was indeed considering that the stress tests would have been run before the changelog commit. As long as we are only changing text files in the new commit, then the logic would be virtually the same as the previous one.

It seems a bit of a chicken and egg problem...

---

trigger CTA main pipeline for artefacts + image -> stress test -> update-changelog (manual) -> release-ready (manual) -> tag -> publish`

This aligns with what I had in mind. But maybe the "release-ready"/"tag" can be triggered automatically by the (manual) merge of the "update-changelog" (if this is possible). This way we only need 2 manual steps (triggering the release pipeline and approving the MR).

The step "publish" would be automatic only for "unstable".

It is indeed a bit of a chicken and egg problem... I'll need to think a bit about how we can do this cleanly

One thing we need to take care of is commits added to main after the changelog MR was created, but before the changelog MR was merged. I have some ideas for this, but let's discuss it in person.

removed Needs Discussion label

created branch 1017-improve-release-cycle-to-allow-for-faster-releases to address this issue

mentioned in merge request !829 (merged)

This first MR adds the mechanism for releasing the CTA RPMs to unstable:

• !829 (merged)

It also slightly changes the source of the RPMs for stable, testing and unstable. Instead of always getting them from the gitlab CI artifacts, it copies the RPMs from the internal cta-repo.

This way, we won't have to face the risk of the CI artifacts expiring after a certain period of time.

Some future changes to consider:

• Instead of copying the files from cta-repo to stable/testing/unstable, create hardlinks inside of EOS. This cannot be done with xrdcp, but can be done directly on the filesystem mounted with fuse.
• Modify update_repos.sh so that it can run on each sub-repo independently. This is low priority, as update_repos.sh should be idempontent.

mentioned in commit 480988fe

mentioned in merge request !844 (merged)

mentioned in commit b13a1c58