Draft: CI: use 'workflow' to avoid duplicate pipelines

In our CI setup, duplicate pipelines are started when pushing to a branch that is attached to an open Merge Request.

For example:

This happens because when a MR is created, a new commit to the attached branch triggers one pipeline for the open MR and one for the branch itself.

The CERN GitLab team suggested me to take a look at those pages:

• https://docs.gitlab.com/ee/ci/pipelines/merge_request_pipelines.html#two-pipelines-when-pushing-to-a-branch
• https://docs.gitlab.com/ee/ci/jobs/job_control.html#avoid-duplicate-pipelines

In this MR, I will try to configure our CI to avoid duplicate pipelines.

added Continuos Integration label

assigned to @rbianchi

added 1 commit

• 82cbeedf - Fix syntax: 'workflow' is global per pipeline, not per job

Compare with previous version

added 1 commit

• fa5bde9f - dummy commit

Compare with previous version

added 1 commit

• d30fba5d - revert

Compare with previous version

Hi @rbianchi, this looks like a dummy MR. Do we still need it? Thanks.

Yes, we still need this. It's a branch where I plan to test things and I'm trying to apply the workflow rules in our GitLab CI configuration to avoid duplicate pipelines.

Currently, in fact, two pipelines are triggered when a commit is pushed to a branch that is also part of an open MR. That is standard plain CI setup in GitLab, but it is not needed and increases the number of jobs running. So, some tweaking to our CI setup is needed to avoid that.

This is part of the work to avoid unnecessary CI jobs being triggered by commits and MRs, which is part of the effort to reduce the overall running jobs of the GeoModel CI.

The work is done not to hit the "200 concurrent running jobs" set by the CERN IT team. That should fix the issue of pipelines not being started that we are observing in the last weeks; for example

The pipeline in the image was not started at all, and if we hover on the error label we can see the message:

The CERN GitLab team told me that that is the sign that, at that particular time, we had too many running jobs and we hit the 200 jobs limit; for that reason, that particular pipeline was not started.

So, in this branch, I will try to reduce the number of jobs and to remove the duplicate pipelines.

P.S. Since it is still WIP, I just marked it as a Draft, so it's more clear.

marked this merge request as draft

Hi @rbianchi, thanks for the feedback! OK, let's keep it open then.

Hi @rbianchi, do we still need this MR? I'm OK with keeping it open if still required. Thanks!

Hi @rbianchi, the same question as the one I posted in November: do we still need to keep this MR open? Thanks.

closed

reopened