Repository "Storage" is huge due to CI "Job Artifacts"

We agreed, in our DD weekly meeting, to add an expiry policy of "1 day" to all artifacts, to keep the CI storage space clean.

Implemented in commit: fdafc154

mentioned in commit fdafc154

After having set the expiry of all artifacts to 1day, we now observe artifacts available to download on the "latest" pipeline only:

This is the artifacts from the "latest" pipeline:

And this is the artifacts list from another pipeline, it is now empty:

This is due to the CI setting:

Now, all artifacts but the latest are deleted. However, we should set the CI to keep the artifacts from 'tags'.

One solution I found is from this SO answer: https://stackoverflow.com/a/65478446

persist-tag-artifacts:
  # Use this job to specify artifacts that should never expire for tag pipelines.
  stage: post-deploy
  rules:
    - if: '$CI_COMMIT_TAG != null'
  script:
    - echo "Persisting artifacts for tag pipelines"
  artifacts:
    # Specify artifacts from previous jobs to be persisted indefinitely
    paths:
      - myArtifactsPathFromJob1
      - myArtifactsPathFromJob2
      - ...
    expire_in: never

I will test this in our CI configuration.

OK, I created another stage and added this new job in this commit: 8519f358

The first test is fine: the post-deploy stage is not triggered at all by a standard commit, that is no tag:

But creating a new tag triggers the post-deploy stage correctly; see the additional stage in the latest pipeline, which is a tag pipeline, compared to the previous one on the same commit:

I now created a dummy commit, which triggered a standard pipeline, that is without the additional post-deploy stage:

When the latest pipeline succeeds (6185279), the pipeline before the tag (6185138) should delete its artifacts tomorrow based on our current expiry policy. But, if the new post-deploy job works correctly, the tag pipeline (6185215) should keep its artifacts.

I'll update the issue tomorrow, after the 24 hours of our expiry policy.

mentioned in issue #89

OK, after > 24h, now the artifacts are available only for the latest standard pipeline and the tag pipeline. The other pipelines have no artifacts to be downloaded.

I'm now writing a Python script to clean the old jobs, with a filter on the tag variable.

All jobs will be deleted except those whose tag variable is set to True.

I use the GitLab erase API: https://docs.gitlab.com/ee/api/jobs.html#erase-a-job

After running the script with a read_api access token (read-only), I run it with a full api token on a subset of jobs, and it successfully erases the jobs, that is it deletes all artifacts and logs related to a job.

This pipeline was the latest one before setting the expiry policy and had all artifacts. Now, it shows "no artifacts":

Also, it successfully keeps the artifacts of the jobs belonging to the "latest" pipeline:

and the ones belonging to a "tag" pipeline:

So, I can now run the script on all jobs. I will let run it overnight and I'll update this page after it is done.

This is the Python script I wrote to clean the GeoModel repository:

https://gitlab.cern.ch/-/snippets/2794

I run the script on all jobs.

From some of the jobs, I get a 403 error; I have to understand why. For those jobs, the artifacts have not been cleaned.

But for all the others I got a 201 success return code, and the artifacts and logs have been erased.

Currently, the project's storage has gone down from ~1.1 TB to ~209 GB!!

Today, @tsulaia pointed out that the size has increased already, from about 200 GB to about 240 GB. So, about 40 GB in about ten days or so.

So, I checked and found that, apparently, the artifacts on the new pipelines are erased correctly following our new rules:

only tags and the "latest" build on main are kept forever
the artifacts from all other jobs are erased after one day

In fact, at the moment, the artifacts from the "latest" are there:

As well as the "latest" MR:

And the tags, for instance 4.6.0:

All other artifacts look as they have been cleaned correctly:

So, the increase should come from the size of our artifacts, much probably due to the Geant4 data included in them.

==> I'll check the size of them.

UPDATE: I investigated the increase in size observed by @tsulaia (summarized here)

Actually, each job produces a bundle of artifacts that goes from about 1 MB to about 1.3 GB. The difference is in the dependency on Geant4. When a job depends on Geant4, the /install/ folder from the Geant4 build job is picked, used, and bundled; which results in a huge, final artifact bundle.

I summed up all jobs from a single pipeline: the result is that each pipeline produces a total amount of about 14.3 GB of artifacts:

In the last 10 days, we had two pipelines whose artifacts were correctly preserved because of our new rules/policy:

The latest build in main: https://gitlab.cern.ch/GeoModelDev/GeoModel/-/pipelines/6262913
The build of the 4.6.0 tag: https://gitlab.cern.ch/GeoModelDev/GeoModel/-/pipelines/6246351

Plus, I noticed that other two pipelines have been preserved as well:

A build of the "latest" commit in the open MR 209: - The latest MR: https://gitlab.cern.ch/GeoModelDev/GeoModel/-/pipelines/6262983
A build of the "latest" commit in the open MR 206: https://gitlab.cern.ch/GeoModelDev/GeoModel/-/pipelines/6245673

So, the total amount of those sums up to about: 14.3 GB x 4 = 57.2 GB, which is compatible with the size increase that @tsulaia observed in the last 10 days.

==> So, that is understood.

However...

While the preservation of the first two preserved pipelines is correct in respect to our rules, the second two pipelines are think they have been preserved because they are flagged as the "latest" builds of open MRs. So, in principle, I think they should be deleted automatically after the MR is closed. But we should take an eye on that and check the successful deletion.

Update: the size went down due to the merge of a couple of MRs, as expected

Therefore, I think that we now correctly understand the behavior of our CI and the size of the footprint of our project.

closed

Repository "Storage" is huge due to CI "Job Artifacts"

Designs

Child items ...

Activity