hep-workloads merge requestshttps://gitlab.cern.ch/hep-benchmarks/hep-workloads/-/merge_requests2019-01-22T17:56:14+01:00https://gitlab.cern.ch/hep-benchmarks/hep-workloads/-/merge_requests/21WIP: Add nano to the LHCb containers2019-01-22T17:56:14+01:00Domenico GiordanoWIP: Add nano to the LHCb containershttps://gitlab.cern.ch/hep-benchmarks/hep-workloads/-/merge_requests/22Add millisecond granularity to Gaudi message service timestamps2019-01-29T17:16:08+01:00Domenico GiordanoAdd millisecond granularity to Gaudi message service timestampshttps://gitlab.cern.ch/hep-benchmarks/hep-workloads/-/merge_requests/23Many fixes and improvements. Replace hardcoded "giordano" by fork name.2019-01-29T17:14:50+01:00Domenico GiordanoMany fixes and improvements. Replace hardcoded "giordano" by fork name.This MR includes many fixes and improvements
- It replaces "giordano" by the fork name in several scripts (and in the file with the command to do a local build).
- It improves the scripts to clear images and containers.
- It improves ...This MR includes many fixes and improvements
- It replaces "giordano" by the fork name in several scripts (and in the file with the command to do a local build).
- It improves the scripts to clear images and containers.
- It improves error handling in several places, ensuring early failure/exit in case of errors.
https://gitlab.cern.ch/hep-benchmarks/hep-workloads/-/merge_requests/24Add millisecond printout to master2019-01-30T12:07:41+01:00Domenico GiordanoAdd millisecond printout to masterCiao Domenico, as discussed.
This includes also in master the millisecond printout that is only in building.
AndreaCiao Domenico, as discussed.
This includes also in master the millisecond printout that is only in building.
Andreahttps://gitlab.cern.ch/hep-benchmarks/hep-workloads/-/merge_requests/25Include Andrea's many changes also in building2019-01-30T12:08:59+01:00Domenico GiordanoInclude Andrea's many changes also in buildingCiao Domenico, as discussed.
This includes also in building the many changes that I incorrectly submitted to master.
I need this because like this I can send you further MRs ONLY to building (but I need building to have all my previous c...Ciao Domenico, as discussed.
This includes also in building the many changes that I incorrectly submitted to master.
I need this because like this I can send you further MRs ONLY to building (but I need building to have all my previous changes).
Apologies for mixing this up!
Andreahttps://gitlab.cern.ch/hep-benchmarks/hep-workloads/-/merge_requests/26New set of fixes2019-02-05T23:53:39+01:00Domenico GiordanoNew set of fixeshttps://gitlab.cern.ch/hep-benchmarks/hep-workloads/-/merge_requests/27Work changes2019-02-05T16:57:50+01:00Domenico GiordanoWork changeshttps://gitlab.cern.ch/hep-benchmarks/hep-workloads/-/merge_requests/28Cosmetic changes in main.sh2019-02-05T23:52:44+01:00Domenico GiordanoCosmetic changes in main.shCiao Domenico,
these look like very many changes in the file, but actually it's a simple tidying up of tabs and empty spaces (look at the detailed logs of the three commits).
I file this as a separate MR for clarity, but can you please a...Ciao Domenico,
these look like very many changes in the file, but actually it's a simple tidying up of tabs and empty spaces (look at the detailed logs of the three commits).
I file this as a separate MR for clarity, but can you please approve it relatively quickly?
Thanks
Andreahttps://gitlab.cern.ch/hep-benchmarks/hep-workloads/-/merge_requests/29align gitlab-ci to qa2019-02-06T10:48:00+01:00Domenico Giordanoalign gitlab-ci to qahttps://gitlab.cern.ch/hep-benchmarks/hep-workloads/-/merge_requests/30Building2019-02-06T11:26:49+01:00Domenico GiordanoBuildinghttps://gitlab.cern.ch/hep-benchmarks/hep-workloads/-/merge_requests/32Fix for LHCb script and cosmetic changes2019-02-07T11:52:57+01:00Domenico GiordanoFix for LHCb script and cosmetic changesCiao Domenico,
this includes the LHCb fix we discussed this morning (return 0 if the LHCb app is successful) and a few harmless cosmetic changes.
You can apply this straight away without worries, I think - and It should make your pipelin...Ciao Domenico,
this includes the LHCb fix we discussed this morning (return 0 if the LHCb app is successful) and a few harmless cosmetic changes.
You can apply this straight away without worries, I think - and It should make your pipelines green.
Then I will submit another MR with "heavier" changes.
Andreahttps://gitlab.cern.ch/hep-benchmarks/hep-workloads/-/merge_requests/33YAML fixes requiring TOML changes2019-02-07T15:12:26+01:00Domenico GiordanoYAML fixes requiring TOML changesCiao Domenico,
this MR is only about the YAML file, but it requires that you also change the TOML and CI runner accordingly:
- This builds the builder from scratch using ci-tools, rather than using the builder itself (which would be ci...Ciao Domenico,
this MR is only about the YAML file, but it requires that you also change the TOML and CI runner accordingly:
- This builds the builder from scratch using ci-tools, rather than using the builder itself (which would be circular). If you do not want to enable shared runners (no need), you must do TWO things: first, in your TOML you need to allow the "gitlab-registry.cern.ch/ci-tools/docker-image-builder:latest" image, or even better set it as default; second, in the GitLab CI configuration, you must add the tag "docker-image-build" to the tags attached to your runners.
- This also sets inside the YAML the four variables that you were previously setting in the TOML. So you can actually safely remove them from the TOML (but you do not need to do that I think). I just find it much clearer that the code itself is self-readable (you know where variables are set, there is nothing "hidden" in the TOML).
- The TOML must also start builds in /scratch/builds and share /scratch with docker, as you suggested. This I did not change, but I added comments in the YAML to make this clearer.
For reference, the TOML I am using is
```
concurrent = 10
check_interval = 0
[session_server]
session_timeout = 3600
[[runners]]
name = "HEP workloads builder"
url = "https://gitlab.cern.ch/"
token = "zzzzzzzzzzzzzzzzzzzzzzzzzzzzzz"
builds_dir = "/scratch/builds"
executor = "docker"
[runners.docker]
tls_verify = false
image = "gitlab-registry.cern.ch/ci-tools/docker-image-builder:latest"
privileged = true
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/cache","/var/run/docker.sock:/var/run/docker.sock","/scratch:/scratch"]
shm_size = 0
allowed_images = ["gitlab-registry.cern.ch/ci-tools/docker-image-builder:latest", "gitlab-registry.cern.ch/hep-benchmarks/hep-workloads/*", "gitlab-registry.cern.ch/giordano/hep-workloads/*", "gitlab-registry.cern.ch/valassi/hep-workloads/*"]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
```
Cheers
Andreahttps://gitlab.cern.ch/hep-benchmarks/hep-workloads/-/merge_requests/34Cosmetic changes (indentation and white spaces)2019-02-13T10:22:56+01:00Domenico GiordanoCosmetic changes (indentation and white spaces)Ciao Domenico, these are purely cosmetic changes. No functional changes.
Can you please merge this rather rapidly?
Thanks AndreaCiao Domenico, these are purely cosmetic changes. No functional changes.
Can you please merge this rather rapidly?
Thanks Andreahttps://gitlab.cern.ch/hep-benchmarks/hep-workloads/-/merge_requests/35Several fixes (BMK-3 and others) and improvements2019-02-13T18:15:32+01:00Domenico GiordanoSeveral fixes (BMK-3 and others) and improvementsCiao Domenico, here's my next batch of fixes, including
- the bug fix for BMK-3 (truncated cvmfs trace file leading to missing json)
- several bug fixes in the return codes of functions (where errors did not cause immediate exit)
- the s...Ciao Domenico, here's my next batch of fixes, including
- the bug fix for BMK-3 (truncated cvmfs trace file leading to missing json)
- several bug fixes in the return codes of functions (where errors did not cause immediate exit)
- the separation of CI yaml script into a separate .gitlab-ci.sh, as discussed last week
- a lot of comments in main.sh to describe what each function does, with which inputs and outputs (this is needed eventually to split this up into several jobs)
- the addition of strace in the image and of the options to use it (commented out), this was useful as a cross check of cvmfs tracing to see which files are really needed
- the optional addition of the CACHEBUST mechanism also for the workload images (commented out)
Please merge and give it a try, but in principle it should work
Thanks Andreahttps://gitlab.cern.ch/hep-benchmarks/hep-workloads/-/merge_requests/36Resolve BMK-312019-02-19T13:21:57+01:00Domenico GiordanoResolve BMK-31Closes BMK-31Closes BMK-31https://gitlab.cern.ch/hep-benchmarks/hep-workloads/-/merge_requests/37added table to visualise pipeline status2019-02-19T15:39:43+01:00Domenico Giordanoadded table to visualise pipeline statusCloses BMK-31Closes BMK-31https://gitlab.cern.ch/hep-benchmarks/hep-workloads/-/merge_requests/38bmk-32 change container tag convention2019-02-19T15:51:26+01:00Domenico Giordanobmk-32 change container tag conventionCloses BMK-32Closes BMK-32https://gitlab.cern.ch/hep-benchmarks/hep-workloads/-/merge_requests/39Qa2019-02-19T17:03:14+01:00Domenico GiordanoQahttps://gitlab.cern.ch/hep-benchmarks/hep-workloads/-/merge_requests/40BMK-32 fix image name2019-02-19T16:18:40+01:00Domenico GiordanoBMK-32 fix image nameCloses BMK-32Closes BMK-32https://gitlab.cern.ch/hep-benchmarks/hep-workloads/-/merge_requests/41BMK-32 fix image tag for latest2019-02-19T16:44:05+01:00Domenico GiordanoBMK-32 fix image tag for latestCloses BMK-32Closes BMK-32