cmssw-docker merge requestshttps://gitlab.cern.ch/cms-cloud/cmssw-docker/-/merge_requests2018-07-07T09:38:43+02:00https://gitlab.cern.ch/cms-cloud/cmssw-docker/-/merge_requests/1Create LICENSE2018-07-07T09:38:43+02:00Clemens LangeCreate LICENSE*Created by: clelange*
*Created by: clelange*
https://gitlab.cern.ch/cms-cloud/cmssw-docker/-/merge_requests/5Add entrypoint and change cmsusr ID to 10002018-07-25T15:53:31+02:00Clemens LangeAdd entrypoint and change cmsusr ID to 1000*Created by: clelange*
Closes #3
Closes #4
@tiborsimko - I've built `clelange/cmssw:5_3_32` with this already, my tests were successful, would be great if you tested it as well.*Created by: clelange*
Closes #3
Closes #4
@tiborsimko - I've built `clelange/cmssw:5_3_32` with this already, my tests were successful, would be great if you tested it as well.https://gitlab.cern.ch/cms-cloud/cmssw-docker/-/merge_requests/6Multistage build2018-07-25T15:53:47+02:00Clemens LangeMultistage build*Created by: clelange*
Overall restructuring of images, also adding grid certificates and vomses.
Closes #2 *Created by: clelange*
Overall restructuring of images, also adding grid certificates and vomses.
Closes #2 https://gitlab.cern.ch/cms-cloud/cmssw-docker/-/merge_requests/7update documentation2018-07-25T16:22:03+02:00Clemens Langeupdate documentation*Created by: clelange*
*Created by: clelange*
https://gitlab.cern.ch/cms-cloud/cmssw-docker/-/merge_requests/8SLC5 container2018-10-25T15:33:00+02:00Clemens LangeSLC5 container*Created by: clelange*
This PR adds an SLC5 container to be able to run CMSSW_4_X_Y.
The entrypoint.sh script is now also nicer and does still allow to start the container when CVMFS is not mounted.*Created by: clelange*
This PR adds an SLC5 container to be able to run CMSSW_4_X_Y.
The entrypoint.sh script is now also nicer and does still allow to start the container when CVMFS is not mounted.https://gitlab.cern.ch/cms-cloud/cmssw-docker/-/merge_requests/9CC72018-10-25T17:36:01+02:00Clemens LangeCC7*Created by: clelange*
Add CERN Cent OS 7 container for CMS software.*Created by: clelange*
Add CERN Cent OS 7 container for CMS software.https://gitlab.cern.ch/cms-cloud/cmssw-docker/-/merge_requests/10add packages needed for CMS bootstrap script2018-10-26T15:32:13+02:00Clemens Langeadd packages needed for CMS bootstrap script*Created by: clelange*
*Created by: clelange*
https://gitlab.cern.ch/cms-cloud/cmssw-docker/-/merge_requests/11Make base image configurable for standalone images2018-10-26T17:31:13+02:00Clemens LangeMake base image configurable for standalone images*Created by: clelange*
*Created by: clelange*
https://gitlab.cern.ch/cms-cloud/cmssw-docker/-/merge_requests/12add pipeline for building docker images2020-06-19T09:52:19+02:00Adelina Eleonora Lintuluotoadd pipeline for building docker imagesThe pipeline contains two different build jobs, building a standalone or a cms/cvmfs version. In addition it contains a job for checking which files have been modified, enabling to trigger the build of the cms/cvmfs version that was chan...The pipeline contains two different build jobs, building a standalone or a cms/cvmfs version. In addition it contains a job for checking which files have been modified, enabling to trigger the build of the cms/cvmfs version that was changed. Only reason for there to be a separate script `check.sh` and not just a multi line script is because I didn't get the multi line syntax to work with a for loop
```
script:
- >
```
If you know how to fix it we can transfer the code in `check.sh` to the pipeline!
-----------------------------
The two build jobs:
* A standalone build is triggered when the variables `CMSSW_VERSION`, `SCRAM_ARCH` and `BASE_IMAGE` is passed, could be passed with a webhook.
* A cms/cvmfs version is built when there has been a change in the directory containing the Dockerfile corresponding to the image. Since there is only one job but five directories, before building, the job has to set the IMAGE_NAME variable to indicate which image to build (cc7-cms/cvmfs, slc6-cms/cvmfs, slc5-cms). This is set with the job `check changed files`
The file `check.sh` calls
```
files=$(git diff-tree --no-commit-id --name-only -r $CI_COMMIT_SHA)
```
which returns a list of all files that has been modified. The directory of the file is checked and the pipeline is triggered again, this time with the variable IMAGE_NAME set to the directory name. (The reason why the job tags docker is because the runner has git installed).https://gitlab.cern.ch/cms-cloud/cmssw-docker/-/merge_requests/13CI for image building2020-07-14T14:23:39+02:00Adelina Eleonora LintuluotoCI for image buildingcloses #16
Add CI for building images.
Different ways of building images:
1. Changes pushed to cms/cvmfs folders
* Trigger build of `cms_cvmfs` template
2. Schedule including the variable `IMAGE_NAME`
* Trigger build ...closes #16
Add CI for building images.
Different ways of building images:
1. Changes pushed to cms/cvmfs folders
* Trigger build of `cms_cvmfs` template
2. Schedule including the variable `IMAGE_NAME`
* Trigger build of `cms_cvmfs` template
3. (e.g.) API call including variables `RELEASE`, `SCRAM_ARCH` and `OS`
* Trigger build of `standalone` template
Changes:
* Use build template for both `cms_cvfms` and `standalone`.
* Trigger `check_changed_files` only when new changes are pushed (avoid triggering on API/pipeline)https://gitlab.cern.ch/cms-cloud/cmssw-docker/-/merge_requests/14CI move date into script2020-07-14T18:19:05+02:00Adelina Eleonora LintuluotoCI move date into scriptMove setting of `DATE` to inside script, since at `before_script` it didn't get picked up when setting variable values. Tested and finsihed successfully in https://gitlab.cern.ch/cms-cloud/cmssw-docker/pipelines/1796224.Move setting of `DATE` to inside script, since at `before_script` it didn't get picked up when setting variable values. Tested and finsihed successfully in https://gitlab.cern.ch/cms-cloud/cmssw-docker/pipelines/1796224.https://gitlab.cern.ch/cms-cloud/cmssw-docker/-/merge_requests/15CI two build templates2020-07-16T10:26:43+02:00Adelina Eleonora LintuluotoCI two build templatesSeparate `cms/cvmfs` and `standalone` into two build templates. Tested and successfully finished
* cms/cvmfs: https://gitlab.cern.ch/cms-cloud/cmssw-docker/pipelines/1797597
* standalone: https://gitlab.cern.ch/cms-cloud/cmssw-docker/pi...Separate `cms/cvmfs` and `standalone` into two build templates. Tested and successfully finished
* cms/cvmfs: https://gitlab.cern.ch/cms-cloud/cmssw-docker/pipelines/1797597
* standalone: https://gitlab.cern.ch/cms-cloud/cmssw-docker/pipelines/1797602
Currently images are only pushed to the gitlab registry, should it be pushed to docker hub and if so which profile?https://gitlab.cern.ch/cms-cloud/cmssw-docker/-/merge_requests/16Move all base images and references to cmscloud/cms-cloud, making the CERN...2020-07-16T12:04:37+02:00Clemens LangeMove all base images and references to cmscloud/cms-cloud, making the CERN...Move all base images and references to cmscloud/cms-cloud, making the CERN registry the default if applicableMove all base images and references to cmscloud/cms-cloud, making the CERN registry the default if applicableAdelina Eleonora LintuluotoAdelina Eleonora Lintuluotohttps://gitlab.cern.ch/cms-cloud/cmssw-docker/-/merge_requests/17CI add comments and test buidling2020-07-16T18:15:57+02:00Adelina Eleonora LintuluotoCI add comments and test buidlingAdded comments to `gitlab-ci.yml` and `check.sh` as well as added a test template.
#### How it works:
The job `check_changes_pipeline` is triggered at push or merge request. It has two jobs
1. to trigger the build of "production images...Added comments to `gitlab-ci.yml` and `check.sh` as well as added a test template.
#### How it works:
The job `check_changes_pipeline` is triggered at push or merge request. It has two jobs
1. to trigger the build of "production images" if on master (`--form ref=$CI_DEFAULT_BRANCH `)
2. to trigger "testing images" if on non-master branch (`--form ref=$CI_COMMIT_BRANCH`)
The job `test_all_pipeline` is triggered only on non-master branches and expects a variable `$TEST` in addition to `$IMAGE_NAME`. (The variable `$TEST` is probably unnecessary since it only runs on non-master branches and `check.sh` only triggers the build of "testing images" only on non-master branches.) The test images are pushed to `https://gitlab.cern.ch/cms-cloud/cmssw-docker/test`.
The jobs `build_cms_cvmfs_pipeline` and `build_standalone_pipeline` are only triggered on master.
#### One issue:
It seems like the first line of `check.sh` (`git diff-tree --no-commit-id --name-only -r $CI_COMMIT_SHA`) is empty if the commit came from a merge request. This means that in a scenario like
1. Changes are pushed to a branch
2. `check.sh` triggers the build of "testing images"
3. If everything looks good we merge the branch into master
4. `check.sh` is triggered however it doesn't find any changed files
5. No "production images" are built
I haven't found a way around this yet.https://gitlab.cern.ch/cms-cloud/cmssw-docker/-/merge_requests/18CI transform release to lowercase2020-07-17T13:10:51+02:00Adelina Eleonora LintuluotoCI transform release to lowercaseMinor change to resolve an issue that occurred when trying to build image `CMSSW_11_1_0_Patatrack`. Docker image name has to be lowercase, why I naively made the release to be given in lowercase. I then later turn it into uppercase when ...Minor change to resolve an issue that occurred when trying to build image `CMSSW_11_1_0_Patatrack`. Docker image name has to be lowercase, why I naively made the release to be given in lowercase. I then later turn it into uppercase when it is passed to the Dockerfile. In this case that lead to `CMSSW_VERSION=CMSSW_11_1_0_PATATRACK` to be passed to the Dockerfile as opposed to the expected version `CMSSW_VERSION=CMSSW_11_1_0_Patatrack`.
Fixed by asking for release in normal typing (with upper and lowercase chars) and then transforming it to lowercase for the image name. The change was tested yesterday and successfully built the image `CMSSW_11_1_0_Patatrack`.https://gitlab.cern.ch/cms-cloud/cmssw-docker/-/merge_requests/19on_success and on_failure2020-07-23T09:44:06+02:00Adelina Eleonora Lintuluotoon_success and on_failureCloses #22
Add on_success and on_failure for the standalone image build. On success it removes all labels and closes corresponding Gitlab issue. On failure changes the label of corresponding Gitlab issue to `Pipeline failed` and commen...Closes #22
Add on_success and on_failure for the standalone image build. On success it removes all labels and closes corresponding Gitlab issue. On failure changes the label of corresponding Gitlab issue to `Pipeline failed` and comments `The pipeline of this request just failed` on issue.
The variables that needs to be added to Settings -> CI/CD -> Variables are
* `BACKEND_IID` 87550
* `BACKEND_URL` https://cmssw-docker.app.cern.ch
* `API_TOKEN` same as `APITOKEN` in the backend
* `GITLAB_TOKEN` same as `GITLABTOKEN` in the backendhttps://gitlab.cern.ch/cms-cloud/cmssw-docker/-/merge_requests/20Remove on_success on_failure2020-08-25T13:50:35+02:00Adelina Eleonora LintuluotoRemove on_success on_failureCloses https://gitlab.cern.ch/cms-cloud/cmssw-docker/-/issues/23
Remove on_success on_failure from the CI since they have been moved and are by https://gitlab.cern.ch/cms-cloud/cmssw-docker-service/-/merge_requests/12 handled by the Go ...Closes https://gitlab.cern.ch/cms-cloud/cmssw-docker/-/issues/23
Remove on_success on_failure from the CI since they have been moved and are by https://gitlab.cern.ch/cms-cloud/cmssw-docker-service/-/merge_requests/12 handled by the Go backend.https://gitlab.cern.ch/cms-cloud/cmssw-docker/-/merge_requests/21Remove grid-deployment repository, HEP_OSlibs should be provided by wlcg repo2020-09-16T10:28:38+02:00Clemens LangeRemove grid-deployment repository, HEP_OSlibs should be provided by wlcg repoSeems to be working fine. Still need to check if the files are identical or not to confirm we're not breaking anything.Seems to be working fine. Still need to check if the files are identical or not to confirm we're not breaking anything.https://gitlab.cern.ch/cms-cloud/cmssw-docker/-/merge_requests/22Update base image to cc7-cms:20200901-1.x86_642020-09-16T12:22:28+02:00Clemens LangeUpdate base image to cc7-cms:20200901-1.x86_64https://gitlab.cern.ch/cms-cloud/cmssw-docker/-/merge_requests/23Tag docker-privileged2020-09-25T17:09:40+02:00Adelina Eleonora LintuluotoTag docker-privilegedCloses #26
Any volume mounted to the Kaniko container is automatically whitelisted and excluded when building the Docker image. `/cvmfs` is mounted to the container via the runner. Hence when executing the Dockerfile, the command `mkd...Closes #26
Any volume mounted to the Kaniko container is automatically whitelisted and excluded when building the Docker image. `/cvmfs` is mounted to the container via the runner. Hence when executing the Dockerfile, the command `mkdir -p /cvfms/cms.cern.ch` does not create the desired directory. This later leads to problems when trying to mount `/cvmfs` at entrypoint.
Currently runners tagged `docker-privileged` does not have `/cvmfs` mounted to them.