cmssw-docker issueshttps://gitlab.cern.ch/cms-cloud/cmssw-docker/-/issues2024-03-26T17:22:25+01:00https://gitlab.cern.ch/cms-cloud/cmssw-docker/-/issues/77eos missing from CMSSW_10_6_30 (and new CMSSW_7_6_7)2024-03-26T17:22:25+01:00Thomas Mc Cauleyeos missing from CMSSW_10_6_30 (and new CMSSW_7_6_7)I have built new versions of the CMSSW_10_6_30 and CMSSW_7_6_7 standalones and placed test images in my registry:
https://gitlab.cern.ch/mccauley/cmssw-docker/container_registry/
If one then builds upon these for equivalent standalones...I have built new versions of the CMSSW_10_6_30 and CMSSW_7_6_7 standalones and placed test images in my registry:
https://gitlab.cern.ch/mccauley/cmssw-docker/container_registry/
If one then builds upon these for equivalent standalones in https://gitlab.cern.ch/cms-cloud/cmssw-docker-opendata the builds fail with `eos: command not found` error.
This seems to be a problem with the images in cms-cloud (which is from an older build):
```plaintext
$ docker run -it --rm gitlab-registry.cern.ch/cms-cloud/cmssw-docker/cmssw_10_6_30-slc7_amd64_gcc700 /bin/bash
Setting up CMSSW_10_6_30
CMSSW should now be available.
This is a standalone image for CMSSW_10_6_30 slc7_amd64_gcc700.
[11:34:12] cmsusr@c8868704491d /code/CMSSW_10_6_30/src $ eos
bash: eos: command not found
```
and in my registry, which has the latest changes:
```plaintext
$ docker run -it --rm gitlab-registry.cern.ch/mccauley/cmssw-docker/cmssw_10_6_30-slc7_amd64_gcc700 /bin/bash
Setting up CMSSW_10_6_30
CMSSW should now be available.
This is a standalone image for CMSSW_10_6_30 slc7_amd64_gcc700.
[11:40:57] cmsusr@1026473e0771 /code/CMSSW_10_6_30/src $ eos
bash: eos: command not found
```
For CMSSW_7_6_7 it works for the older container images in cms-cloud:
```plaintext
$ docker run -it --rm gitlab-registry.cern.ch/cms-cloud/cmssw-docker/cmssw_7_6_7-slc6_amd64_gcc493 /bin/bash
Setting up CMSSW_7_6_7
CMSSW should now be available.
This is a standalone image for CMSSW_7_6_7 slc6_amd64_gcc493.
(/code/CMSSW_7_6_7/src) eos
# ---------------------------------------------------------------------------
# EOS Copyright (C) 2011 CERN/Switzerland
# This program comes with ABSOLUTELY NO WARRANTY; for details type `license'.
# This is free software, and you are welcome to redistribute it
# under certain conditions; type `license' for details.
# ---------------------------------------------------------------------------
```
But for the latest build which is in my registry it does not:
```plaintext
$ docker run -it --rm gitlab-registry.cern.ch/mccauley/cmssw-docker/cmssw_7_6_7-slc6_amd64_gcc493 /bin/bash
Setting up CMSSW_7_6_7
CMSSW should now be available.
This is a standalone image for CMSSW_7_6_7 slc6_amd64_gcc493.
[11:42:58] cmsusr@7a434d69591c /code/CMSSW_7_6_7/src $ eos
bash: eos: command not found
```
We seem to have lost something in our new builds.https://gitlab.cern.ch/cms-cloud/cmssw-docker/-/issues/76For discussion: stop building base image for SLC62024-02-29T13:28:32+01:00Clemens LangeFor discussion: stop building base image for SLC6Building a base image for SLC5 broke quite some time ago and SLC6 is getting more and more difficult. I would suggest we do the following:
- define a legacy base image for SLC5 and archive it somewhere, e.g. on open data portal (and als...Building a base image for SLC5 broke quite some time ago and SLC6 is getting more and more difficult. I would suggest we do the following:
- define a legacy base image for SLC5 and archive it somewhere, e.g. on open data portal (and also double check that we have it in the registry here and on Docker Hub)
- define a legacy base image for SLC6 and archive it somewhere, e.g. on open data portal (and also double check that we have it in the registry here and on Docker Hub)
This refers to the images that do not have a CMSSW release installed.
In the future, install a minimal set of updated packages if needed, but start from those images.
I don't think there's a point trying to exercise the building of those really old distributions, it only adds to our maintenance load.
What do you think? @kati @mccauleyhttps://gitlab.cern.ch/cms-cloud/cmssw-docker/-/issues/75Add twiki link regarding cmspkg to README2024-02-28T10:19:22+01:00Clemens LangeAdd twiki link regarding cmspkg to READMEJust came across this: https://twiki.cern.ch/twiki/bin/view/CMSPublic/SDTCMSSW_aptinstaller
It basically contains the reason and instructions on what we do for the standalone images and it would be good to have that in the README somewhere.Just came across this: https://twiki.cern.ch/twiki/bin/view/CMSPublic/SDTCMSSW_aptinstaller
It basically contains the reason and instructions on what we do for the standalone images and it would be good to have that in the README somewhere.https://gitlab.cern.ch/cms-cloud/cmssw-docker/-/issues/74Add cms-nanoAOD/jsonpog-integration files to standalone images2024-01-29T08:44:14+01:00Julie HoganAdd cms-nanoAOD/jsonpog-integration files to standalone imagesThe correctionLib files live in /cvmfs/cms.cern.ch/rsync/cms-nanoAOD/jsonpog-integration. Coming up soon these need to be accessible in the Open Data containers that are built off these images. Can a local version of the contents from th...The correctionLib files live in /cvmfs/cms.cern.ch/rsync/cms-nanoAOD/jsonpog-integration. Coming up soon these need to be accessible in the Open Data containers that are built off these images. Can a local version of the contents from this path (or the path itself) be added to the standalone images? Relevant for CMSSW_10_6_30 and newer.https://gitlab.cern.ch/cms-cloud/cmssw-docker/-/issues/71Standalone builds don't get triggered through web pipeline anymore2023-10-17T10:18:56+02:00Clemens LangeStandalone builds don't get triggered through web pipeline anymoreUp until the GitLab upgrade to version 16, it used to be possible to trigger a standalone image build by providing `RELEASE` and `SCRAM_ARCH` variables using the web UI and then trigger a pipeline like that. Trying to do this now, this d...Up until the GitLab upgrade to version 16, it used to be possible to trigger a standalone image build by providing `RELEASE` and `SCRAM_ARCH` variables using the web UI and then trigger a pipeline like that. Trying to do this now, this doesn't work anymore. Probably needs the `.gitlab-ci.yml` file to be updated to enable `web` explicitly.https://gitlab.cern.ch/cms-cloud/cmssw-docker/-/issues/70JobConfig files need update.2023-09-29T15:36:11+02:00Artur KalinowskiJobConfig files need update.At lest since version 13 CMSSW jobs require data-access block, like this:
```XML
<data-access>
<catalog volume="CERN_EOS" protocol="XRootD"/>
<catalog site="local" volume="CERN_EOS" protocol="XRootD"/>
</data-access>
```
...At lest since version 13 CMSSW jobs require data-access block, like this:
```XML
<data-access>
<catalog volume="CERN_EOS" protocol="XRootD"/>
<catalog site="local" volume="CERN_EOS" protocol="XRootD"/>
</data-access>
```
later the jobs looks for storage.xml. The setting as on lxplus, directing to storage.xml in SITECONF/T1_IT_CNAF/storage.json did not work for me - navigation with relative paths did not work.https://gitlab.cern.ch/cms-cloud/cmssw-docker/-/issues/68Pushing to Docker Hub fails2023-05-04T11:59:27+02:00Clemens LangePushing to Docker Hub failsThis seems to happen from time to time, started about 2 weeks ago.
Examples:
- https://gitlab.cern.ch/cms-cloud/cmssw-docker/-/jobs/29315937
- https://gitlab.cern.ch/cms-cloud/cmssw-docker/-/jobs/29315938
Errors:
```
error pushing im...This seems to happen from time to time, started about 2 weeks ago.
Examples:
- https://gitlab.cern.ch/cms-cloud/cmssw-docker/-/jobs/29315937
- https://gitlab.cern.ch/cms-cloud/cmssw-docker/-/jobs/29315938
Errors:
```
error pushing image: failed to push to destination cmscloud/slc6-cvmfs:latest: Post "https://index.docker.io/v2/cmscloud/slc6-cvmfs/blobs/uploads/?from=cms-cloud%2Fcmssw-docker%2Fslc6-cms&mount=sha256%3Acfaf302ba060b3d02ea08aca7f0c75226afeccaaffa96859a9233c7f2397b8f0&origin=gitlab-registry.cern.ch": GET https://auth.docker.io/token?scope=repository%3Acmscloud%2Fslc6-cvmfs%3Apull%2Cpush+repository%3Acms-cloud%2Fcmssw-docker%2Fslc6-cms%3Apull&scope=repository%3Acmscloud%2Fslc6-cvmfs%3Apush%2Cpull&service=registry.docker.io: unexpected status code 400 Bad Request: {"details":"invalid repository name"}
```
and
```
INFO[0105] Pushing image to cmscloud/cc7-cvmfs:latest
error pushing image: failed to push to destination cmscloud/cc7-cvmfs:latest: Post "https://index.docker.io/v2/cmscloud/cc7-cvmfs/blobs/uploads/?from=cms-cloud%2Fcmssw-docker%2Fcc7-cms&mount=sha256%3Ab2fa7dce2ca759a3a10c80851c74afdebbd3f73db0ca8f018f9ddfb1dedbe1a9&origin=gitlab-registry.cern.ch": GET https://auth.docker.io/token?scope=repository%3Acmscloud%2Fcc7-cvmfs%3Apull%2Cpush+repository%3Acms-cloud%2Fcmssw-docker%2Fcc7-cms%3Apull&scope=repository%3Acmscloud%2Fcc7-cvmfs%3Apush%2Cpull&service=registry.docker.io: unexpected status code 400 Bad Request: {"details":"invalid repository name"}
```https://gitlab.cern.ch/cms-cloud/cmssw-docker/-/issues/57SLC5 base image builds failing2023-10-03T19:47:16+02:00Clemens LangeSLC5 base image builds failingSee e.g. https://gitlab.cern.ch/cms-cloud/cmssw-docker/-/jobs/22479001
Seems to be related to the EGI repository and https... Can you please have a look @mccauley ?See e.g. https://gitlab.cern.ch/cms-cloud/cmssw-docker/-/jobs/22479001
Seems to be related to the EGI repository and https... Can you please have a look @mccauley ?Thomas Mc CauleyClemens LangeThomas Mc Cauleyhttps://gitlab.cern.ch/cms-cloud/cmssw-docker/-/issues/47cc7 build for CMSSW_7_6_7 and ROOT2021-09-21T16:36:15+02:00Thomas Mc Cauleycc7 build for CMSSW_7_6_7 and ROOTROOT does not seem to work for cc7 builds of CMSSW_7_6_7
See https://hypernews.cern.ch/HyperNews/CMS/get/swDevelopment/3592.htmlROOT does not seem to work for cc7 builds of CMSSW_7_6_7
See https://hypernews.cern.ch/HyperNews/CMS/get/swDevelopment/3592.htmlhttps://gitlab.cern.ch/cms-cloud/cmssw-docker/-/issues/44Authorization problem in slc7 CMSSW_7_6_7 container2021-09-21T16:36:15+02:00Andro PetkovicAuthorization problem in slc7 CMSSW_7_6_7 containerWhen I issue `xrdcp root://eospublic.cern.ch//eos/opendata/cms/Run2015D/DoubleMuon/MINIAOD/16Dec2015-v1/10000/00039A2E-D7A7-E511-98EE-3417EBE64696.root test.root` I get `Run: [FATAL] Auth failed`. I also cannot open that file when I do `...When I issue `xrdcp root://eospublic.cern.ch//eos/opendata/cms/Run2015D/DoubleMuon/MINIAOD/16Dec2015-v1/10000/00039A2E-D7A7-E511-98EE-3417EBE64696.root test.root` I get `Run: [FATAL] Auth failed`. I also cannot open that file when I do `cmsRun poet_cfg.py` which is defined here https://github.com/apetkovi1/PhysObjectExtractorTool/blob/MiniAOD/PhysObjectExtractor/python/poet_cfg.py. The error that I get is:
>18-Aug-2021 16:13:33 CEST Initiating request to open file root://eospublic.cern.ch//eos/opendata/cms/Run2015D/DoubleMuon/MINIAOD/16Dec2015-v1/10000/00039A2E-D7A7-E511-98EE-3417EBE64696.root [2021-08-18 16:13:33.711855 +0200][Error ][AsyncSock ] [eospublic.cern.ch:1094 #0.0] Socket error while handshaking: [FATAL] Auth failed
>[2021-08-18 16:13:33.712063 +0200][Error ][PostMaster ] [eospublic.cern.ch:1094 #0] Unable to recover: [FATAL] Auth failed.
>[2021-08-18 16:13:33.712121 +0200][Error ][XRootD ] [eospublic.cern.ch:1094] Impossible to send message kXR_prepare (flags: kXR_stage , priority: 0, paths: /eos/opendata/cms/Run2015D/DoubleMuon/MINIAOD/16Dec2015-v1/10000/00039A2E-D7A7-E511-98EE-3417EBE64696.root). Trying to recover.
>[2021-08-18 16:13:33.712228 +0200][Error ][XRootD ] [eospublic.cern.ch:1094] Impossible to send message kXR_query (code: kXR_Qconfig, arg length: 9). Trying to recover.
>[2021-08-18 16:13:33.712702 +0200][Error ][XRootD ] [eospublic.cern.ch:1094] Unable to send the message kXR_open (file: /eos/opendata/cms/Run2015D/DoubleMuon/MINIAOD/16Dec2015-v1/10000/00039A2E-D7A7-E511-98EE-3417EBE64696.root, mode: 0660, flags: kXR_open_read kXR_async kXR_retstat ): [FATAL] Auth failed
>----- Begin Fatal Exception 18-Aug-2021 16:13:33 CEST-----------------------
>An exception of category 'FileOpenError' occurred while
[0] Constructing the EventProcessor [1] Constructing input source of type PoolSource [2] Calling RootFileSequenceBase::initTheFile() [3] Calling StorageFactory::open() [4] Calling XrdFile::open() Exception Message: Failed to open the file 'root://eospublic.cern.ch//eos/opendata/cms/Run2015D/DoubleMuon/MINIAOD/16Dec2015-v1/10000/00039A2E-D7A7-E511-98EE-3417EBE64696.root' Additional Info: [a] Input file root://eospublic.cern.ch//eos/opendata/cms/Run2015D/DoubleMuon/MINIAOD/16Dec2015-v1/10000/00039A2E-D7A7-E511-98EE-3417EBE64696.root could not be opened. [b] XrdCl::File::Open(name='root://eospublic.cern.ch//eos/opendata/cms/Run2015D/DoubleMuon/MINIAOD/16Dec2015-v1/10000/00039A2E-D7A7-E511-98EE-3417EBE64696.root', flags=0x10, permissions=0660) => error '[FATAL] Auth failed' (errno=0, code=204) [c] Remote server already encountered a fatal error; no redirections were performed. ----- End Fatal Exception -------------------------------------------------https://gitlab.cern.ch/cms-cloud/cmssw-docker/-/issues/42Add eos to slc52021-08-05T21:41:38+02:00Thomas Mc CauleyAdd eos to slc5eos has been added to slc6 and cc7 (via !43) but slc5 will be, and has proven to have been, trickier. See https://gitlab.cern.ch/cms-cloud/cmssw-docker/-/issues/18#note_4653631.eos has been added to slc6 and cc7 (via !43) but slc5 will be, and has proven to have been, trickier. See https://gitlab.cern.ch/cms-cloud/cmssw-docker/-/issues/18#note_4653631.https://gitlab.cern.ch/cms-cloud/cmssw-docker/-/issues/39CI tests fail for standalone2022-11-17T15:37:48+01:00Clemens LangeCI tests fail for standaloneSee e.g. https://gitlab.cern.ch/cms-cloud/cmssw-docker/-/jobs/14974149 which is triggered by a recent merge request.
The jobs should run with tag `cms-fat` so that our own runners are used (needs change in `.gitlab-ci.yml`) or the `chec...See e.g. https://gitlab.cern.ch/cms-cloud/cmssw-docker/-/jobs/14974149 which is triggered by a recent merge request.
The jobs should run with tag `cms-fat` so that our own runners are used (needs change in `.gitlab-ci.yml`) or the `check.sh` script.https://gitlab.cern.ch/cms-cloud/cmssw-docker/-/issues/38Export PATH and LD_LIBRARY_PATH consistently2021-08-11T15:19:28+02:00Clemens LangeExport PATH and LD_LIBRARY_PATH consistentlyAs discussed in #37, there is some confusion about where and when the `PATH` and `LD_LIBRARY_PATH` are changed/prepended so that the updated git version etc. are picked up. This should be done consistently and following "docker best prac...As discussed in #37, there is some confusion about where and when the `PATH` and `LD_LIBRARY_PATH` are changed/prepended so that the updated git version etc. are picked up. This should be done consistently and following "docker best practices" (to be determined).https://gitlab.cern.ch/cms-cloud/cmssw-docker/-/issues/35myproxy missing.2021-08-11T15:19:28+02:00Artur Kalinowskimyproxy missing.* the cc7-cvmfs image misses myproxy package. This package is necessary for submitting crab jobs
* also it would be usefull to have the grid.cern.ch cvmfs repository available* the cc7-cvmfs image misses myproxy package. This package is necessary for submitting crab jobs
* also it would be usefull to have the grid.cern.ch cvmfs repository availablehttps://gitlab.cern.ch/cms-cloud/cmssw-docker/-/issues/34Missing fonts2021-08-11T15:19:28+02:00Clemens LangeMissing fontsIn an SLC6 container, ROOT complains about missing fonts:
```shell
Couldn't find font "-adobe-helvetica-medium-r-*-*-10-*-*-*-*-*-iso8859-1",
trying "fixed". Please fix your system so helvetica can be found,
this font typically is in th...In an SLC6 container, ROOT complains about missing fonts:
```shell
Couldn't find font "-adobe-helvetica-medium-r-*-*-10-*-*-*-*-*-iso8859-1",
trying "fixed". Please fix your system so helvetica can be found,
this font typically is in the rpm (or pkg equivalent) package
XFree86-[75,100]dpi-fonts or fonts-xorg-[75,100]dpi.
```
This can probably be fixed by installing more packages, see https://root-forum.cern.ch/t/adobe-helvetica-on-ubuntu-11-04/12319/10https://gitlab.cern.ch/cms-cloud/cmssw-docker/-/issues/33Git binary hardlinks cause huge docker image2021-08-11T15:19:28+02:00Clemens LangeGit binary hardlinks cause huge docker imagehttps://gitlab.cern.ch/cms-cloud/cmssw-docker/-/merge_requests/35 introduced updated binaries. It turns out that the hard-links in `/usr/local/libexec/git-core` lead to an increase of the image size by 2 GB.https://gitlab.cern.ch/cms-cloud/cmssw-docker/-/merge_requests/35 introduced updated binaries. It turns out that the hard-links in `/usr/local/libexec/git-core` lead to an increase of the image size by 2 GB.Clemens LangeClemens Langehttps://gitlab.cern.ch/cms-cloud/cmssw-docker/-/issues/28Consider making SITECONF configurable2021-08-11T15:19:28+02:00Clemens LangeConsider making SITECONF configurableNeed to be authenticated to pick one from https://gitlab.cern.ch/SITECONF/ and also access this list.Need to be authenticated to pick one from https://gitlab.cern.ch/SITECONF/ and also access this list.https://gitlab.cern.ch/cms-cloud/cmssw-docker/-/issues/24T2_CH_CERN site-local-config not accessible outside of CERN2021-08-11T15:19:27+02:00Alexx PerloffT2_CH_CERN site-local-config not accessible outside of CERNThe site-local-config.xml setup in the standalone image (https://gitlab.cern.ch/cms-cloud/cmssw-docker/-/blob/master/standalone/Dockerfile#L39-41) references T2_CH_CERN. This site is not accessible outside of CERN. We got around this usi...The site-local-config.xml setup in the standalone image (https://gitlab.cern.ch/cms-cloud/cmssw-docker/-/blob/master/standalone/Dockerfile#L39-41) references T2_CH_CERN. This site is not accessible outside of CERN. We got around this using the prescription [here](https://gitlab.cern.ch/TreeMaker/cmssw-docker/-/commit/2c2fe7c36f8938365227ed71f12835150c4e7517). We stored a replacement site-local-config.xml in the repo because of all of the changes necessary. If these images are going to be used by people around the world, it would be nice if the configurations were setup to be accessible outside of CERN. I can create a PR to make this change if you'd like.https://gitlab.cern.ch/cms-cloud/cmssw-docker/-/issues/17Add CI step to check images with trivy2021-08-11T15:19:27+02:00Clemens LangeAdd CI step to check images with trivySee https://github.com/aquasecurity/trivySee https://github.com/aquasecurity/trivy