LbNightlyTools issueshttps://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues2020-10-12T10:57:39+02:00https://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/18Ability to abort /ci-test slots2020-10-12T10:57:39+02:00Rosen MatevAbility to abort /ci-test slotsWe need to
1. Provide a url which when clicked sets the abort flag in couchdb.
1. Poll from checkout/build/tests jobs whenever possible and abort.
1. Publish an abort link in the feedback from the main job.
1. Hide aborted slots from the...We need to
1. Provide a url which when clicked sets the abort flag in couchdb.
1. Poll from checkout/build/tests jobs whenever possible and abort.
1. Publish an abort link in the feedback from the main job.
1. Hide aborted slots from the dashboard
1. Adapt the logic for reusing slots such that the check fails for aborted slotsRosen MatevRosen Matevhttps://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/89Add automatic retry of failed tests (in CMake builds)2021-03-15T12:42:24+01:00Marco Clemencicmarco.clemencic@cern.chAdd automatic retry of failed tests (in CMake builds)CTest offers an option to automatically retry failed (or timed out) tests.
Since we have a few tests that are not stable and may fail because of environmental issues (e.g. overloaded machines), we should enable the retry.CTest offers an option to automatically retry failed (or timed out) tests.
Since we have a few tests that are not stable and may fail because of environmental issues (e.g. overloaded machines), we should enable the retry.3.0.28Marco Clemencicmarco.clemencic@cern.chMarco Clemencicmarco.clemencic@cern.chhttps://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/69Add command line option to specify platform in lbn-get-new-refs2020-05-14T14:16:53+02:00Ben CouturierAdd command line option to specify platform in lbn-get-new-refsAdd command line option to specify platform in lbn-get-new-refs, as requested by Marco.
This would be more practical than dealing with environment variables.Add command line option to specify platform in lbn-get-new-refs, as requested by Marco.
This would be more practical than dealing with environment variables.3.0.9Marco Clemencicmarco.clemencic@cern.chMarco Clemencicmarco.clemencic@cern.chhttps://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/41Add CouchDB view to list slots including a MR2019-12-19T17:12:39+01:00Rosen MatevAdd CouchDB view to list slots including a MRIn order to display a MR-centric overview of what is being tested in which slot, we need a way to get the slots that include a given MR.In order to display a MR-centric overview of what is being tested in which slot, we need a way to get the slots that include a given MR.2.2.3Marco Clemencicmarco.clemencic@cern.chMarco Clemencicmarco.clemencic@cern.chhttps://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/27Add date field to CouchDB document created by main Jenkins job2019-11-20T15:02:59+01:00Marco Clemencicmarco.clemencic@cern.chAdd date field to CouchDB document created by main Jenkins jobWith the correct date field, the slot will appear in the summary page as soon as the Jenkins *main* job creates the document (at the moment we have to wait the the checkout job updates the document).With the correct date field, the slot will appear in the summary page as soon as the Jenkins *main* job creates the document (at the moment we have to wait the the checkout job updates the document).2.2.0Marco Clemencicmarco.clemencic@cern.chMarco Clemencicmarco.clemencic@cern.chhttps://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/24Add metadata field to Slot class/configuration dict2019-12-11T11:00:13+01:00Marco Clemencicmarco.clemencic@cern.chAdd metadata field to Slot class/configuration dictadding a dict to the slot as *metadata* would be extremely useful to record information that are not strictly related to how to build the slot, but could be used to enhance what we display on web pages.
For example we could have:
- a fl...adding a dict to the slot as *metadata* would be extremely useful to record information that are not strictly related to how to build the slot, but could be used to enhance what we display on web pages.
For example we could have:
- a flag to state that the slot should not appear in the main summaries page (e.g. ref and mr slots are not very useful in that page)
- record the name and id of reference slot build for a MR (/ci-test) job
In principle, also the description of a slot would go in the metadata section, but it can be done at a later stage.2.2.0Marco Clemencicmarco.clemencic@cern.chMarco Clemencicmarco.clemencic@cern.chhttps://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/108Add protection against missing BUILD_URL environment variable2022-07-27T15:47:16+02:00Marco Clemencicmarco.clemencic@cern.chAdd protection against missing BUILD_URL environment variableWhen Jenkins gets upgraded the configured server URL is not correctly taken into account until the configuration is saved again by hand.
Unfortunately this results in the environment variable `BUILD_URL` not being set and we are implici...When Jenkins gets upgraded the configured server URL is not correctly taken into account until the configuration is saved again by hand.
Unfortunately this results in the environment variable `BUILD_URL` not being set and we are implicitly relaying on its presence with the net result that the test results are not correctly reported and unusable.https://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/50Add semantics checks for nightly slots configurations2023-06-20T14:11:06+02:00Marco Clemencicmarco.clemencic@cern.chAdd semantics checks for nightly slots configurationsat the moment we only check that the slot configurations can be loaded, but that only accounts for some basic errors, and not that the configuration is valid (see lhcb-core/LHCbNightlyConf!304).at the moment we only check that the slot configurations can be loaded, but that only accounts for some basic errors, and not that the configuration is valid (see lhcb-core/LHCbNightlyConf!304).4.0.0Marco Clemencicmarco.clemencic@cern.chMarco Clemencicmarco.clemencic@cern.chhttps://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/93Add support for `lhcbproject.json` metadata file2021-06-18T16:29:37+02:00Marco Clemencicmarco.clemencic@cern.chAdd support for `lhcbproject.json` metadata fileThe advantages of using `lhcbproject.json` instead of `lhcbproject.yml` is that JSON does not require external Python modules and that CMake (3.19) has [builtin JSON support](https://cmake.org/cmake/help/latest/command/string.html#json).The advantages of using `lhcbproject.json` instead of `lhcbproject.yml` is that JSON does not require external Python modules and that CMake (3.19) has [builtin JSON support](https://cmake.org/cmake/help/latest/command/string.html#json).3.0.33Marco Clemencicmarco.clemencic@cern.chMarco Clemencicmarco.clemencic@cern.chhttps://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/48Allow force of old cmake style build for projects that use "project" instead ...2020-01-17T16:16:20+01:00Marco Clemencicmarco.clemencic@cern.chAllow force of old cmake style build for projects that use "project" instead of "gaudi_project"The logic to select which CMake build style to use is bound to the presence of the call to `find_package(GaudiProject)` in the top level `CMakeLists.txt` file.
We have at least one case where this logic is not correct: [LHCbIntegrationT...The logic to select which CMake build style to use is bound to the presence of the call to `find_package(GaudiProject)` in the top level `CMakeLists.txt` file.
We have at least one case where this logic is not correct: [LHCbIntegrationTests](https://gitlab.cern.ch/lhcb/LHCbIntegrationTests/).
We need a way to tell the CMake style selection logic that a project must use the old style even if it looks like it's new.3.0.2Marco Clemencicmarco.clemencic@cern.chMarco Clemencicmarco.clemencic@cern.chhttps://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/72Allow full MR URLs in ci-test trigger2020-09-29T14:30:39+02:00Rosen MatevAllow full MR URLs in ci-test triggerhttps://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/92Allow new platform ids when building against LCG nightlies2021-06-10T12:16:18+02:00Marco Clemencicmarco.clemencic@cern.chAllow new platform ids when building against LCG nightliesWith the switch to platform ids with microarchitecture levels (lhcb-core/LHCbNightlyConf!625) we fail to match the platform we want to build with the corresponding LCG nightly build platform. See lhcb-core/LHCbNightlyConf#10
We have to ...With the switch to platform ids with microarchitecture levels (lhcb-core/LHCbNightlyConf!625) we fail to match the platform we want to build with the corresponding LCG nightly build platform. See lhcb-core/LHCbNightlyConf#10
We have to add a different way to parametrize slot configurations based on the platform id.3.0.32Marco Clemencicmarco.clemencic@cern.chMarco Clemencicmarco.clemencic@cern.chhttps://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/71Allow not running tests which are unaffected by MR2023-06-20T12:03:04+02:00Daniel Campora PerezAllow not running tests which are unaffected by MRAt present, issuing a `/ci-test` will build and test all projects involved in the project where the command was run.
It would be very useful to be able to select not to test projects which are unaffected by a MR. It would reduce runtime...At present, issuing a `/ci-test` will build and test all projects involved in the project where the command was run.
It would be very useful to be able to select not to test projects which are unaffected by a MR. It would reduce runtime of these tests and liberate resources in the CI build system.https://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/110Allow selection of projects to run in ci-test2023-09-27T15:28:56+02:00Marco Clemencicmarco.clemencic@cern.chAllow selection of projects to run in ci-testSometime a MR only modifies files affecting one project (e.g. adding a test, modify a qmt file). In these cases the default logic of building and testing downstream projects is a waste of resources.
I should be possible to override the ...Sometime a MR only modifies files affecting one project (e.g. adding a test, modify a qmt file). In these cases the default logic of building and testing downstream projects is a waste of resources.
I should be possible to override the default logic used to select which projects to build or test with some "command line" option.https://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/26Apply LHCb formatting to Python code2019-12-09T17:43:29+01:00Marco Clemencicmarco.clemencic@cern.chApply LHCb formatting to Python code2.2.0Marco Clemencicmarco.clemencic@cern.chMarco Clemencicmarco.clemencic@cern.chhttps://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/119Apptainer occasionally fails to start containers2023-09-27T15:13:46+02:00Rosen MatevApptainer occasionally fails to start containersApptainer occasionally fails to start containers with a pretty cryptic message
```
ERROR : Failed to get file information for file descriptor 3: Bad file descriptor
ERROR : Could not write info to setgroups: Permission denied
```
Thi...Apptainer occasionally fails to start containers with a pretty cryptic message
```
ERROR : Failed to get file information for file descriptor 3: Bad file descriptor
ERROR : Could not write info to setgroups: Permission denied
```
This is disruptive to testing as often we need to re-build (depending on which platform fails).
We should probably
1. Pass `--debug` to apptainer to get a sample of debug logs when the problem happens (to understand what this file descriptor is pointing to).
2. Detect and handle the failure to start the container. (In the second example below, the build of a project fails, and the install and the rest of the projects happily continue.)
3. Until the underlying problem is understood, we should probably retry running apptainer. We should consider if the retrial should be done on any error. The exit code is for example also 1 (the same as when apptainer fails to start) when the build (ninja) fails for a genuine problem.
Some related discussions: https://github.com/apptainer/apptainer/issues/430 https://github.com/apptainer/singularity/pull/4953 https://github.com/apptainer/singularity/issues/5206
### Examples
https://jenkins-lhcb-nightlies.web.cern.ch/job/nightly-builds/job/build/337143/consoleFull
```
2023-09-23 12:15:56,463:DEBUG : running cmake --install LHCb/build --prefix LHCb/InstallArea/x86_64_v2-centos7-clang12-opt
2023-09-23 12:15:56,463:DEBUG : apptainer command: /cvmfs/lhcbdev.cern.ch/nightly-environments/5ced04d9e2e8ad4a44b36e93198a8c2f88c23ebb2313ab50b202d5e241e8cb8d/bin/apptainer exec --contain --bind /cvmfs --bind /home/lblocal/jenkins-build/workspace/nightly-builds/build@2:/workspace --bind /home/lblocal/jenkins-build/workspace/nightly-builds/build@2 --pwd /workspace/build --env PATH=/cvmfs/lhcb.cern.ch/lib/bin/x86_64-centos7:/cvmfs/lhcb.cern.ch/lib/bin/x86_64-centos7:/cvmfs/lhcb.cern.ch/lib/bin/Linux-x86_64:/cvmfs/lhcb.cern.ch/lib/bin:/cvmfs/lhcbdev.cern.ch/nightly-environments/5ced04d9e2e8ad4a44b36e93198a8c2f88c23ebb2313ab50b202d5e241e8cb8d/bin:/cvmfs/lhcbdev.cern.ch/conda/miniconda/linux-64/1622055603/condabin:/usr/sue/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/opt/puppetlabs/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin /cvmfs/lhcb.cern.ch/containers/os-base/centos7-devel/prod/amd64 cmake --install LHCb/build --prefix LHCb/InstallArea/x86_64_v2-centos7-clang12-opt
2023-09-23 12:15:56,620:DEBUG : �[91mERROR : Failed to get file information for file descriptor 3: Bad file descriptor
2023-09-23 12:15:56,621:DEBUG : �[0m�[91mERROR : Could not write info to setgroups: Permission denied
2023-09-23 12:15:56,622:DEBUG : command exited with code 1
2023-09-23 12:15:56,622:DEBUG : Completed at: 2023-09-23 12:15:56.622240
```
https://jenkins-lhcb-nightlies.web.cern.ch/job/nightly-builds/job/build/337186/consoleFull
```
2023-09-23 19:48:31,458:DEBUG : running cmake --build Detector/build -j 10 -- -k0
2023-09-23 19:48:31,458:DEBUG : apptainer command: /cvmfs/lhcbdev.cern.ch/nightly-environments/5ced04d9e2e8ad4a44b36e93198a8c2f88c23ebb2313ab50b202d5e241e8cb8d/bin/apptainer exec --contain --bind /cvmfs --bind /home/lblocal/jenkins-build/workspace/nightly-builds/build@2:/workspace --bind /home/lblocal/jenkins-build/workspace/nightly-builds/build@2 --pwd /workspace/build --env PATH=/cvmfs/lhcb.cern.ch/lib/bin/x86_64-el9:/cvmfs/lhcb.cern.ch/lib/bin/x86_64-centos7:/cvmfs/lhcb.cern.ch/lib/bin/Linux-x86_64:/cvmfs/lhcb.cern.ch/lib/bin:/cvmfs/lhcbdev.cern.ch/nightly-environments/5ced04d9e2e8ad4a44b36e93198a8c2f88c23ebb2313ab50b202d5e241e8cb8d/bin:/cvmfs/lhcbdev.cern.ch/conda/miniconda/linux-64/1622055603/condabin:/usr/sue/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/opt/puppetlabs/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin /cvmfs/lhcb.cern.ch/containers/os-base/alma9-devel/prod/amd64 cmake --build Detector/build -j 10 -- -k0
2023-09-23 19:48:31,609:DEBUG : �[91mERROR : Failed to get file information for file descriptor 3: Bad file descriptor
2023-09-23 19:48:31,609:DEBUG : �[0m�[91mERROR : Could not write info to setgroups: Permission denied
2023-09-23 19:48:31,610:DEBUG : command exited with code 1
2023-09-23 19:48:31,615:DEBUG : running cmake --install Detector/build --prefix Detector/InstallArea/x86_64_v3-el9-gcc12+cuda12_1-opt+g
```
/cc @clemenci @cburr @sponce4.0.2Marco Clemencicmarco.clemencic@cern.chMarco Clemencicmarco.clemencic@cern.chhttps://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/60Avoid crashes when git submodules cannot be cloned2020-02-25T17:32:49+01:00Marco Clemencicmarco.clemencic@cern.chAvoid crashes when git submodules cannot be cloned3.0.6Marco Clemencicmarco.clemencic@cern.chMarco Clemencicmarco.clemencic@cern.chhttps://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/75Broken ci-test argument parsing2023-06-20T12:03:04+02:00Rosen MatevBroken ci-test argument parsing`--platforms x86_64-centos7-gcc9-opt` should work (and not only `--platforms=x86_64-centos7-gcc9-opt`). Consider the following broken example:
```
DEBUG:LbNightlyTools.Scripts.GitlabMR:comment is
/ci-test --merge --platforms x86_64-cent...`--platforms x86_64-centos7-gcc9-opt` should work (and not only `--platforms=x86_64-centos7-gcc9-opt`). Consider the following broken example:
```
DEBUG:LbNightlyTools.Scripts.GitlabMR:comment is
/ci-test --merge --platforms x86_64-centos7-gcc9-opt
INFO:LbNightlyTools.Scripts.GitlabMR:Found trigger with arguments Namespace(merge=True, params=[], platforms=u'lhcb/x86_64-centos7-gcc9-opt', sources=['lhcb/Rec!2232'])
DEBUG:LbNightlyTools.GitlabUtils:_getGitlabProject invoked with (('lhcb/Rec',), {})
DEBUG:LbNightlyTools.GitlabUtils:result not in cache, invoking the function
DEBUG:LbNightlyTools.GitlabUtils:_gitlabServer invoked with ((), {})
DEBUG:LbNightlyTools.GitlabUtils:result in cache
DEBUG:urllib3.connectionpool:https://gitlab.cern.ch:443 "GET //api/v4/projects/lhcb%2FRec HTTP/1.1" 200 2989
DEBUG:urllib3.connectionpool:https://gitlab.cern.ch:443 "GET //api/v4/projects/401/merge_requests/2232 HTTP/1.1" 200 3568
INFO:LbNightlyTools.Scripts.GitlabMR:json job configuration:
{
"sources": [
"lhcb/Rec!2232"
],
"merge": true,
"trigger": {
"merge_request_iid": 2232,
"project_id": 401,
"discussion_id": "23aebfe2fd2ee90485633e906ad9a80787bff6f7",
"note_id": 3875543
},
"platforms": "lhcb/x86_64-centos7-gcc9-opt"
}
```Marco Clemencicmarco.clemencic@cern.chMarco Clemencicmarco.clemencic@cern.chhttps://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/45Build script fails when using new CMake project and no lcg-toolchains available2020-01-16T14:24:33+01:00Marco Clemencicmarco.clemencic@cern.chBuild script fails when using new CMake project and no lcg-toolchains availableExceptions are used to report errors in the configuration step of the build tool for new style CMake projects, but the caller is not instrumented to trap them.
Although the use of exceptions would be more natural (also in the context of...Exceptions are used to report errors in the configuration step of the build tool for new style CMake projects, but the caller is not instrumented to trap them.
Although the use of exceptions would be more natural (also in the context of LBCORE-1711), we have to change the code to prevent spurious failures of the jenkins scripts.3.0.1Marco Clemencicmarco.clemencic@cern.chMarco Clemencicmarco.clemencic@cern.chhttps://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/84Checkout of data packages with non-conventional versions should create conven...2021-03-18T14:19:14+01:00Marco Clemencicmarco.clemencic@cern.chCheckout of data packages with non-conventional versions should create conventional symlinksWhen we checkout a data package *HEAD* we create the symlinks `v999r999` and `vXr999` pointing to the checked out directory.
We should do the same for other *non conventional* versions, like `master` or `Sim10`.
/cc @gcortiWhen we checkout a data package *HEAD* we create the symlinks `v999r999` and `vXr999` pointing to the checked out directory.
We should do the same for other *non conventional* versions, like `master` or `Sim10`.
/cc @gcorti3.0.28Marco Clemencicmarco.clemencic@cern.chMarco Clemencicmarco.clemencic@cern.ch