LbNightlyTools issueshttps://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues2024-01-25T13:16:32+01:00https://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/126use of str.trim instead of the correct method str.strip2024-01-25T13:16:32+01:00Marco Clemencicmarco.clemencic@cern.chuse of str.trim instead of the correct method str.strip```
File "/cvmfs/lhcbdev.cern.ch/nightly-environments/f5a190a09f1257f1556c21d1dbe303e3cecfa6dba878c57dc0c226351eaf8641/lib/python3.11/site-packages/LbNightlyTools/MergeRequestBuilds.py", line 108, in <listcomp>
platforms = [p.trim()...```
File "/cvmfs/lhcbdev.cern.ch/nightly-environments/f5a190a09f1257f1556c21d1dbe303e3cecfa6dba878c57dc0c226351eaf8641/lib/python3.11/site-packages/LbNightlyTools/MergeRequestBuilds.py", line 108, in <listcomp>
platforms = [p.trim() for p in platforms.split(",")]
^^^^^^
AttributeError: 'str' object has no attribute 'trim'. Did you mean: 'strip'?
```4.0.5Marco Clemencicmarco.clemencic@cern.chMarco Clemencicmarco.clemencic@cern.chhttps://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/125mr slots wrongly flagged as valid models for ci-test2024-01-25T15:24:21+01:00Marco Clemencicmarco.clemencic@cern.chmr slots wrongly flagged as valid models for ci-test4.0.5Marco Clemencicmarco.clemencic@cern.chMarco Clemencicmarco.clemencic@cern.chhttps://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/122Starting a ci-test with "--branch" fails to generate slot configuration2023-09-27T15:14:58+02:00Marco Clemencicmarco.clemencic@cern.chStarting a ci-test with "--branch" fails to generate slot configurationTrying to trigger a ci-test with `--branch` from Gaudi (to bypass merge conflicts with the tag used in lhcb-master) resulted in a crash of the script in the Jenkins job "main".
The function `LbNightlyTools.Scripts.EnabledSlots.make_mr_s...Trying to trigger a ci-test with `--branch` from Gaudi (to bypass merge conflicts with the tag used in lhcb-master) resulted in a crash of the script in the Jenkins job "main".
The function `LbNightlyTools.Scripts.EnabledSlots.make_mr_slots` tried to create a (reference?) slot with `lhcb/GaussinoExtLibs` while it should have been `Gaussino/GaussinoExtLibs`.
See the stack trace:
```py
2023-09-27 10:22:31,843:DEBUG : getLastCommit invoked with (('lhcb/GaussinoExtLibs', '2023-09-20T17:55:49', 'master'), {})
2023-09-27 10:22:31,843:DEBUG : result not in cache, invoking the function
2023-09-27 10:22:31,843:DEBUG : _getGitlabProject invoked with (('lhcb/GaussinoExtLibs',), {})
2023-09-27 10:22:31,843:DEBUG : result not in cache, invoking the function
2023-09-27 10:22:31,843:DEBUG : _gitlabServer invoked with ((), {})
2023-09-27 10:22:31,843:DEBUG : result in cache
2023-09-27 10:22:31,877:DEBUG : https://gitlab.cern.ch:443 "GET /api/v4/projects/lhcb%2FGaussinoExtLibs HTTP/1.1" 404 35
Traceback (most recent call last):
File "/cvmfs/lhcbdev.cern.ch/nightly-environments/5ced04d9e2e8ad4a44b36e93198a8c2f88c23ebb2313ab50b202d5e241e8cb8d/lib/python3.11/site-packages/LbNightlyTools/MergeRequestBuilds.py", line 154, in set_last_commits
commit = commits[name]
~~~~~~~^^^^^^
KeyError: 'GaussinoExtLibs'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/cvmfs/lhcbdev.cern.ch/nightly-environments/5ced04d9e2e8ad4a44b36e93198a8c2f88c23ebb2313ab50b202d5e241e8cb8d/lib/python3.11/site-packages/gitlab/exceptions.py", line 336, in wrapped_f
return f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/cvmfs/lhcbdev.cern.ch/nightly-environments/5ced04d9e2e8ad4a44b36e93198a8c2f88c23ebb2313ab50b202d5e241e8cb8d/lib/python3.11/site-packages/gitlab/mixins.py", line 122, in get
server_data = self.gitlab.http_get(path, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/cvmfs/lhcbdev.cern.ch/nightly-environments/5ced04d9e2e8ad4a44b36e93198a8c2f88c23ebb2313ab50b202d5e241e8cb8d/lib/python3.11/site-packages/gitlab/client.py", line 828, in http_get
result = self.http_request(
^^^^^^^^^^^^^^^^^^
File "/cvmfs/lhcbdev.cern.ch/nightly-environments/5ced04d9e2e8ad4a44b36e93198a8c2f88c23ebb2313ab50b202d5e241e8cb8d/lib/python3.11/site-packages/gitlab/client.py", line 794, in http_request
raise gitlab.exceptions.GitlabHttpError(
gitlab.exceptions.GitlabHttpError: 404: 404 Project Not Found
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/cvmfs/lhcbdev.cern.ch/nightly-environments/5ced04d9e2e8ad4a44b36e93198a8c2f88c23ebb2313ab50b202d5e241e8cb8d/bin/lbn-enabled-slots", line 11, in <module>
sys.exit(enabled_slots())
^^^^^^^^^^^^^^^
File "/cvmfs/lhcbdev.cern.ch/nightly-environments/5ced04d9e2e8ad4a44b36e93198a8c2f88c23ebb2313ab50b202d5e241e8cb8d/lib/python3.11/site-packages/LbNightlyTools/Scripts/_entry_points.py", line 67, in enabled_slots
return Script().run()
^^^^^^^^^^^^^^
File "/cvmfs/lhcbdev.cern.ch/nightly-environments/5ced04d9e2e8ad4a44b36e93198a8c2f88c23ebb2313ab50b202d5e241e8cb8d/lib/python3.11/site-packages/LbNightlyTools/Scripts/Common.py", line 61, in run
return _PlainScript.run(self, args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/cvmfs/lhcbdev.cern.ch/nightly-environments/5ced04d9e2e8ad4a44b36e93198a8c2f88c23ebb2313ab50b202d5e241e8cb8d/lib/python3.11/site-packages/LbCommon/Script.py", line 107, in run
rc = self.main()
^^^^^^^^^^^
File "/cvmfs/lhcbdev.cern.ch/nightly-environments/5ced04d9e2e8ad4a44b36e93198a8c2f88c23ebb2313ab50b202d5e241e8cb8d/lib/python3.11/site-packages/LbNightlyTools/Scripts/EnabledSlots.py", line 152, in main
ref_slot, mr_slot = make_mr_slots(mr_slots_config, slots)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/cvmfs/lhcbdev.cern.ch/nightly-environments/5ced04d9e2e8ad4a44b36e93198a8c2f88c23ebb2313ab50b202d5e241e8cb8d/lib/python3.11/site-packages/LbNightlyTools/MergeRequestBuilds.py", line 382, in make_mr_slots
ref_slot, test_slot = create_mr_slots(
^^^^^^^^^^^^^^^^
File "/cvmfs/lhcbdev.cern.ch/nightly-environments/5ced04d9e2e8ad4a44b36e93198a8c2f88c23ebb2313ab50b202d5e241e8cb8d/lib/python3.11/site-packages/LbNightlyTools/MergeRequestBuilds.py", line 318, in create_mr_slots
set_last_commits(ref_slot, ref_date, ref_commits)
File "/cvmfs/lhcbdev.cern.ch/nightly-environments/5ced04d9e2e8ad4a44b36e93198a8c2f88c23ebb2313ab50b202d5e241e8cb8d/lib/python3.11/site-packages/LbNightlyTools/MergeRequestBuilds.py", line 156, in set_last_commits
commit = getLastCommit(full_name(p), ref_date, p.version)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/cvmfs/lhcbdev.cern.ch/nightly-environments/5ced04d9e2e8ad4a44b36e93198a8c2f88c23ebb2313ab50b202d5e241e8cb8d/lib/python3.11/site-packages/LbNightlyTools/GitlabUtils.py", line 45, in wrapper
result = _cache[key] = f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/cvmfs/lhcbdev.cern.ch/nightly-environments/5ced04d9e2e8ad4a44b36e93198a8c2f88c23ebb2313ab50b202d5e241e8cb8d/lib/python3.11/site-packages/LbNightlyTools/MergeRequestBuilds.py", line 135, in getLastCommit
project = _getGitlabProject(project)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/cvmfs/lhcbdev.cern.ch/nightly-environments/5ced04d9e2e8ad4a44b36e93198a8c2f88c23ebb2313ab50b202d5e241e8cb8d/lib/python3.11/site-packages/LbNightlyTools/GitlabUtils.py", line 45, in wrapper
result = _cache[key] = f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/cvmfs/lhcbdev.cern.ch/nightly-environments/5ced04d9e2e8ad4a44b36e93198a8c2f88c23ebb2313ab50b202d5e241e8cb8d/lib/python3.11/site-packages/LbNightlyTools/GitlabUtils.py", line 62, in _getGitlabProject
return _gitlabServer().projects.get(project_id)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/cvmfs/lhcbdev.cern.ch/nightly-environments/5ced04d9e2e8ad4a44b36e93198a8c2f88c23ebb2313ab50b202d5e241e8cb8d/lib/python3.11/site-packages/gitlab/v4/objects/projects.py", line 860, in get
return cast(Project, super().get(id=id, lazy=lazy, **kwargs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/cvmfs/lhcbdev.cern.ch/nightly-environments/5ced04d9e2e8ad4a44b36e93198a8c2f88c23ebb2313ab50b202d5e241e8cb8d/lib/python3.11/site-packages/gitlab/exceptions.py", line 338, in wrapped_f
raise error(e.error_message, e.response_code, e.response_body) from e
gitlab.exceptions.GitlabGetError: 404: 404 Project Not Found
Build step 'Execute shell' marked build as failure
Sending e-mails to: lhcb-core-soft-alarms@cern.ch
Finished: FAILURE
```4.0.2https://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/121UserWarning from gitlab Python bindings2023-09-27T15:14:47+02:00Marco Clemencicmarco.clemencic@cern.chUserWarning from gitlab Python bindingsI noticed this message in the logs of Jenkins "main" job:
```
2023-09-27 10:22:27,993:DEBUG : https://gitlab.cern.ch:443 "GET /api/v4/projects/30597/repository/commits?until=2023-09-20T17%3A55%3A49&ref_name=master HTTP/1.1" 200 14716
/...I noticed this message in the logs of Jenkins "main" job:
```
2023-09-27 10:22:27,993:DEBUG : https://gitlab.cern.ch:443 "GET /api/v4/projects/30597/repository/commits?until=2023-09-20T17%3A55%3A49&ref_name=master HTTP/1.1" 200 14716
/cvmfs/lhcbdev.cern.ch/nightly-environments/5ced04d9e2e8ad4a44b36e93198a8c2f88c23ebb2313ab50b202d5e241e8cb8d/lib/python3.11/site-packages/LbNightlyTools/MergeRequestBuilds.py:138: UserWarning: Calling a `list()` method without specifying `get_all=True` or `iterator=True` will return a maximum of 20 items. Your query returned 20 of many items. See https://python-gitlab.readthedocs.io/en/v3.15.0/api-usage.html#pagination for more details. If this was done intentionally, then this warning can be supressed by adding the argument `get_all=False` to the `list()` call. (python-gitlab: /cvmfs/lhcbdev.cern.ch/nightly-environments/5ced04d9e2e8ad4a44b36e93198a8c2f88c23ebb2313ab50b202d5e241e8cb8d/lib/python3.11/site-packages/gitlab/client.py:976)
commit = project.commits.list(until=ref_date, ref_name=branch)[0]
```4.0.2https://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/120Crash after failure to publish to gitlab in ci-test2023-09-27T15:16:53+02:00Marco Clemencicmarco.clemencic@cern.chCrash after failure to publish to gitlab in ci-testTrying to trigger a ci-test from Gaudi resulted in a crash of the script in the Jenkins job "main":
```py
2023-09-27 10:27:31,689:INFO : GitLab feedback: 'Started [integration test](https://lhcb-nightlies.web.cern.ch//nightly/lhcb-mas...Trying to trigger a ci-test from Gaudi resulted in a crash of the script in the Jenkins job "main":
```py
2023-09-27 10:27:31,689:INFO : GitLab feedback: 'Started [integration test](https://lhcb-nightlies.web.cern.ch//nightly/lhcb-master-mr/9321/) build. Once done, check the [results](https://lhcb-nightlies.web.cern.ch//nightly/lhcb-master-mr/9321/) or the [comparison](https://lhcb-nightlies.web.cern.ch//compare?a=nightly/lhcb-master/2173&b=nightly/lhcb-master-mr/9321) to a [reference](https://lhcb-nightlies.web.cern.ch//nightly/lhcb-master/2173/) build.'
2023-09-27 10:27:31,689:DEBUG : _gitlabServer invoked with ((), {})
2023-09-27 10:27:31,689:DEBUG : result in cache
2023-09-27 10:27:31,806:DEBUG : https://gitlab.cern.ch:443 "GET /api/v4/projects/38 HTTP/1.1" 200 4263
2023-09-27 10:27:32,022:DEBUG : https://gitlab.cern.ch:443 "GET /api/v4/projects/38/merge_requests/1495 HTTP/1.1" 200 4590
2023-09-27 10:27:32,132:DEBUG : https://gitlab.cern.ch:443 "GET /api/v4/projects/38/merge_requests/1495/discussions/3f0237d7981aa0c5c3be207c0ee97d96f8bc40f4 HTTP/1.1" 200 562
2023-09-27 10:27:32,332:DEBUG : https://gitlab.cern.ch:443 "POST /api/v4/projects/38/merge_requests/1495/discussions/3f0237d7981aa0c5c3be207c0ee97d96f8bc40f4/notes HTTP/1.1" 201 951
2023-09-27 10:27:32,393:DEBUG : https://gitlab.cern.ch:443 "PUT /api/v4/projects/38/merge_requests/1495 HTTP/1.1" 403 27
Traceback (most recent call last):
File "/cvmfs/lhcbdev.cern.ch/nightly-environments/5ced04d9e2e8ad4a44b36e93198a8c2f88c23ebb2313ab50b202d5e241e8cb8d/lib/python3.11/site-packages/gitlab/exceptions.py", line 336, in wrapped_f
return f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/cvmfs/lhcbdev.cern.ch/nightly-environments/5ced04d9e2e8ad4a44b36e93198a8c2f88c23ebb2313ab50b202d5e241e8cb8d/lib/python3.11/site-packages/gitlab/mixins.py", line 368, in update
result = http_method(path, post_data=new_data, files=files, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/cvmfs/lhcbdev.cern.ch/nightly-environments/5ced04d9e2e8ad4a44b36e93198a8c2f88c23ebb2313ab50b202d5e241e8cb8d/lib/python3.11/site-packages/gitlab/client.py", line 1075, in http_put
result = self.http_request(
^^^^^^^^^^^^^^^^^^
File "/cvmfs/lhcbdev.cern.ch/nightly-environments/5ced04d9e2e8ad4a44b36e93198a8c2f88c23ebb2313ab50b202d5e241e8cb8d/lib/python3.11/site-packages/gitlab/client.py", line 794, in http_request
raise gitlab.exceptions.GitlabHttpError(
gitlab.exceptions.GitlabHttpError: 403: 403 Forbidden
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/cvmfs/lhcbdev.cern.ch/nightly-environments/5ced04d9e2e8ad4a44b36e93198a8c2f88c23ebb2313ab50b202d5e241e8cb8d/lib/python3.11/site-packages/LbNightlyTools/MergeRequestBuilds.py", line 448, in post_gitlab_feedback
mr.save()
File "/cvmfs/lhcbdev.cern.ch/nightly-environments/5ced04d9e2e8ad4a44b36e93198a8c2f88c23ebb2313ab50b202d5e241e8cb8d/lib/python3.11/site-packages/gitlab/mixins.py", line 503, in save
server_data = self.manager.update(obj_id, updated_data, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/cvmfs/lhcbdev.cern.ch/nightly-environments/5ced04d9e2e8ad4a44b36e93198a8c2f88c23ebb2313ab50b202d5e241e8cb8d/lib/python3.11/site-packages/gitlab/exceptions.py", line 338, in wrapped_f
raise error(e.error_message, e.response_code, e.response_body) from e
gitlab.exceptions.GitlabUpdateError: 403: 403 Forbidden
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/cvmfs/lhcbdev.cern.ch/nightly-environments/5ced04d9e2e8ad4a44b36e93198a8c2f88c23ebb2313ab50b202d5e241e8cb8d/bin/lbn-enabled-slots", line 11, in <module>
sys.exit(enabled_slots())
^^^^^^^^^^^^^^^
File "/cvmfs/lhcbdev.cern.ch/nightly-environments/5ced04d9e2e8ad4a44b36e93198a8c2f88c23ebb2313ab50b202d5e241e8cb8d/lib/python3.11/site-packages/LbNightlyTools/Scripts/_entry_points.py", line 67, in enabled_slots
return Script().run()
^^^^^^^^^^^^^^
File "/cvmfs/lhcbdev.cern.ch/nightly-environments/5ced04d9e2e8ad4a44b36e93198a8c2f88c23ebb2313ab50b202d5e241e8cb8d/lib/python3.11/site-packages/LbNightlyTools/Scripts/Common.py", line 61, in run
return _PlainScript.run(self, args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/cvmfs/lhcbdev.cern.ch/nightly-environments/5ced04d9e2e8ad4a44b36e93198a8c2f88c23ebb2313ab50b202d5e241e8cb8d/lib/python3.11/site-packages/LbCommon/Script.py", line 107, in run
rc = self.main()
^^^^^^^^^^^
File "/cvmfs/lhcbdev.cern.ch/nightly-environments/5ced04d9e2e8ad4a44b36e93198a8c2f88c23ebb2313ab50b202d5e241e8cb8d/lib/python3.11/site-packages/LbNightlyTools/Scripts/EnabledSlots.py", line 333, in main
post_gitlab_feedback(
File "/cvmfs/lhcbdev.cern.ch/nightly-environments/5ced04d9e2e8ad4a44b36e93198a8c2f88c23ebb2313ab50b202d5e241e8cb8d/lib/python3.11/site-packages/LbNightlyTools/MergeRequestBuilds.py", line 451, in post_gitlab_feedback
logger.error("Could not post feedback to gitlab: " + e.message)
^^^^^^^^^
AttributeError: 'GitlabUpdateError' object has no attribute 'message'
Build step 'Execute shell' marked build as failure
Sending e-mails to: lhcb-core-soft-alarms@cern.ch
Finished: FAILURE
```4.0.2Marco Clemencicmarco.clemencic@cern.chMarco Clemencicmarco.clemencic@cern.chhttps://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/119Apptainer occasionally fails to start containers2023-09-27T15:13:46+02:00Rosen MatevApptainer occasionally fails to start containersApptainer occasionally fails to start containers with a pretty cryptic message
```
ERROR : Failed to get file information for file descriptor 3: Bad file descriptor
ERROR : Could not write info to setgroups: Permission denied
```
Thi...Apptainer occasionally fails to start containers with a pretty cryptic message
```
ERROR : Failed to get file information for file descriptor 3: Bad file descriptor
ERROR : Could not write info to setgroups: Permission denied
```
This is disruptive to testing as often we need to re-build (depending on which platform fails).
We should probably
1. Pass `--debug` to apptainer to get a sample of debug logs when the problem happens (to understand what this file descriptor is pointing to).
2. Detect and handle the failure to start the container. (In the second example below, the build of a project fails, and the install and the rest of the projects happily continue.)
3. Until the underlying problem is understood, we should probably retry running apptainer. We should consider if the retrial should be done on any error. The exit code is for example also 1 (the same as when apptainer fails to start) when the build (ninja) fails for a genuine problem.
Some related discussions: https://github.com/apptainer/apptainer/issues/430 https://github.com/apptainer/singularity/pull/4953 https://github.com/apptainer/singularity/issues/5206
### Examples
https://jenkins-lhcb-nightlies.web.cern.ch/job/nightly-builds/job/build/337143/consoleFull
```
2023-09-23 12:15:56,463:DEBUG : running cmake --install LHCb/build --prefix LHCb/InstallArea/x86_64_v2-centos7-clang12-opt
2023-09-23 12:15:56,463:DEBUG : apptainer command: /cvmfs/lhcbdev.cern.ch/nightly-environments/5ced04d9e2e8ad4a44b36e93198a8c2f88c23ebb2313ab50b202d5e241e8cb8d/bin/apptainer exec --contain --bind /cvmfs --bind /home/lblocal/jenkins-build/workspace/nightly-builds/build@2:/workspace --bind /home/lblocal/jenkins-build/workspace/nightly-builds/build@2 --pwd /workspace/build --env PATH=/cvmfs/lhcb.cern.ch/lib/bin/x86_64-centos7:/cvmfs/lhcb.cern.ch/lib/bin/x86_64-centos7:/cvmfs/lhcb.cern.ch/lib/bin/Linux-x86_64:/cvmfs/lhcb.cern.ch/lib/bin:/cvmfs/lhcbdev.cern.ch/nightly-environments/5ced04d9e2e8ad4a44b36e93198a8c2f88c23ebb2313ab50b202d5e241e8cb8d/bin:/cvmfs/lhcbdev.cern.ch/conda/miniconda/linux-64/1622055603/condabin:/usr/sue/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/opt/puppetlabs/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin /cvmfs/lhcb.cern.ch/containers/os-base/centos7-devel/prod/amd64 cmake --install LHCb/build --prefix LHCb/InstallArea/x86_64_v2-centos7-clang12-opt
2023-09-23 12:15:56,620:DEBUG : �[91mERROR : Failed to get file information for file descriptor 3: Bad file descriptor
2023-09-23 12:15:56,621:DEBUG : �[0m�[91mERROR : Could not write info to setgroups: Permission denied
2023-09-23 12:15:56,622:DEBUG : command exited with code 1
2023-09-23 12:15:56,622:DEBUG : Completed at: 2023-09-23 12:15:56.622240
```
https://jenkins-lhcb-nightlies.web.cern.ch/job/nightly-builds/job/build/337186/consoleFull
```
2023-09-23 19:48:31,458:DEBUG : running cmake --build Detector/build -j 10 -- -k0
2023-09-23 19:48:31,458:DEBUG : apptainer command: /cvmfs/lhcbdev.cern.ch/nightly-environments/5ced04d9e2e8ad4a44b36e93198a8c2f88c23ebb2313ab50b202d5e241e8cb8d/bin/apptainer exec --contain --bind /cvmfs --bind /home/lblocal/jenkins-build/workspace/nightly-builds/build@2:/workspace --bind /home/lblocal/jenkins-build/workspace/nightly-builds/build@2 --pwd /workspace/build --env PATH=/cvmfs/lhcb.cern.ch/lib/bin/x86_64-el9:/cvmfs/lhcb.cern.ch/lib/bin/x86_64-centos7:/cvmfs/lhcb.cern.ch/lib/bin/Linux-x86_64:/cvmfs/lhcb.cern.ch/lib/bin:/cvmfs/lhcbdev.cern.ch/nightly-environments/5ced04d9e2e8ad4a44b36e93198a8c2f88c23ebb2313ab50b202d5e241e8cb8d/bin:/cvmfs/lhcbdev.cern.ch/conda/miniconda/linux-64/1622055603/condabin:/usr/sue/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/opt/puppetlabs/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin /cvmfs/lhcb.cern.ch/containers/os-base/alma9-devel/prod/amd64 cmake --build Detector/build -j 10 -- -k0
2023-09-23 19:48:31,609:DEBUG : �[91mERROR : Failed to get file information for file descriptor 3: Bad file descriptor
2023-09-23 19:48:31,609:DEBUG : �[0m�[91mERROR : Could not write info to setgroups: Permission denied
2023-09-23 19:48:31,610:DEBUG : command exited with code 1
2023-09-23 19:48:31,615:DEBUG : running cmake --install Detector/build --prefix Detector/InstallArea/x86_64_v3-el9-gcc12+cuda12_1-opt+g
```
/cc @clemenci @cburr @sponce4.0.2Marco Clemencicmarco.clemencic@cern.chMarco Clemencicmarco.clemencic@cern.chhttps://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/118Do not assume that the default branch of a project is called "master"2023-09-26T17:38:39+02:00Marco Clemencicmarco.clemencic@cern.chDo not assume that the default branch of a project is called "master"The default branch of a Git repository is not anymore "master" by default, but it is user-defined and "main" is more common nowadays.
We should to change the logic of the scripts such that "HEAD" does not mean "master", but rather point...The default branch of a Git repository is not anymore "master" by default, but it is user-defined and "main" is more common nowadays.
We should to change the logic of the scripts such that "HEAD" does not mean "master", but rather points to the default branch.4.0.2Marco Clemencicmarco.clemencic@cern.chMarco Clemencicmarco.clemencic@cern.chhttps://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/117Fix use of deprecated networkx class2023-09-26T12:09:53+02:00Marco Clemencicmarco.clemencic@cern.chFix use of deprecated networkx classwe are using OrderedDiGraph which is deprecated and removed in networkx 3 (see https://networkx.org/documentation/networkx-2.8.8/reference/classes/ordered.html#networkx.OrderedDiGraph)we are using OrderedDiGraph which is deprecated and removed in networkx 3 (see https://networkx.org/documentation/networkx-2.8.8/reference/classes/ordered.html#networkx.OrderedDiGraph)4.0.2Marco Clemencicmarco.clemencic@cern.chMarco Clemencicmarco.clemencic@cern.chhttps://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/116Issues in the 4.0.0 release2023-09-29T03:44:36+02:00Chris BurrIssues in the 4.0.0 release
- [x] Missing `lbmessaging` dependency or out of date script?
```python
+ lbq-getteststorun --help
Traceback (most recent call last):
File "/home/cburr/miniconda3/conda-bld/lbnightlytools_1689087410658/_test_env_placehold_placehold_p...
- [x] Missing `lbmessaging` dependency or out of date script?
```python
+ lbq-getteststorun --help
Traceback (most recent call last):
File "/home/cburr/miniconda3/conda-bld/lbnightlytools_1689087410658/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_p/bin/lbq-getteststorun", line 11, in <module>
sys.exit(lbq_getteststorun())
File "/home/cburr/miniconda3/conda-bld/lbnightlytools_1689087410658/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_p/lib/python3.9/site-packages/LbNightlyTools/Scripts/_entry_points.py", line 719, in lbq_getteststorun
from lbmessaging.exchanges.Common import check_channel, get_connection
ModuleNotFoundError: No module named 'lbmessaging'
```
- [x] Use entrypoints instead of scripts4.0.2https://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/113Fix compatibility with Python 3.92023-06-26T14:11:54+02:00Marco Clemencicmarco.clemencic@cern.chFix compatibility with Python 3.9I'm trying to run the nightly script directly from LbEnv, which means using Python 3.9 and I encountered a number of issues I'm listing here
- [x] `cgi.escape` has been moved to `html.escape` (see https://docs.python.org/3/whatsnew/3.8....I'm trying to run the nightly script directly from LbEnv, which means using Python 3.9 and I encountered a number of issues I'm listing here
- [x] `cgi.escape` has been moved to `html.escape` (see https://docs.python.org/3/whatsnew/3.8.html#api-and-feature-removals)
- [x] LbNightlyTools/BuildMethods.py:492 tries to write a `str` to a file opened in binary mode
- [x] LbNightlyTools/BuildMethods.py:623 tries to join `bytes` and `str`
- [x] LbNightlyTools/BuildMethods.py:659 tries to join `bytes` and `str`
- [x] LbNightlyTools/Utils.py:733 -> `TypeError: a bytes-like object is required, not 'str'`
- [x] LbNightlyTools/Utils.py:823 -> infinite loop trying to compute md5sum because `iter` is invoked to loop over a binary file expecting and empty `str` as sentinel
- [x] LbRPMTools/LHCbRPMSpecBuilder.py:65 uses `cmp` that has been removed in Python 3, see https://docs.python.org/3.0/whatsnew/3.0.html#ordering-comparisonshttps://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/112Migrate from 2to3 to a unified code base2023-06-20T11:51:10+02:00Marco Clemencicmarco.clemencic@cern.chMigrate from 2to3 to a unified code baseWhile #111 can be fixed easily sticking to an older version of setuptools, it's still a good idea to refresh a bit the code making it compatible with Python 2 and 3 at the same time.
This does not mean fix the fact that the nightly buil...While #111 can be fixed easily sticking to an older version of setuptools, it's still a good idea to refresh a bit the code making it compatible with Python 2 and 3 at the same time.
This does not mean fix the fact that the nightly builds do not work under Python 3, but it could be a starting point to decommission Python 2 in the old version of the nightly builds.https://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/111Fix compatibility with setuptools 58 (or use an older version)2023-02-20T13:21:19+01:00Marco Clemencicmarco.clemencic@cern.chFix compatibility with setuptools 58 (or use an older version)The latest version of the container used for deployment contains setuptools 59.6.0 (from Python 3.6.15), but since 58.0.0 it does not allow `use_2to3` anymore. See https://setuptools.pypa.io/en/latest/history.html#v58-0-0
We could use a...The latest version of the container used for deployment contains setuptools 59.6.0 (from Python 3.6.15), but since 58.0.0 it does not allow `use_2to3` anymore. See https://setuptools.pypa.io/en/latest/history.html#v58-0-0
We could use an older version of Python (3.6.13 used to work) or move to a unified code base.https://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/110Allow selection of projects to run in ci-test2023-09-27T15:28:56+02:00Marco Clemencicmarco.clemencic@cern.chAllow selection of projects to run in ci-testSometime a MR only modifies files affecting one project (e.g. adding a test, modify a qmt file). In these cases the default logic of building and testing downstream projects is a waste of resources.
I should be possible to override the ...Sometime a MR only modifies files affecting one project (e.g. adding a test, modify a qmt file). In these cases the default logic of building and testing downstream projects is a waste of resources.
I should be possible to override the default logic used to select which projects to build or test with some "command line" option.https://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/109Optional LHCbPR tests with ci-test2023-06-15T06:48:05+02:00Rosen MatevOptional LHCbPR tests with ci-test## Motivation
Some LHCbPR tests are too expensive to run and/or don't make sense to be run on every ci-test. Therefore, it would be good to be able to define "optional" LHCbPR tests and have a way to run them as needed.
## User interfa...## Motivation
Some LHCbPR tests are too expensive to run and/or don't make sense to be run on every ci-test. Therefore, it would be good to be able to define "optional" LHCbPR tests and have a way to run them as needed.
## User interface
- An optional test does not run by default.
- Labeling any MR included in a build will enable running an optional test. The label is defined in the test schedule xml.
- Labeling should ideally work for non ci-test builds as well.
- For /ci-test, reference slots (`lhcb-master` or `lhcb-master-ref` with `/ci-test --build-reference ...`) can not be affected by labels (since a build could be a reference to many lhcb-master-mr builds). If an automatic comparison is necessary, the test schedule should be such that the tests always run for the reference slots.
## Implementation
### Declaring optional tests
```diff
<periodictest>
<schedule type="week" time="13:45">Mon,Tue,Wed,Thu,Fri,Sat,Sun</schedule>
<slot>lhcb-master-mr</slot>
+ <label>PR/hlt2_rate_and_size|PR/rate tests<label>
<project>Moore</project>
<platform>x86_64_v3-centos7-gcc11-opt+g</platform>
<test runner="lhcbpr" group="Moore_hlt2_rate_and_size" env="lb-run-force-platform|RateTestHandler"/>
<os_label>nightly-tests-j16</os_label>
<count>1</count>
</periodictest>
```
- `<label>` is a tag similar to `<slot>` that is regex that should match any MR label. In the example above, having a label `PR/rate tests` would trigger the optional test.
- Can the `<label>` tag be optional as to avoid adding `<label></label>` everywhere?
### Collect labels into the CouchDB document
- Example document [here](https://lhcb-nightlies.web.cern.ch/api/v1/nightly/lhcb-master-mr/5957/)
- Somewhere after [EnabledSlots.py#L156](https://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/blob/305a8d6d498cb00634c91078155f9a97b2432100/python/LbNightlyTools/Scripts/EnabledSlots.py#L156) we can collect the union of all labels of all MRs for all projects. We probably do not need to distinguish labels from different projects.
- We can put the list of labels into the CouchDB document under `config/metadata/labels`.
### Extend the matching logic when a build completes
- [this](https://gitlab.cern.ch/lhcb-core/nightlies-jenkins-scripts/-/blob/master/jenkins/periodic-tests/tests-pollqueue.sh) script runs in jenkins every 5 min
- it runs essentially [this code](https://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/blob/master/python/LbPeriodicTools/_entry_points.py#L112)
- in particular, [here](https://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/blob/master/python/LbPeriodicTools/_entry_points.py#L228-256) we check the available new builds against test_schedule2.xml
/cc @clemenci @maszyman @shunan @mstahlRosen MatevRosen Matevhttps://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/108Add protection against missing BUILD_URL environment variable2022-07-27T15:47:16+02:00Marco Clemencicmarco.clemencic@cern.chAdd protection against missing BUILD_URL environment variableWhen Jenkins gets upgraded the configured server URL is not correctly taken into account until the configuration is saved again by hand.
Unfortunately this results in the environment variable `BUILD_URL` not being set and we are implici...When Jenkins gets upgraded the configured server URL is not correctly taken into account until the configuration is saved again by hand.
Unfortunately this results in the environment variable `BUILD_URL` not being set and we are implicitly relaying on its presence with the net result that the test results are not correctly reported and unusable.https://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/103ci-test --platforms should not affect reference slot2021-10-06T14:38:27+02:00Rosen Matevci-test --platforms should not affect reference slotWhen a reference slot is started, it should always be done with all platforms. This is so that slots can be re-used.
If #102 is implemented or we switch to the new nightlies, this task will be obsolete.
/cc @chasseWhen a reference slot is started, it should always be done with all platforms. This is so that slots can be re-used.
If #102 is implemented or we switch to the new nightlies, this task will be obsolete.
/cc @chasseChristoph HasseChristoph Hassehttps://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/102Use lhcb-master instead of lhcb-master-ref as reference for ci-test2021-11-10T12:03:58+01:00Rosen MatevUse lhcb-master instead of lhcb-master-ref as reference for ci-testCurrently, building the lhcb-master-ref slots can take a lot of resources. We can probably use the latest lhcb-master build as reference without losing much. The lhcb-master-mr CouchDB document should reflect that accordingly.
It's desi...Currently, building the lhcb-master-ref slots can take a lot of resources. We can probably use the latest lhcb-master build as reference without losing much. The lhcb-master-mr CouchDB document should reflect that accordingly.
It's desirable, if not too hard, to keep the ability to also launch a new lhcb-master-ref slot with /ci-test (for the "branch" mode, as opposed to `--merge`).
/cc @chasse3.0.43Christoph HasseChristoph Hassehttps://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/101Make ci-test --merge the default mode2021-11-10T12:03:58+01:00Rosen MatevMake ci-test --merge the default modeAnd have the "branch" mode be selected with `--branch` or `--no-merge` (perhaps less good).
/cc @chasseAnd have the "branch" mode be selected with `--branch` or `--no-merge` (perhaps less good).
/cc @chasse3.0.43Christoph HasseChristoph Hassehttps://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/100Installation fails with setuptools 582021-09-20T10:15:16+02:00Chris BurrInstallation fails with setuptools 58Support for `use_2to3` has been removed:
```
Processing $SRC_DIR
Created temporary directory: /tmp/pip-req-build-9ibxyvp0
DEPRECATION: A future pip version will change local packages to be built in-place without first copying to a t...Support for `use_2to3` has been removed:
```
Processing $SRC_DIR
Created temporary directory: /tmp/pip-req-build-9ibxyvp0
DEPRECATION: A future pip version will change local packages to be built in-place without first copying to a temporary directory. We recommend you use --use-feature=in-tree-build to test your packages with this new behavior before it becomes the default.
pip 21.3 will remove support for this functionality. You can find discussion regarding this at https://github.com/pypa/pip/issues/7555.
Added file://$SRC_DIR to build tracker '/tmp/pip-req-tracker-igouiec2'
Running setup.py (path:/tmp/pip-req-build-9ibxyvp0/setup.py) egg_info for package from file://$SRC_DIR
Created temporary directory: /tmp/pip-pip-egg-info-bqzku7dh
Running command python setup.py egg_info
error in LbNightlyTools setup command: use_2to3 is invalid.
WARNING: Discarding file:///home/conda/feedstock_root/build_artifacts/lbnightlytools_1631892614564/work. Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
```Chris BurrChris Burrhttps://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/99versions override are not correctly taken into account with new CMake configu...2021-10-06T14:26:25+02:00Marco Clemencicmarco.clemencic@cern.chversions override are not correctly taken into account with new CMake configurationNew CMake configuration of projects do not fix the versions of the dependencies (at most there's a constraint on the minimum version), so CMake, by default, picks up the first compatible version it finds, which might not be the one we wa...New CMake configuration of projects do not fix the versions of the dependencies (at most there's a constraint on the minimum version), so CMake, by default, picks up the first compatible version it finds, which might not be the one we want to use for the slot.
For example, a slot configured to build LHCb/master on the released Gaudi/v35r4 is not built correctly as it picks up Gaudi v36r0 (the latest release at the moment).Marco Clemencicmarco.clemencic@cern.chMarco Clemencicmarco.clemencic@cern.ch