LbNightlyTools issueshttps://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues2023-06-23T19:27:30+02:00https://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/1Follow-up from "Fixes to setuptools configuration"2023-06-23T19:27:30+02:00Marco Clemencicmarco.clemencic@cern.chFollow-up from "Fixes to setuptools configuration"The following discussions from !181 should be addressed:
- [x] @clemenci started a [discussion](https://gitlab.cern.ch/lhcb-core/LbNightlyTools/merge_requests/181#note_1633585):
> We should run proper unit tests on vanilla Python ...The following discussions from !181 should be addressed:
- [x] @clemenci started a [discussion](https://gitlab.cern.ch/lhcb-core/LbNightlyTools/merge_requests/181#note_1633585):
> We should run proper unit tests on vanilla Python CI jobs
- [ ] @clemenci started a [discussion](https://gitlab.cern.ch/lhcb-core/LbNightlyTools/merge_requests/181#note_1633586): (+1 comment)
> We may think about using https://pypi.org/project/whichcraft/ instead
- [ ] @clemenci started a [discussion](https://gitlab.cern.ch/lhcb-core/LbNightlyTools/merge_requests/181#note_1633587):
> FIXME: to be adapted to new layout
- [ ] @clemenci started a [discussion](https://gitlab.cern.ch/lhcb-core/LbNightlyTools/merge_requests/181#note_1633831):
> FIXME: this test relies too much on conditions and the time of writing (not met anymore)
- [ ] @clemenci started a [discussion](https://gitlab.cern.ch/lhcb-core/LbNightlyTools/merge_requests/181#note_1633832):
> FIXME: to be adapted to new layout
- [ ] @clemenci started a [discussion](https://gitlab.cern.ch/lhcb-core/LbNightlyTools/merge_requests/181#note_1633833):
> FIXME: to review (required CouchDB connection?)
- [ ] @clemenci started a [discussion](https://gitlab.cern.ch/lhcb-core/LbNightlyTools/merge_requests/181#note_1633836):
> FIXME: to review (required CouchDB connection?)
- [ ] @clemenci started a [discussion](https://gitlab.cern.ch/lhcb-core/LbNightlyTools/merge_requests/181#note_1633837):
> FIXME: we need better way to manage this list... util we drop PARAMMarco Clemencicmarco.clemencic@cern.chMarco Clemencicmarco.clemencic@cern.chhttps://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/2If a build fails without error messages in the output it is reported as succe...2019-02-19T12:05:29+01:00Marco Clemencicmarco.clemencic@cern.chIf a build fails without error messages in the output it is reported as successfulAn environment problem caused a build to fail like this:
```
Started at: 2019-02-19 00:17:12.213075
running make -j10 -k all
/cvmfs/lhcb.cern.ch/lib/var/lib/LbEnv/306/dev/x86_64-centos7/lib/python2.7/site-packages/LbDevTools/data/Makefil...An environment problem caused a build to fail like this:
```
Started at: 2019-02-19 00:17:12.213075
running make -j10 -k all
/cvmfs/lhcb.cern.ch/lib/var/lib/LbEnv/306/dev/x86_64-centos7/lib/python2.7/site-packages/LbDevTools/data/Makefile-cmt-common.mk:188: *** Recursive variable `CMTPROJECTPATH' references itself (eventually). Stop.
command exited with code 2
Completed at: 2019-02-19 00:17:12.594313
Elapsed time: 0:00:00.381238
```
but the specific error message from `make` is not recognized and the exit code is ignored (assuming that you always get an error message), so the build was reported as successful.https://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/6Remove all svn references and usages in LbNightliyTools2019-07-15T11:06:54+02:00Stefan-Gabriel ChiticRemove all svn references and usages in LbNightliyToolshttps://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/8retried builds cause the same tests to be run multiple times2019-07-19T17:05:37+02:00Marco Clemencicmarco.clemencic@cern.chretried builds cause the same tests to be run multiple timesWhen a build is retried after an infrastructure failure, the "ready-builds" special entry in CouchDB (https://lhcb-couchdb.cern.ch/nightlies-nightly/ready-builds) contains multiple entries for the same test job (once per build).
I mitig...When a build is retried after an infrastructure failure, the "ready-builds" special entry in CouchDB (https://lhcb-couchdb.cern.ch/nightlies-nightly/ready-builds) contains multiple entries for the same test job (once per build).
I mitigated the problem with 226774b92f113e65ef71fc0f73fb92f25f23e243, bf474a24b10414ed22d78f8b8a34977da6bbbe6c (and the fix 119f64e02dfe02033fd27e98dae169ebc94821cb), but we should:
- [ ] clean the list of tests that are made obsolete by a restarted build (similar to what we do for the build logs)
- [ ] before running the test make sure it still make sense (test trigger time compatible with build completion time)
- [ ] before publishing the test results make sure they are not made obsolete by a retry (again. checking test and build times)https://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/12Follow-up from "WIP: Add options to resolve the MRs aliases in the main job"2019-10-12T16:34:30+02:00Marco Clemencicmarco.clemencic@cern.chFollow-up from "WIP: Add options to resolve the MRs aliases in the main job"The following discussion from !236 should be addressed:
- [ ] @clemenci started a [discussion](https://gitlab.cern.ch/lhcb-core/LbNightlyTools/merge_requests/236#note_2904629):
> The Gitlab related functions in Utils.py should be ...The following discussion from !236 should be addressed:
- [ ] @clemenci started a [discussion](https://gitlab.cern.ch/lhcb-core/LbNightlyTools/merge_requests/236#note_2904629):
> The Gitlab related functions in Utils.py should be moved here (and possibly adapted to use the new helper functions).https://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/15Follow-up from "Add support for new architectures"2019-12-09T17:33:16+01:00Marco Clemencicmarco.clemencic@cern.chFollow-up from "Add support for new architectures"The following discussion from !240 should be addressed:
- [ ] @clemenci started a [discussion](https://gitlab.cern.ch/lhcb-core/LbNightlyTools/merge_requests/240#note_2922444):
> once all nodes in the build farm are correctly labe...The following discussion from !240 should be addressed:
- [ ] @clemenci started a [discussion](https://gitlab.cern.ch/lhcb-core/LbNightlyTools/merge_requests/240#note_2922444):
> once all nodes in the build farm are correctly labeled, we can remove the special cases.https://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/20Add a build status badge to GitLab feedback2019-11-05T07:26:08+01:00Rosen MatevAdd a build status badge to GitLab feedbackWe have two types of feedback right now:
- `Validation started with lhcb-gaudi-head#2427` for nightly builds
- `Started reference and integration test builds. Once done, check the comparison of build and test results.` for ci-test builds...We have two types of feedback right now:
- `Validation started with lhcb-gaudi-head#2427` for nightly builds
- `Started reference and integration test builds. Once done, check the comparison of build and test results.` for ci-test builds
Both can profit from a badge integrated into the comment, which shows the progress of a build or a couple of builds.
The badge image can be served by either couchdb directly or the dashboard.Marco Clemencicmarco.clemencic@cern.chMarco Clemencicmarco.clemencic@cern.chhttps://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/22Investigate GitLab commit statuses for build feedback2019-10-24T23:08:19+02:00Rosen MatevInvestigate GitLab commit statuses for build feedbackSee https://about.gitlab.com/blog/2015/10/22/gitlab-8-1-released/#commit-status-api and
https://docs.gitlab.com/ce/api/commits.html#post-the-build-status-to-a-commitSee https://about.gitlab.com/blog/2015/10/22/gitlab-8-1-released/#commit-status-api and
https://docs.gitlab.com/ce/api/commits.html#post-the-build-status-to-a-commithttps://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/28Follow-up from "Add date field to CouchDB document from EnabledSlots"2019-11-12T14:21:07+01:00Marco Clemencicmarco.clemencic@cern.chFollow-up from "Add date field to CouchDB document from EnabledSlots"The following discussion from !256 should be addressed:
- [ ] @clemenci started a [discussion](https://gitlab.cern.ch/lhcb-core/LbNightlyTools/merge_requests/256#note_2978582):
> We can avoid that the checkout overrides the date d...The following discussion from !256 should be addressed:
- [ ] @clemenci started a [discussion](https://gitlab.cern.ch/lhcb-core/LbNightlyTools/merge_requests/256#note_2978582):
> We can avoid that the checkout overrides the date defined by the main job with something as simple as
> ```patch
> diff --git a/python/LbNightlyTools/Scripts/Common.py b/python/LbNightlyTools/Scripts/Common.py
> index 842bc66..cf27000 100755
> --- a/python/LbNightlyTools/Scripts/Common.py
> +++ b/python/LbNightlyTools/Scripts/Common.py
> @@ -324,6 +324,9 @@ class DashboardUpdate(object):
> d = {'_rev': d['_rev'], '_id': d['_id']}
> else:
> d = {}
> + if 'date' in d:
> + # do not override existing date
> + del data['date']
> return recursive_update(d, data)
> self.dashboard.update(self.doc_name, changes)
>
> ```
> but I'm not convinced it's what we want.
>
> The alternative is that *checkout* does not set the date, unless forced to (with a `--date` option).https://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/31Validate platforms in /ci-test2019-11-27T11:09:27+01:00Rosen MatevValidate platforms in /ci-testhttps://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/33Test job artifacts should overwrite old copies on EOS2019-12-06T01:38:18+01:00Marco Clemencicmarco.clemencic@cern.chTest job artifacts should overwrite old copies on EOStest artifacts are copied to EOS with `--no-overwrite`, which is wrong, because a test that was restarted (e.g. because of a glitch) should upload the latest version of the logstest artifacts are copied to EOS with `--no-overwrite`, which is wrong, because a test that was restarted (e.g. because of a glitch) should upload the latest version of the logshttps://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/35harmonize handling of BINARY_TAG/CMTCONFIG with lhcb-core/LbEnv!462019-12-03T09:59:32+01:00Marco Clemencicmarco.clemencic@cern.chharmonize handling of BINARY_TAG/CMTCONFIG with lhcb-core/LbEnv!46https://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/42Follow-up from "Disable projects not relevant for ci-test builds"2019-12-19T09:10:41+01:00Marco Clemencicmarco.clemencic@cern.chFollow-up from "Disable projects not relevant for ci-test builds"The following discussion from !271 should be addressed:
- [ ] @clemenci started a [discussion](https://gitlab.cern.ch/lhcb-core/LbNightlyTools/merge_requests/271#note_3078708):
> I'm not too convinced of the dependency on networkx...The following discussion from !271 should be addressed:
- [ ] @clemenci started a [discussion](https://gitlab.cern.ch/lhcb-core/LbNightlyTools/merge_requests/271#note_3078708):
> I'm not too convinced of the dependency on networkx here.
>
> I'm tempted to move it in `Checkout.py`, or to rely more on it in `Configuration.py`, for example for the topological sort needed during the build.https://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/44Extraneous acknowledgment emoji in comments with /ci-test2020-10-29T12:58:10+01:00Rosen MatevExtraneous acknowledgment emoji in comments with /ci-testComments that contain `/ci-test` that is not on a single line should not acknowledge with a robot emoji
see https://gitlab.cern.ch/gaudi/Gaudi/merge_requests/1026#note_3099465Comments that contain `/ci-test` that is not on a single line should not acknowledge with a robot emoji
see https://gitlab.cern.ch/gaudi/Gaudi/merge_requests/1026#note_3099465https://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/46Support for new convention of nightly builds gitlab labels2020-01-16T12:36:41+01:00Marco Clemencicmarco.clemencic@cern.chSupport for new convention of nightly builds gitlab labelsas discussed in https://gitlab.cern.ch/lhcb-core/LHCbNightlyConf/merge_requests/295#note_3105377, we want to
- [ ] use `nightly:*` instead of `all-slots`
- [ ] apply a MR labelled `nightly:some-name` to the slots `some-name`, `lhcb-some...as discussed in https://gitlab.cern.ch/lhcb-core/LHCbNightlyConf/merge_requests/295#note_3105377, we want to
- [ ] use `nightly:*` instead of `all-slots`
- [ ] apply a MR labelled `nightly:some-name` to the slots `some-name`, `lhcb-some-name` and `test-some-name`https://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/54Upgrade/change CouchDB Python bindings2020-01-31T18:01:06+01:00Marco Clemencicmarco.clemencic@cern.chUpgrade/change CouchDB Python bindingsWe currently use [CouchDB](https://pypi.org/project/CouchDB/) (unmaintained), but there are alternatives:
- [pycouchdb](https://pypi.org/project/pycouchdb/), obsolete but (optionally) used by [celery](https://pypi.org/project/celery/)
- ...We currently use [CouchDB](https://pypi.org/project/CouchDB/) (unmaintained), but there are alternatives:
- [pycouchdb](https://pypi.org/project/pycouchdb/), obsolete but (optionally) used by [celery](https://pypi.org/project/celery/)
- [CouchDB2](https://pypi.org/project/CouchDB2/)https://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/55Seg faulting Jenkins Job if it tries to checkout projects from svn/can't chec...2021-02-24T15:17:16+01:00Ross John HunterSeg faulting Jenkins Job if it tries to checkout projects from svn/can't checkout a projectThis issue is prompted by the recently nightlies failure, where `MooreAnalysis` was added to the nightly slots. It was not specified in `softdb` that this project should be checked-out from gitlab, and so we got this failure:
```bash
...This issue is prompted by the recently nightlies failure, where `MooreAnalysis` was added to the nightly slots. It was not specified in `softdb` that this project should be checked-out from gitlab, and so we got this failure:
```bash
2020-02-06 00:46:21,715:INFO : checking out MooreAnalysis/HEAD
2020-02-06 00:46:21,715:DEBUG : checkout_update
2020-02-06 00:46:21,715:DEBUG : with args (), {u'merges': [[1, u'3d5ff97d18889ea0ea66701ae35fe278b52a10af']], u'commit': u'42558722d6801c369b10fe09f193d59aa396ac22'}
2020-02-06 00:46:21,716:DEBUG : updating https://lhcb-couchdb.cern.ch/nightlies-nightly/lhcb-gaudi-head.2524
2020-02-06 00:46:21,716:DEBUG : Started at: 2020-02-06 00:46:21.716273
2020-02-06 00:46:21,717:DEBUG : getting source URI from SoftDB
2020-02-06 00:46:21,723:DEBUG : got MOOREANALYSIS#HEAD
Traceback (most recent call last):
File "/build/jenkins-tests/workspace/nightly-builds/checkout/venv/bin/lbn-checkout", line 14, in <module>
sys.exit(Script().run())
File "/build/jenkins-tests/workspace/nightly-builds/checkout/venv/lib/python2.7/site-packages/LbCommon/Script.py", line 107, in run
rc = self.main()
File "/build/jenkins-tests/workspace/nightly-builds/checkout/venv/lib/python2.7/site-packages/LbNightlyTools/Scripts/Checkout.py", line 203, in main
context=CheckoutContext)
File "/build/jenkins-tests/workspace/nightly-builds/checkout/venv/lib/python2.7/site-packages/LbNightlyTools/Configuration.py", line 1730, in checkout
results[project.name] = project.checkout(**kwargs)
File "/build/jenkins-tests/workspace/nightly-builds/checkout/venv/lib/python2.7/site-packages/LbNightlyTools/Configuration.py", line 265, in extend_kwargs_wrapper
return method(self, *args, **opts)
File "/build/jenkins-tests/workspace/nightly-builds/checkout/venv/lib/python2.7/site-packages/LbNightlyTools/Configuration.py", line 231, in log_enter_step_wrapper
return method(self, *args, **kwargs)
File "/build/jenkins-tests/workspace/nightly-builds/checkout/venv/lib/python2.7/site-packages/LbNightlyTools/Configuration.py", line 184, in log_recorder
result = method(self, *args, **kwargs)
File "/build/jenkins-tests/workspace/nightly-builds/checkout/venv/lib/python2.7/site-packages/LbNightlyTools/Configuration.py", line 208, in log_timing_wrapper
result = method(*args, **kwargs)
File "/build/jenkins-tests/workspace/nightly-builds/checkout/venv/lib/python2.7/site-packages/LbNightlyTools/CheckoutMethods.py", line 1018, in default
return getpack(proj, *args, **kwargs)
TypeError: getpack() got an unexpected keyword argument 'merges'
Build step 'Execute shell' marked build as failure
```
Ideally, if something like this were to happen, the job should fail gracefully, omit the offending project and build everything else as best as it can.
EDIT: (@clemenci)
There's two parts in this:
- [x] change the default so that new projects are assumed to come from Gitlab
- [ ] fix the script so that exceptions in checout tasks are properly handledhttps://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/56Non-descriptive failure in a job: job summary page doesn't match the log in t...2020-02-06T15:55:53+01:00Ross John HunterNon-descriptive failure in a job: job summary page doesn't match the log in the Jenkins JobIn this ci-test https://lhcb-nightlies.web.cern.ch/nightly/lhcb-master-mr/build/445/, the x86_64+avx2+fma-centos7-gcc9-opt build looks to have failed very badly, with the strange complaint that it cannot find a `CMakeLists.txt` when buil...In this ci-test https://lhcb-nightlies.web.cern.ch/nightly/lhcb-master-mr/build/445/, the x86_64+avx2+fma-centos7-gcc9-opt build looks to have failed very badly, with the strange complaint that it cannot find a `CMakeLists.txt` when building `Rec`: https://lhcb-nightlies.web.cern.ch/logs/build/nightly/lhcb-master-mr/445/x86_64+avx2+fma-centos7-gcc9-opt/Rec/.
However, when you look on the log of the Jenkins Job, it seems to start building `Rec` happily, with a failure about 58% into the build. Unfortunately, this failure is not described at all in the log of the Jenkins Job:
https://jenkins-lhcb-nightlies.web.cern.ch/job/nightly-builds/job/build/145574//consoleFull
Just making a note here in case anyone wants to investigate this behaviour. Ideally the failure should be explained in the Jenkins log, and it should agree with what we see in the LHCb Nightly build summary page. https://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/58Errors in commands not reported in the build summary2020-02-11T12:25:49+01:00Marco Clemencicmarco.clemencic@cern.chErrors in commands not reported in the build summaryIf a commands fail with a non-zero exit code, but does not print the magic keyword `error`, the problem is not reported in the build logs.
If there's no error printed, we should check for `[command exited with XYZ]`, printed by `lbn-wra...If a commands fail with a non-zero exit code, but does not print the magic keyword `error`, the problem is not reported in the build logs.
If there's no error printed, we should check for `[command exited with XYZ]`, printed by `lbn-wrapper`.https://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/61Convert shell scripts to Python entry-points2020-02-27T10:59:13+01:00Marco Clemencicmarco.clemencic@cern.chConvert shell scripts to Python entry-pointsThe following discussion from !296 should be addressed:
- @clemenci started a [discussion](https://gitlab.cern.ch/lhcb-core/LbNightlyTools/merge_requests/296#note_3221806):
> There's still 3 shell scripts that we have to convert t...The following discussion from !296 should be addressed:
- @clemenci started a [discussion](https://gitlab.cern.ch/lhcb-core/LbNightlyTools/merge_requests/296#note_3221806):
> There's still 3 shell scripts that we have to convert to Python first.
These are the scripts
- [ ] `scripts/lbn-get-configs` (trivial)
- [ ] `scripts/lbn-wrapcmd` (easy, but not sure it should be done)
- [ ] `scripts/lbpr-collect` (less easy, up to @maszyman to comment)