LbNightlyTools issueshttps://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues2021-03-03T21:07:26+01:00https://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/88Only allow ci-test trigger from top level comments2021-03-03T21:07:26+01:00Rosen MatevOnly allow ci-test trigger from top level commentsThis will ensure that a /ci-test trigger always follows the "system notes" that show the push events.
Also, it will be simpler to scan through the MR page and see all (or the last) ci-test triggers.This will ensure that a /ci-test trigger always follows the "system notes" that show the push events.
Also, it will be simpler to scan through the MR page and see all (or the last) ci-test triggers.https://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/124Follow-up from "Add workaround for transient apptainer failures"2023-09-27T15:13:00+02:00Marco Clemencicmarco.clemencic@cern.chFollow-up from "Add workaround for transient apptainer failures"The following discussion from !410 should be addressed:
- [ ] @clemenci started a [discussion](https://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/merge_requests/410#note_7154078):
> It would be nice to record somewhere that apptain...The following discussion from !410 should be addressed:
- [ ] @clemenci started a [discussion](https://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/merge_requests/410#note_7154078):
> It would be nice to record somewhere that apptainer failed to start.
>
> As it is handled here we would have to scan Jenkins logs to find how often apptainer fails.https://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/123Follow-up from "Fix GaussinoExtLibs gitlab group and clean up"2023-09-27T14:55:45+02:00Marco Clemencicmarco.clemencic@cern.chFollow-up from "Fix GaussinoExtLibs gitlab group and clean up"The following discussion from !413 should be addressed:
- [ ] @clemenci started a [discussion](https://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/merge_requests/413#note_7153944):
> This should check if the project already declares...The following discussion from !413 should be addressed:
- [ ] @clemenci started a [discussion](https://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/merge_requests/413#note_7153944):
> This should check if the project already declares a Git URL in checkout_options and use that instead of guessing.https://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/115Use uniform convention between bytes and str in subprocess outputs2023-06-26T13:36:48+02:00Marco Clemencicmarco.clemencic@cern.chUse uniform convention between bytes and str in subprocess outputsThe following discussion from !399 should be addressed:
- [ ] @clemenci started a [discussion](https://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/merge_requests/399#note_6856043):
> I made this change because I found a problem in a...The following discussion from !399 should be addressed:
- [ ] @clemenci started a [discussion](https://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/merge_requests/399#note_6856043):
> I made this change because I found a problem in a checkout Jenkins job with result objects using not consistently bytes and str. Only later I realized I fixed the same issue with the exact opposite convention in other places (and tests).
>
> Since this change made the test build work I decided to keep it instead of trying to unify the conventions.https://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/114Test data package MRs with ci-test2023-05-08T22:56:20+02:00Rosen MatevTest data package MRs with ci-testRight now if we do /ci-test from a data package MR (without project MRs) we don't have a useful build as every project is disabled: https://lhcb-nightlies.web.cern.ch//nightly/lhcb-master-mr/7803/
It would be nice to do something useful...Right now if we do /ci-test from a data package MR (without project MRs) we don't have a useful build as every project is disabled: https://lhcb-nightlies.web.cern.ch//nightly/lhcb-master-mr/7803/
It would be nice to do something useful in that case. For example,
1. build all projects (easier)
2. build projects explicitly given on the "command line"https://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/107prevent projects to be added twice to the list of projects2022-04-08T16:57:12+02:00Marco Clemencicmarco.clemencic@cern.chprevent projects to be added twice to the list of projectsThis is currently working, but should fail:
```python
from LbNightlyTools.Configuration import Slot, DBASE
s = Slot("test", projects=[DBASE()])
s.projects.append(DBASE()) # we should get and exception
print([p.name for p in s.projects])...This is currently working, but should fail:
```python
from LbNightlyTools.Configuration import Slot, DBASE
s = Slot("test", projects=[DBASE()])
s.projects.append(DBASE()) # we should get and exception
print([p.name for p in s.projects]) # this prints ['DBASE', 'DBASE']
```
(see lhcb-core/LHCbNightlyConf!810)https://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/106Remove filtering of checkout log2022-03-31T17:48:45+02:00Rosen MatevRemove filtering of checkout logIt seems somewhere in the system the string `/workspace/build/` is replaced with an empty string. The jenkins logs have it but not the checkout logs on the nightly dashboard.
This is pretty misleading so I suggest not doing it unless th...It seems somewhere in the system the string `/workspace/build/` is replaced with an empty string. The jenkins logs have it but not the checkout logs on the nightly dashboard.
This is pretty misleading so I suggest not doing it unless there is a very good reason.
Probably the same applies to the new nightlies.https://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/105lbn-install: allow forcing HTTP instead of EOS2021-12-13T16:47:22+01:00Marco Clemencicmarco.clemencic@cern.chlbn-install: allow forcing HTTP instead of EOSAfter lhcb-core/LbNightlyTools!298 `lbn-install` uses EOS by default, but that works only for users in the right AFS/Kerberos group. For unauthenticated installations we need to be able to force HTTP.After lhcb-core/LbNightlyTools!298 `lbn-install` uses EOS by default, but that works only for users in the right AFS/Kerberos group. For unauthenticated installations we need to be able to force HTTP.https://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/104Fail ci-test when project version is different than target branch2024-01-24T09:38:11+01:00Rosen MatevFail ci-test when project version is different than target branchE.g. in https://lhcb-nightlies.web.cern.ch/nightly/lhcb-master-mr/3276/ a Gaudi MR was merged into a pinned Gaudi v36r2.
Check can happen for example before https://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/blob/b2d43d2f8c03edfd989055e4...E.g. in https://lhcb-nightlies.web.cern.ch/nightly/lhcb-master-mr/3276/ a Gaudi MR was merged into a pinned Gaudi v36r2.
Check can happen for example before https://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/blob/b2d43d2f8c03edfd989055e47d2712fee294b4a0/python/LbNightlyTools/MergeRequestBuilds.py#L323https://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/91Treat "#pragma message:" compiler printouts as warnings2021-05-21T12:23:17+02:00Marco Clemencicmarco.clemencic@cern.chTreat "#pragma message:" compiler printouts as warningsBoost uses `#pragma message` to print warnings, but they do not write `warning` in the message, so they are not spot by our parser and might be missed.Boost uses `#pragma message` to print warnings, but they do not write `warning` in the message, so they are not spot by our parser and might be missed.https://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/83Follow-up from "Fixes for new stack definitions"2021-02-24T15:13:09+01:00Marco Clemencicmarco.clemencic@cern.chFollow-up from "Fixes for new stack definitions"The following discussion from !335 should be addressed:
- [ ] @bcouturi started a [discussion](https://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/merge_requests/335#note_4218746): (+1 comment)
> It's ok for the moment but we should...The following discussion from !335 should be addressed:
- [ ] @bcouturi started a [discussion](https://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/merge_requests/335#note_4218746): (+1 comment)
> It's ok for the moment but we should parametrize this/have our own copy at some pointhttps://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/81ci-test should not start if any MR fails to merge2023-08-04T11:10:37+02:00Rosen Matevci-test should not start if any MR fails to mergeConsider the ci-test started at https://gitlab.cern.ch/lhcb/Moore/-/merge_requests/744#note_4212214
Moore!748 had conflicts but nevertheless the ci-test started.
A more complicated case to consider is if MR A and MR B have no conflicts...Consider the ci-test started at https://gitlab.cern.ch/lhcb/Moore/-/merge_requests/744#note_4212214
Moore!748 had conflicts but nevertheless the ci-test started.
A more complicated case to consider is if MR A and MR B have no conflicts wrt the target but merging B cannot be done after merging A. This might only be practical to check at checkout time, so we might need a "strict" checkout mode that aborts if any MR fails to merge.https://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/80Link to the actual commits used for ci-test and "validation started" messages2021-02-16T10:15:11+01:00Rosen MatevLink to the actual commits used for ci-test and "validation started" messagesWith this we should be able to more easily answer the question "what was changed since this was tested in this build".
we can link to commits or the corresponding version in the diff MR pageWith this we should be able to more easily answer the question "what was changed since this was tested in this build".
we can link to commits or the corresponding version in the diff MR pagehttps://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/77Better ci-test reference without --merge2020-12-17T09:31:41+01:00Rosen MatevBetter ci-test reference without --mergeWe should only consider merge commits when we construct a reference for a ci-test without `--merge`. See this counter example which shows this is not the case:
https://lhcb-nightlies.web.cern.ch/logs/checkout/nightly/lhcb-master-ref/111...We should only consider merge commits when we construct a reference for a ci-test without `--merge`. See this counter example which shows this is not the case:
https://lhcb-nightlies.web.cern.ch/logs/checkout/nightly/lhcb-master-ref/1112/LHCb/
```
1 Started at: 2020-12-16 18:10:03.053706
2 cloning git repository https://gitlab.cern.ch/lhcb/LHCb.git
3 (/run/user/25133/checkout-69889)$ 'git' 'clone' '--no-checkout' 'https://gitlab.cern.ch/lhcb/LHCb.git' u'LHCb/'
4 Cloning into 'LHCb'...
5 getting merge-requests branches
6 (/run/user/25133/checkout-69889/LHCb)$ 'git' 'config' '--add' 'remote.origin.fetch' '+refs/merge-requests/*/head:refs/remotes/origin/merge-requests/*'
7 (/run/user/25133/checkout-69889/LHCb)$ 'git' 'fetch' '-q' 'origin'
8 checking out LHCb/master from https://gitlab.cern.ch/lhcb/LHCb.git (4039310afc2abec7bce571b0a3ca20d88ce5d49a)
9 checkout commit 4039310afc2abec7bce571b0a3ca20d88ce5d49a for LHCb/master
10 (/run/user/25133/checkout-69889/LHCb)$ 'git' 'checkout' u'4039310afc2abec7bce571b0a3ca20d88ce5d49a'
11 Note: switching to '4039310afc2abec7bce571b0a3ca20d88ce5d49a'.
```https://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/73Better exception handling in ci-test2023-06-09T10:01:00+02:00Rosen MatevBetter exception handling in ci-testWhen the model slot cannot be determined as in [this job](https://jenkins-lhcb-nightlies.web.cern.ch/job/nightly-builds/job/main/4675/console), we should fail in a nice way by catching the exception and replying to the discussion. At the...When the model slot cannot be determined as in [this job](https://jenkins-lhcb-nightlies.web.cern.ch/job/nightly-builds/job/main/4675/console), we should fail in a nice way by catching the exception and replying to the discussion. At the moment one just sees the failure in Jenkins:
```
2020-06-23 11:03:21,465:DEBUG : https://gitlab.cern.ch:443 "GET //api/v4/projects/lhcb%2FLHCb HTTP/1.1" 200 2718
2020-06-23 11:03:22,080:DEBUG : https://gitlab.cern.ch:443 "GET //api/v4/projects/399/merge_requests/2552 HTTP/1.1" 200 2196
2020-06-23 11:03:22,081:INFO : Getting model slot
2020-06-23 11:03:22,081:ERROR : Cannot determine Slot configuration from target branch "reco14-patches".
Traceback (most recent call last):
File "/build/jenkins-tests/workspace/nightly-builds/main/venv/bin/lbn-enabled-slots", line 8, in <module>
sys.exit(enabled_slots())
File "/build/jenkins-tests/workspace/nightly-builds/main/venv/lib/python2.7/site-packages/LbNightlyTools/Scripts/_entry_points.py", line 54, in enabled_slots
return Script().run()
File "/build/jenkins-tests/workspace/nightly-builds/main/venv/lib/python2.7/site-packages/LbCommon/Script.py", line 107, in run
rc = self.main()
File "/build/jenkins-tests/workspace/nightly-builds/main/venv/lib/python2.7/site-packages/LbNightlyTools/Scripts/EnabledSlots.py", line 140, in main
ref_slot, mr_slot = make_mr_slots(mr_slots_config, slots)
File "/build/jenkins-tests/workspace/nightly-builds/main/venv/lib/python2.7/site-packages/LbNightlyTools/MergeRequestBuilds.py", line 369, in make_mr_slots
model_slots=model_slots)
File "/build/jenkins-tests/workspace/nightly-builds/main/venv/lib/python2.7/site-packages/LbNightlyTools/MergeRequestBuilds.py", line 279, in create_mr_slots
model_slot = get_model_slot(target_branch, model_slots)
File "/build/jenkins-tests/workspace/nightly-builds/main/venv/lib/python2.7/site-packages/LbNightlyTools/MergeRequestBuilds.py", line 182, in get_model_slot
raise NotImplementedError(message)
NotImplementedError: Cannot determine Slot configuration from target branch "reco14-patches".
```https://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/67lbn-get-new-refs broken with python 32020-04-28T10:19:34+02:00Rosen Matevlbn-get-new-refs broken with python 3```
echo $LBENV_SOURCED
# 2.0.2
CMTCONFIG=x86_64+avx2+fma-centos7-gcc9-opt BINARY_TAG=$CMTCONFIG lbn-get-new-refs lhcb-master 1088 Moore
```
gives this
```
looking for slot: lhcb-master, day: 1088, app: Moore, platform: x86_64+avx2+fma-c...```
echo $LBENV_SOURCED
# 2.0.2
CMTCONFIG=x86_64+avx2+fma-centos7-gcc9-opt BINARY_TAG=$CMTCONFIG lbn-get-new-refs lhcb-master 1088 Moore
```
gives this
```
looking for slot: lhcb-master, day: 1088, app: Moore, platform: x86_64+avx2+fma-centos7-gcc9-opt
Getting data from: https://lhcb-nightlies-artifacts.web.cern.ch/lhcb-nightlies-artifacts/nightly/lhcb-master/1088/tests/x86_64+avx2+fma-centos7-gcc9-opt/newrefs/Moore.zip
Traceback (most recent call last):
File "/cvmfs/lhcb.cern.ch/lib/var/lib/LbEnv/779/unstable/linux-64/bin/lbn-get-new-refs", line 8, in <module>
sys.exit(main())
File "/cvmfs/lhcb.cern.ch/lib/var/lib/LbEnv/779/unstable/linux-64/lib/python3.8/site-packages/LbNightlyTools/GetNightlyRefs.py", line 132, in main
arch = ZipFile(StringIO(arch_data))
TypeError: initial_value must be str or None, not bytes
```https://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/66Build of datapackages fail when binary tag is not set2020-04-14T14:49:06+02:00Marco Clemencicmarco.clemencic@cern.chBuild of datapackages fail when binary tag is not setfor data packages the build happens during the checkout phase, so it's relying on the default platform defined in that phase (anyway they are platform independent).
With the latest `LbEnv` there's no *default platform*, and the build of...for data packages the build happens during the checkout phase, so it's relying on the default platform defined in that phase (anyway they are platform independent).
With the latest `LbEnv` there's no *default platform*, and the build of the data packages uses CMT, so they fail with something like:
```
(DBASE/ProdConf/v3r1/cmt)$ 'cmt' 'make'
#CMT---> Info: Execute action make => gmake bin=..//
#CMT---> (Makefile.header) Rebuilding ..//.make
/bin/sh: line 3: 5738 Segmentation fault /cvmfs/lhcb.cern.ch/lib/contrib/CMT/v1r20p20090520/Linux-x86_64/cmt.exe -tag=, build tag_makefile > ..//.make
#CMT---> (Makefile.header) Rebuilding ..//setup.make
/bin/sh: line 7: 5744 Segmentation fault /cvmfs/lhcb.cern.ch/lib/contrib/CMT/v1r20p20090520/Linux-x86_64/cmt.exe -tag=, show setup > ..//setup$$.make
#CMT---> (Makefile.header) Rebuilding ..//constituents.make
/bin/sh: line 7: 5751 Segmentation fault /cvmfs/lhcb.cern.ch/lib/contrib/CMT/v1r20p20090520/Linux-x86_64/cmt.exe -tag=, build constituents_makefile -out=..//constituents.make
#CMT> Error: ..//constituents.make: Cannot generate
gmake[2]: ..//constituents.make: No such file or directory
gmake[2]: *** No rule to make target `..//constituents.make'. Stop.
gmake[1]: *** [common_target] Error 2
gmake: *** [check_config] Error 2
#CMT---> Error: execution_failed : make
```
which, is not reported in the summary page, because of the *not* handling of checkout logs.Marco Clemencicmarco.clemencic@cern.chMarco Clemencicmarco.clemencic@cern.chhttps://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/64Investigate the possibility of using ansi2html instead of the custom solution2020-03-11T23:25:26+01:00Marco Clemencicmarco.clemencic@cern.chInvestigate the possibility of using ansi2html instead of the custom solutionSome time ago I developed the `ANSI2HTML` and `XTerm2HTML` classes, but now I discovered https://github.com/ralphbean/ansi2htmlSome time ago I developed the `ANSI2HTML` and `XTerm2HTML` classes, but now I discovered https://github.com/ralphbean/ansi2htmlhttps://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/63Specialize "validation started" notifications for ci-test2021-06-28T13:35:52+02:00Rosen MatevSpecialize "validation started" notifications for ci-testThe recent improvement to the feedback messages (lhcb-core/LbNightlyTools!285) makes the "validation started" messages appear in a separate thread even for ci-test triggers. See e.g.g https://gitlab.cern.ch/lhcb/Rec/merge_requests/1941#n...The recent improvement to the feedback messages (lhcb-core/LbNightlyTools!285) makes the "validation started" messages appear in a separate thread even for ci-test triggers. See e.g.g https://gitlab.cern.ch/lhcb/Rec/merge_requests/1941#note_3239170
We should improve thishttps://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/62lbn-install does not create the right symlinks for CMT projects2020-02-27T15:12:02+01:00Marco Clemencicmarco.clemencic@cern.chlbn-install does not create the right symlinks for CMT projectsThe fix !213 to the problem introduced in !210 only takes into account CMake projects by creating symlinks like `LHCb_master` for `LHCb`, but for CMT we need `LHCB/LHCB_master`.
See https://lhcb-talk.web.cern.ch/t/how-to-locally-install...The fix !213 to the problem introduced in !210 only takes into account CMake projects by creating symlinks like `LHCb_master` for `LHCb`, but for CMT we need `LHCB/LHCB_master`.
See https://lhcb-talk.web.cern.ch/t/how-to-locally-install-a-nightly-slot-not-available-on-cvmfs/254Marco Clemencicmarco.clemencic@cern.chMarco Clemencicmarco.clemencic@cern.ch