LbNightlyTools issueshttps://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues2019-12-13T15:06:10+01:00https://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/38reference slot metadata in "mr" slots not added2019-12-13T15:06:10+01:00Marco Clemencicmarco.clemencic@cern.chreference slot metadata in "mr" slots not addedit should have been introduced with !266, but the implementation is wrongit should have been introduced with !266, but the implementation is wrong2.2.2Marco Clemencicmarco.clemencic@cern.chMarco Clemencicmarco.clemencic@cern.chhttps://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/37test conversion to JSON in lbn-check-config2019-12-11T10:23:55+01:00Marco Clemencicmarco.clemencic@cern.chtest conversion to JSON in lbn-check-configA key step in the use of the code in LHCbNightlyConf is converting the Slot instances to JSON (to store them on CouchDB).
A perfectly valid Python dictionary might not be JSON-serializable, so we have to check for it.A key step in the use of the code in LHCbNightlyConf is converting the Slot instances to JSON (to store them on CouchDB).
A perfectly valid Python dictionary might not be JSON-serializable, so we have to check for it.2.2.1Marco Clemencicmarco.clemencic@cern.chMarco Clemencicmarco.clemencic@cern.chhttps://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/36do not run check-formatting for tags2019-12-11T10:32:34+01:00Marco Clemencicmarco.clemencic@cern.chdo not run check-formatting for tags2.2.1Marco Clemencicmarco.clemencic@cern.chMarco Clemencicmarco.clemencic@cern.chhttps://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/34Fix LBCORE-1836: getNightlyRefs does not respect BINARY_TAG2019-12-03T09:53:00+01:00Marco Clemencicmarco.clemencic@cern.chFix LBCORE-1836: getNightlyRefs does not respect BINARY_TAG2.2.0Marco Clemencicmarco.clemencic@cern.chMarco Clemencicmarco.clemencic@cern.chhttps://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/32Disable projects not relevant for ci-test builds2020-01-17T12:49:15+01:00Rosen MatevDisable projects not relevant for ci-test buildsFor example, if we trigger a `/ci-test` from a Moore MR (without any other MRs), we don't need to build and test DaVinci. The deps are known at checkout time, which is when we can detect what can be disabled and do itFor example, if we trigger a `/ci-test` from a Moore MR (without any other MRs), we don't need to build and test DaVinci. The deps are known at checkout time, which is when we can detect what can be disabled and do it2.2.3Marco Clemencicmarco.clemencic@cern.chMarco Clemencicmarco.clemencic@cern.chhttps://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/30/ci-test --merge to use a complete set of platforms2020-02-26T10:55:33+01:00Rosen Matev/ci-test --merge to use a complete set of platforms`/ci-test --merge` is supposed to replace the testing of MRs one at a time in a slot like `lhcb-gaudi-head`, and thus should test the complete set of platforms.
Must be done after lhcb-core/LbNightlyTools#19 to minimize the load on the...`/ci-test --merge` is supposed to replace the testing of MRs one at a time in a slot like `lhcb-gaudi-head`, and thus should test the complete set of platforms.
Must be done after lhcb-core/LbNightlyTools#19 to minimize the load on the build cluster.https://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/29lbn-install does not undestand latest index page from EOSWEB2019-11-20T09:57:41+01:00Marco Clemencicmarco.clemencic@cern.chlbn-install does not undestand latest index page from EOSWEBSee https://lblogbook.cern.ch/Core+Software+processes+and+tools/277See https://lblogbook.cern.ch/Core+Software+processes+and+tools/2772.1.4Marco Clemencicmarco.clemencic@cern.chMarco Clemencicmarco.clemencic@cern.chhttps://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/27Add date field to CouchDB document created by main Jenkins job2019-11-20T15:02:59+01:00Marco Clemencicmarco.clemencic@cern.chAdd date field to CouchDB document created by main Jenkins jobWith the correct date field, the slot will appear in the summary page as soon as the Jenkins *main* job creates the document (at the moment we have to wait the the checkout job updates the document).With the correct date field, the slot will appear in the summary page as soon as the Jenkins *main* job creates the document (at the moment we have to wait the the checkout job updates the document).2.2.0Marco Clemencicmarco.clemencic@cern.chMarco Clemencicmarco.clemencic@cern.chhttps://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/26Apply LHCb formatting to Python code2019-12-09T17:43:29+01:00Marco Clemencicmarco.clemencic@cern.chApply LHCb formatting to Python code2.2.0Marco Clemencicmarco.clemencic@cern.chMarco Clemencicmarco.clemencic@cern.chhttps://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/25process hangs when getting errors from couchdb2019-11-11T11:25:21+01:00Marco Clemencicmarco.clemencic@cern.chprocess hangs when getting errors from couchdbIf there's a temporary problem with httpd or couchdb, the build job may hang in some cases (e.g. while exiting for an unhandled exception).
The stack trace we got in one of those cases was:
```py
Exception in thread Thread-1:
Traceback ...If there's a temporary problem with httpd or couchdb, the build job may hang in some cases (e.g. while exiting for an unhandled exception).
The stack trace we got in one of those cases was:
```py
Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner
self.run()
File "/usr/lib64/python2.7/threading.py", line 765, in run
self.__target(*self.__args, **self.__kwargs)
File "/workspace/venv/centos7/lib/python2.7/site-packages/LbNightlyTools/Utils.py", line 525, in worker
action(*args, **kwargs)
File "/workspace/venv/centos7/lib/python2.7/site-packages/LbNightlyTools/Scripts/Common.py", line 679, in send
receiver.accept(msg)
File "/workspace/venv/centos7/lib/python2.7/site-packages/LbNightlyTools/Scripts/Common.py", line 498, in accept
self.project_build_completed(msg)
File "/workspace/venv/centos7/lib/python2.7/site-packages/LbNightlyTools/Scripts/Common.py", line 439, in project_build_completed
self.dashboard.update(self.doc_name, data)
File "/workspace/venv/centos7/lib/python2.7/site-packages/LbNightlyTools/Utils.py", line 487, in update
self.db[key] = new
File "/workspace/venv/centos7/lib/python2.7/site-packages/couchdb/client.py", line 447, in __setitem__
status, headers, data = resource.put_json(body=content)
File "/workspace/venv/centos7/lib/python2.7/site-packages/couchdb/http.py", line 578, in put_json
**params)
File "/workspace/venv/centos7/lib/python2.7/site-packages/couchdb/http.py", line 596, in _request_json
headers=headers, **params)
File "/workspace/venv/centos7/lib/python2.7/site-packages/couchdb/http.py", line 592, in _request
credentials=self.credentials)
File "/workspace/venv/centos7/lib/python2.7/site-packages/couchdb/http.py", line 429, in request
raise ServerError((status, error))
ServerError: (500, (u'unknown_error', u'undefined'))
```2.1.3Marco Clemencicmarco.clemencic@cern.chMarco Clemencicmarco.clemencic@cern.chhttps://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/24Add metadata field to Slot class/configuration dict2019-12-11T11:00:13+01:00Marco Clemencicmarco.clemencic@cern.chAdd metadata field to Slot class/configuration dictadding a dict to the slot as *metadata* would be extremely useful to record information that are not strictly related to how to build the slot, but could be used to enhance what we display on web pages.
For example we could have:
- a fl...adding a dict to the slot as *metadata* would be extremely useful to record information that are not strictly related to how to build the slot, but could be used to enhance what we display on web pages.
For example we could have:
- a flag to state that the slot should not appear in the main summaries page (e.g. ref and mr slots are not very useful in that page)
- record the name and id of reference slot build for a MR (/ci-test) job
In principle, also the description of a slot would go in the metadata section, but it can be done at a later stage.2.2.0Marco Clemencicmarco.clemencic@cern.chMarco Clemencicmarco.clemencic@cern.chhttps://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/23Cross reference ref and mr slots in descriptions2019-12-16T15:39:42+01:00Rosen MatevCross reference ref and mr slots in descriptionsIt would be nice to be able to navigate between the linked ref and MR slots for a ci-test.
In the main job, once build ids are acquired, we can update the descriptions of the two slots to reference each other and save the document again.It would be nice to be able to navigate between the linked ref and MR slots for a ci-test.
In the main job, once build ids are acquired, we can update the descriptions of the two slots to reference each other and save the document again.2.2.2Marco Clemencicmarco.clemencic@cern.chMarco Clemencicmarco.clemencic@cern.chhttps://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/21Hide comparison of checkout with previous slot build in '/ci-test' jobs2019-12-12T10:06:25+01:00Marco Clemencicmarco.clemencic@cern.chHide comparison of checkout with previous slot build in '/ci-test' jobsCurrently we show in checkout logs which commits have bee removed/added since the previous build.
While it make sense for regular slots, it is useless for slots triggered by the '/ci-test' command, where the previous build could be comp...Currently we show in checkout logs which commits have bee removed/added since the previous build.
While it make sense for regular slots, it is useless for slots triggered by the '/ci-test' command, where the previous build could be completely unrelated.
What we need is:
* [x] flag the slot as special via metadata (#24)
* [x] extend checkout logic to avoid the comparison for the special slots2.2.1Marco Clemencicmarco.clemencic@cern.chMarco Clemencicmarco.clemencic@cern.chhttps://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/19Do not create slots that exist already2019-12-11T13:12:54+01:00Rosen MatevDo not create slots that exist alreadyToday couchdb knows everything that fully describes a slot. In the main job we should check if the slot we are trying to create does not already exist.
- We should decide what identifies a slot. Clearly the list of platforms, projects a...Today couchdb knows everything that fully describes a slot. In the main job we should check if the slot we are trying to create does not already exist.
- We should decide what identifies a slot. Clearly the list of platforms, projects and their versions, but maybe also the versions of some of the tools? (e.g. consider a build failing because of issue in checkout logic)
- We should have a way to query couchdb for the existence of a slot. Iterating over all slots or something hash based?2.2.0https://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/18Ability to abort /ci-test slots2020-10-12T10:57:39+02:00Rosen MatevAbility to abort /ci-test slotsWe need to
1. Provide a url which when clicked sets the abort flag in couchdb.
1. Poll from checkout/build/tests jobs whenever possible and abort.
1. Publish an abort link in the feedback from the main job.
1. Hide aborted slots from the...We need to
1. Provide a url which when clicked sets the abort flag in couchdb.
1. Poll from checkout/build/tests jobs whenever possible and abort.
1. Publish an abort link in the feedback from the main job.
1. Hide aborted slots from the dashboard
1. Adapt the logic for reusing slots such that the check fails for aborted slotsRosen MatevRosen Matevhttps://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/17The checkout from '/ci-test' fails if the merge request comes from a fork2019-10-24T16:45:53+02:00Marco Clemencicmarco.clemencic@cern.chThe checkout from '/ci-test' fails if the merge request comes from a fork2.1.1Marco Clemencicmarco.clemencic@cern.chMarco Clemencicmarco.clemencic@cern.chhttps://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/16Migrate to pytest2023-06-26T14:11:54+02:00Rosen MatevMigrate to pytestSeems like a good idea to use something more modern. Most tests run just fine with pytest and adapting them looks straightforward.Seems like a good idea to use something more modern. Most tests run just fine with pytest and adapting them looks straightforward.https://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/14Move gitlab-mr slot logic from LHCbNighltyConf to here2019-10-24T12:43:58+02:00Rosen MatevMove gitlab-mr slot logic from LHCbNighltyConf to hereWhat happens in the main job (see [here](https://gitlab.cern.ch/lhcb-core/LbNightlyTools/blob/master/python/LbNightlyTools/Scripts/EnabledSlots.py)) is more appropriately placed in this repo. Then LHCbNightlyTools won't need to have anyt...What happens in the main job (see [here](https://gitlab.cern.ch/lhcb-core/LbNightlyTools/blob/master/python/LbNightlyTools/Scripts/EnabledSlots.py)) is more appropriately placed in this repo. Then LHCbNightlyTools won't need to have anything related to the gitlab-mr slots.Rosen MatevRosen Matevhttps://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/13Follow-up from "Add options to resolve the MRs aliases in the main job"2019-10-15T20:07:28+02:00Marco Clemencicmarco.clemencic@cern.chFollow-up from "Add options to resolve the MRs aliases in the main job"The following discussion from !236 should be addressed:
- [x] @clemenci started a [discussion](https://gitlab.cern.ch/lhcb-core/LbNightlyTools/merge_requests/236#note_2910164):
> This unconditional override of 'commit' breaks lhcb...The following discussion from !236 should be addressed:
- [x] @clemenci started a [discussion](https://gitlab.cern.ch/lhcb-core/LbNightlyTools/merge_requests/236#note_2910164):
> This unconditional override of 'commit' breaks lhcb-core/LHCbNightlyConf!1852.0.2Marco Clemencicmarco.clemencic@cern.chMarco Clemencicmarco.clemencic@cern.chhttps://gitlab.cern.ch/lhcb-core/LbNightlyTools/-/issues/11fix misleading debug message2019-10-28T16:07:20+01:00Marco Clemencicmarco.clemencic@cern.chfix misleading debug messagesee https://gitlab.cern.ch/lhcb-core/LbNightlyTools/commit/d6ab130350c82105f59328d7f16aca1740a666b8#note_2904305see https://gitlab.cern.ch/lhcb-core/LbNightlyTools/commit/d6ab130350c82105f59328d7f16aca1740a666b8#note_29043052.1.1Marco Clemencicmarco.clemencic@cern.chMarco Clemencicmarco.clemencic@cern.ch