wrong test status in CouchDB

lhcb-master-mr/2587 shows 3 failures for LHCb/x86_64_v2-centos7-gcc10-dbg (as in CouchDB) but that does not match the Test.xml (also attached as Test.xml.zip), which has no failures

changed the description

This kind of inconsistency usually happens when the test job is restarted (for whichever reason).

The entries in CouchDB are updated, but the results on EOS are not (issue with the way we invoke eos cp). LBCORE-1706 is meant to address the issue at the root.

Weirdly enough, though, looking at the logs I do not see trace of the inconsistency, so I'll try to see if there's an obvious bug in the code that extracts the summary for couchdb.

mentioned in merge request !364 (merged)

I think I found a possible explanation.

It may happen that two Jenkins jobs are running for the same test (e.g. connection lost on the first one and a retry is automatically started). If we assume that the first instance has some failures and the second not, this is what may happen:

• the first create the entry for the test results in couchdb
• the second overwrite the entry with its details (start time, build_url), it's normal as we have to support the automatic retries
• the second job updates the entry with its results (a dictionary {'PASS': [...]}, note no FAIL)
• the second job uploads the results
• the first job updates the entry with its results (a dictionary {'PASS': [...], 'FAIL', [...]})
• the first job tries to upload the results without success (because of eos cp without overwrite)

In principle the update should be done only by the job the owns the entry. See !364 (merged)

The fix I prepared does not cover the case we the summary is inconsistent with the full report, but I cannot easily fix it in the old system.

mentioned in commit 2626c92c

closed with merge request !364 (merged)