New script to automatically update CITest references

I mentioned this in the SW weekly yesterday, but this is a first draft of an automated script to update references (I think it belongs in PROCTools but of course I'm happy to move it elsewhere).

So, let me explain what the current workflow is:

• in case we have a reference changing MR (e.g. !63410 (merged), which is what I've been testing on), we need to follow the instructions here, but in summary:
  • Look at all failing tests, and work out whether digest and/or pool files need updating
    • If digest, either run locally and copy new digest to Tools/PROCTools/data, or try to generate patch from logfile (risky)
    • if pool, look in logfile to see where the new file was copied, create new directories on EOS and copy them there. Update References.py
  • In both cases, you now need to push changes to the MR.

So there are a lot of steps, and a lot of chances for mistakes ... and it also takes hours to do.

This script attempts to simplify this by:

• scraping the bigpanda CI Build summary page, e.g https://bigpanda.cern.ch/ciview/?rel=MR-63410-2023-10-09-12-27 for the MR above.
• it then loops through the projects to get the URLs of the project test results, the scrapes the URLs of all failing tests and their logfiles.
• from these logfiles it extracts all of the necessary information and performs all of the steps above, up to any git operations (this can perhaps be added in a second pass).
• in the event that there are two tests modifying the same file, it will abort (since we have not agreed how to handle this).

And the script also adds a text file to each version directory, explaining the provenance of the change e.g.

-bash-4.2$ cat /eos/atlas/atlascerngroupdisk/data-art/grid-input/WorkflowReferences/main/s4006/v4/info.txt 
Merge URL: https://gitlab.cern.ch/atlas/athena/-/merge_requests/63410
Date: 2023:10:09 at 12:27
AMI: s4006
Test name: CITest_SimulationRun3FullSimLegacy-test
Files copied from: /eos/atlas/atlascerngroupdisk/proj-sit/gitlabci/MR63410_a84345c776e93f0d7f25d00c9e91e35bcb965d09/SimulationRun3FullSimLegacy/

Below I copy the text output from the script for the a test run on !66829 (merged):

Run command:

-bash-4.2$lsetup gitlab
-bash-4.2$ cp Tools/PROCTools/scripts/update_ci_reference_files.py .. # This is only necessary before the MR is accepted, and allows us to switch branch
-bash-4.2$ ../update_ci_reference_files.py --test-run url="https://bigpanda.cern.ch/ciview/?rel=MR-66829-2023-10-31-10-01"

Output:

Update reference files for URL: https://bigpanda.cern.ch/ciview/?rel=MR-66829-2023-10-31-10-01
 -> Running in test mode so will not touch EOS, but will only modify files locally (these changes can easily be reverted).
========================
Summary of tests which need work:
 * Branch: Athena
   -  CITest_RecoRun3Data_Checks-test:q449 MR: 66829 Data file change :  v38 -> v39
   -  CITest_RecoRun3MC-test:q445 MR: 66829 Data file change :  v34 -> v35
========================
The next step is to update the MR with the new content i.e. the References.py file and the digest files.
 IMPORTANT: before you do this, you must first make sure that the local repository is on same branch as the MR!
i.e. you would go to the MR: https://gitlab.cern.ch/atlas/athena/-/merge_requests/66829
 and then copy the branch name and do:
 $ git remote add <MR_AUTHOR> <URL_TO_FORK>
 $ git fetch <MR_AUTHOR>
 $ git switch -c <MR_BRANCH> yoyamagu/TightWP
 $ git rebase upstream/main

Would you like to (locally) update digest ref files and/or versions in References.py? (y/N) y
true
Updating reference files
========================
Processing test: CITest_RecoRun3Data_Checks-test on branch Athena
 * This is a DiffPool test, and currently has version v38 of q449. Will update References.py with new version.
Processing test: CITest_RecoRun3MC-test on branch Athena
 * This is a DiffPool test, and currently has version v34 of q445. Will update References.py with new version.
 -> In test-run mode, so only local files have been changed. In normal mode we would also have executed:
     cp /eos/atlas/atlascerngroupdisk/proj-sit/gitlabci/MR66829_b9d0db9c02d5774408188c6bf15896ab08b9b1b9/RecoRun3Data_Checks/* /eos/atlas/atlascerngroupdisk/data-art/grid-input/WorkflowReferences/main/q449/v39
     cp -n /eos/atlas/atlascerngroupdisk/data-art/grid-input/WorkflowReferences/main/q449/v38/* /eos/atlas/atlascerngroupdisk/data-art/grid-input/WorkflowReferences/main/q449/v39
     cp /eos/atlas/atlascerngroupdisk/proj-sit/gitlabci/MR66829_b9d0db9c02d5774408188c6bf15896ab08b9b1b9/RecoRun3MC/* /eos/atlas/atlascerngroupdisk/data-art/grid-input/WorkflowReferences/main/q445/v35
     cp -n /eos/atlas/atlascerngroupdisk/data-art/grid-input/WorkflowReferences/main/q445/v34/* /eos/atlas/atlascerngroupdisk/data-art/grid-input/WorkflowReferences/main/q445/v35

cc @wlampl @abarton @tsulaia @fwinkl @elmsheus @ahabouel @dta @jmaurer @jcatmore @nstyles @christos @jdandoy @atlas-release-coordinators