WIP Fix stripping21-patches

Major overhaul of the repository to make it consistent with the procedure for Run 2 campaigns.

added High priority Stripping21rXp2 labels

changed milestone to %Re-stripping of Run 1 & Run 2 LHCb data

assigned to @cvazquez

mentioned in issue #35 (closed)

added 1 commit

• f58d50e2 - Update TestMyWGfromSelections_Run1.py

Compare with previous version

• [2019-02-14 00:15] Validation started with lhcb-stripping21-patches#634
• [2019-02-15 00:07] Validation started with lhcb-stripping21-patches#635

@rvazquez I guess you've seen that this breaks one of the nightly tests: https://lhcb-nightlies.cern.ch/logs/build/nightly/lhcb-stripping21-patches/634/x86_64-slc6-gcc48-opt/Stripping/

yeah, in fact I have located the problem and the fix is easy, but didn't have time yet to take action (I was investigating the timeout issue).

added 2 commits

• 04e5fe3c - Fixed indentation
• b593a423 - Merge branch 'fix_stripping21-patches' of...

Compare with previous version

Fixed, let's check tomorrow.

@cvazquez @erodrigu following on from discussions in https://its.cern.ch/jira/browse/LBOPG-103 and https://its.cern.ch/jira/browse/LHCBGAUSS-1458, I don't think it's correct to include this MR on this branch - and in fact you should revert also fbfb7609, d0a639c4 and eef8ea4b.

The preparation for S21rXp2 should not be on stripping21-patches but on the branch used/derived for the S21rXp1 incremental stripping

Hi, I will look into all these S21 matters tomorrow, to confirm I understand all issues and agree with the exact stack we want ...

mentioned in merge request !917 (closed)

mentioned in merge request !868 (closed)

This MR is wrong. stripping2-patches should only be used for "technical" maintenance of the original S21r0 and S21r1 stripping releases. For the incremental restripping, a new branch will have to be created starting from the version used for S21r0p1 and S21r1p1

Making this WIP for the moment, bu probably should be closed.

marked as a Work In Progress

mentioned in merge request !980 (merged)

mentioned in merge request Phys!497 (closed)

@cattanem I'm off today but I will have a look into this tomorrow (and all the related changes that need to be reverted/moved to another branch).

I'm working my way through the release of the stack on stripping21-patches and will try and tag Stripping too, if I have the rights for it. Otherwise I will ping Nicole ...

why do you need to tag the Stripping (I'm still halfway checking my inbox so I might missed a mail somewhere)?

I'm finalising the stack Marco prepared, see https://its.cern.ch/jira/browse/LHCBGAUSS-1458?focusedCommentId=2479466&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-2479466.

I'm done with Phys and Analysis so now would go ahead, merge !980 (merged), and tag Stripping and then finalise with DaVinci … I will proceed in the next few minutes …

I can tag the Stripping.

OK, cool! I tagged Phys v19r2p4 and Analysis v13r1p4. Let me know when you are done. Thanks!

https://gitlab.cern.ch/lhcb/Stripping/tags/v9r2p4 done

Thanks. And I've finalised the stack and triggered a release build. Will keep you posted ...

@cattanem, there is an issue when I run lb-sdb-import DaVinci v36r1p6:

2019-02-18 17:52:23 - WARNING: (root) Find project LHCB v38r2p3
2019-02-18 17:52:23 - WARNING: (ariadne.py2neo.rest) Request failed (BadStatusLine), retrying
2019-02-18 17:52:23 - WARNING: (root) No Project/Version found for LHCB v38r2p3
2019-02-18 17:52:24 - WARNING: (root) LHCB/v38r2p3 - Using import URI: gitlab-cern:lhcb/LHCb#v38r2p3
2019-02-18 17:52:24 - WARNING: (root) Project LHCb is in Gitlab URI:gitlab-cern:lhcb/LHCb#v38r2p3
2019-02-18 17:52:24 - WARNING: (root) Looking for CMakeLists.txt
2019-02-18 17:52:24 - WARNING: (root) Could not find CMakeLists at: https://gitlab.cern.ch/lhcb/LHCb/raw/v38r2p3/CMakeLi                                                                     sts.txt
2019-02-18 17:52:24 - WARNING: (root) Looking for projectConfig.json
2019-02-18 17:52:24 - WARNING: (root) Could not find projectConfig.json at: https://gitlab.cern.ch/lhcb/LHCb/raw/v38r2p3                                                                     /dist-tools/projectConfig.json
2019-02-18 17:52:24 - WARNING: (root) Looking for legacy CMT project.cmt
2019-02-18 17:52:24 - WARNING: (root) Could not find Project.cmt at: https://gitlab.cern.ch/lhcb/LHCb/raw/v38r2p3/cmt/pr                                                                     oject.cmt
2019-02-18 17:52:24 - ERROR: (root) Could not find project dependency metadata
Traceback (most recent call last):
  File "/cvmfs/lhcb.cern.ch/lib/lhcb/LBSCRIPTS/LBSCRIPTS_v9r2p6/LbRelease/python/LbRelease/SoftConfDB/LbSdbImport.py", l                                                                     ine 81, in <module>
    sys.exit(s.run())
  File "/cvmfs/lhcb.cern.ch/lib/lhcb/LBSCRIPTS/LBSCRIPTS_v9r2p6/InstallArea/python/LbUtils/Script.py", line 107, in run
    rc = self.main()
  File "/cvmfs/lhcb.cern.ch/lib/lhcb/LBSCRIPTS/LBSCRIPTS_v9r2p6/LbRelease/python/LbRelease/SoftConfDB/LbSdbImport.py", l                                                                     ine 69, in main
    sourceuri=opts.sourceuri)
  File "/cvmfs/lhcb.cern.ch/lib/lhcb/LBSCRIPTS/LBSCRIPTS_v9r2p6/InstallArea/python/LbRelease/SoftConfDB/AppImporter.py",                                                                      line 335, in processProjectVersion
    return self.gitlabProcessProjectVersion(p, v, alturi=sourceuri, saveURIinPV=forcedSourceURI)
  File "/cvmfs/lhcb.cern.ch/lib/lhcb/LBSCRIPTS/LBSCRIPTS_v9r2p6/InstallArea/python/LbRelease/SoftConfDB/AppImporter.py",                                                                      line 298, in gitlabProcessProjectVersion
    node_child = self.processProjectVersion(dp, dv, alreadyDone, recreate)
  File "/cvmfs/lhcb.cern.ch/lib/lhcb/LBSCRIPTS/LBSCRIPTS_v9r2p6/InstallArea/python/LbRelease/SoftConfDB/AppImporter.py",                                                                      line 335, in processProjectVersion
    return self.gitlabProcessProjectVersion(p, v, alturi=sourceuri, saveURIinPV=forcedSourceURI)
  File "/cvmfs/lhcb.cern.ch/lib/lhcb/LBSCRIPTS/LBSCRIPTS_v9r2p6/InstallArea/python/LbRelease/SoftConfDB/AppImporter.py",                                                                      line 298, in gitlabProcessProjectVersion
    node_child = self.processProjectVersion(dp, dv, alreadyDone, recreate)
  File "/cvmfs/lhcb.cern.ch/lib/lhcb/LBSCRIPTS/LBSCRIPTS_v9r2p6/InstallArea/python/LbRelease/SoftConfDB/AppImporter.py",                                                                      line 335, in processProjectVersion
    return self.gitlabProcessProjectVersion(p, v, alturi=sourceuri, saveURIinPV=forcedSourceURI)
  File "/cvmfs/lhcb.cern.ch/lib/lhcb/LBSCRIPTS/LBSCRIPTS_v9r2p6/InstallArea/python/LbRelease/SoftConfDB/AppImporter.py",                                                                      line 298, in gitlabProcessProjectVersion
    node_child = self.processProjectVersion(dp, dv, alreadyDone, recreate)
  File "/cvmfs/lhcb.cern.ch/lib/lhcb/LBSCRIPTS/LBSCRIPTS_v9r2p6/InstallArea/python/LbRelease/SoftConfDB/AppImporter.py",                                                                      line 335, in processProjectVersion
    return self.gitlabProcessProjectVersion(p, v, alturi=sourceuri, saveURIinPV=forcedSourceURI)
  File "/cvmfs/lhcb.cern.ch/lib/lhcb/LBSCRIPTS/LBSCRIPTS_v9r2p6/InstallArea/python/LbRelease/SoftConfDB/AppImporter.py",                                                                      line 298, in gitlabProcessProjectVersion
    node_child = self.processProjectVersion(dp, dv, alreadyDone, recreate)
  File "/cvmfs/lhcb.cern.ch/lib/lhcb/LBSCRIPTS/LBSCRIPTS_v9r2p6/InstallArea/python/LbRelease/SoftConfDB/AppImporter.py",                                                                      line 335, in processProjectVersion
    return self.gitlabProcessProjectVersion(p, v, alturi=sourceuri, saveURIinPV=forcedSourceURI)
  File "/cvmfs/lhcb.cern.ch/lib/lhcb/LBSCRIPTS/LBSCRIPTS_v9r2p6/InstallArea/python/LbRelease/SoftConfDB/AppImporter.py",                                                                      line 298, in gitlabProcessProjectVersion
    node_child = self.processProjectVersion(dp, dv, alreadyDone, recreate)
  File "/cvmfs/lhcb.cern.ch/lib/lhcb/LBSCRIPTS/LBSCRIPTS_v9r2p6/InstallArea/python/LbRelease/SoftConfDB/AppImporter.py",                                                                      line 335, in processProjectVersion
    return self.gitlabProcessProjectVersion(p, v, alturi=sourceuri, saveURIinPV=forcedSourceURI)
  File "/cvmfs/lhcb.cern.ch/lib/lhcb/LBSCRIPTS/LBSCRIPTS_v9r2p6/InstallArea/python/LbRelease/SoftConfDB/AppImporter.py",                                                                      line 254, in gitlabProcessProjectVersion
    deps =  gp.getDependencies()
  File "/cvmfs/lhcb.cern.ch/lib/lhcb/LBSCRIPTS/LBSCRIPTS_v9r2p6/InstallArea/python/LbRelease/SoftConfDB/AppImporter.py",                                                                      line 195, in getDependencies
    raise Exception("Could not find project metadata")
Exception: Could not find project metadata

Seems not all (CMT) files got updated for the release ...

v38r2p3 has been tagged as v32r2p3: https://gitlab.cern.ch/lhcb/LHCb/tags/v32r2p3

Also, this file hasn't been updated: https://gitlab.cern.ch/lhcb/LHCb/blob/v32r2p3/LHCbSys/CMakeLists.txt

Merde! I have retagged with the correct tag.... I don't think the LHCbSys file needs to be updated, we were doing it for some time and then stopped.

I've just retriggered a release build. Thanks for the speedy fix. Bon app.

The build is done, see https://lhcb-nightlies.cern.ch/release/lhcb-release/build/3208/. But for one of the platforms I see a message "dependencies not fulfilled." Seems odd as I see nothing fishy on the prompt with lb-sdb-query d DaVinci v36r1p6. Any idea? Else I will still request the deployment tomorrow, pointing out this fact, if nothing else strange comes in … I will keep checking ...

The dependencies not fulfilled is, I think, a red herring, probably some missing LCG package that you don’t need (otherwise build or tests would have failed). I had something similar for the LCG_95 stack

The deployment is requested - https://its.cern.ch/jira/browse/LHCBDEP-4062.

@cattanem which would be the name of the branch used for the Run 1 re-stripping? I would say run1-patches? I'm about to create one from the Stripping tag for S21rXp1.

I thought of that as well but we should check what is used for Reco14-patches, we risk further confusion if it’s a different set of releases. Let me think about it

OK, I'll wait for you to let me know an appropriate branch name. There is also another thing I thought about, slc6 support will end by 2020, so this might be a problem if we want to re-strip in the next years. Wouldn't we need a LCG version with centos7 support as well? What do you think?

I have checked, Reco14 was on a much older stack, so there is no risk of confusion there. What bothers me a bit with calling the new branches run1-patches is that, by analogy with run2-patches, we might eventually forget that it's a branch specifically for S21 incremental restrippings and start putting there e.g. reconstruction fixes that might not be S21 compatible. Wouldn't it be better to rename the existing stripping21-patches (e.g. to stripping21-firstpass-patches) and reuse the stripping21-patches name for the incremental restrippings? If you agree, I can do the git work to set it all up.

Concerning the slc6 support, that's not a problem, because we run our productions in virtual machines. For Moore steps we still use slc5, no problem.

I agree that your proposal seems best in the longer run. We just need to make sure that the documentation in the stripping pages gets updated. Maybe also add a comment in https://twiki.cern.ch/twiki/bin/view/Main/ProcessingPasses?

For the record, S20r0p1 and S20r1p1 used DaVinci v39r1p1 which has the following dependencies:

lb-sdb-query listDependencies DaVinci v39r1p1
STRIPPING	v10r1p1
PHYS	v21r1
REC	v19r0
LBCOM	v18r0
LHCB	v40r0
GAUDI	v27r0
LCG	83
LCGCMT	83
ANALYSIS	v15r1

So the new stripping21-patches branches should be made starting from those versions.

I propose the following recipe:

In Gitlab -> Settings -> Repository -> Protected branches, change the setting of *-patches to allow you to push

On local machine:

git clone https://:@gitlab.cern.ch:8443/lhcb/LHCb.git
cd LHCb
# local rename
git branch -m stripping21-patches stripping21-firstpass-patches 
git checkout stripping21-firstpass-patches
git push origin stripping21-firstpass-patches
# go to Gitlab and check that the new branch is there and looking correct (same history as old branch)
# delete old branch on remote
git push origin -d stripping21-patches
# and create it again starting from new branch point
git checkout v40r0
git checkout -b stripping21-patches
git push origin stripping21-patches

Go back to gitlab, check everything OK and protect again *-patches branches.

@clemenci @rmatev does this look reasonable / correct?

It looks correct, but I would take a couple of shortcuts:

git clone --bare https://:@gitlab.cern.ch:8443/lhcb/LHCb.git
cd LHCb.git
# create new remote branch matching current stripping21-patches
git push origin stripping21-patches:stripping21-firstpass-patches
# replace remote stripping21-patches
git push -f origin v40r0:stripping21-patches

or you could do everything from the web interface:

• create branch stripping21-firstpass-patches from stripping21-patches
• unprotect
• remove stripping21-patches
• protect
• create branch stripping21-patches from v40r0

I confirm that @clemenci 's web interface recipe works for me. I've done LHCb, will do Lbcom and Rec, and leave it to @erodrigu for the rest of the stack.

I'll also set up a new nightly slot to maintain the stripping21-firstpass-patches stack.

@cvazquez when you do this operation for Stripping, the merge requests currently open against stripping21-patches branch will be automatically closed, they will have to be reopened and rebased. I suggest you inform the stripping community that they have to refresh their git projects (git fetch) and then adapt their incremental S21 line branches accordingly

Nightly slot created.

lhcb-stripping21-firstpass-patches will pick up automatically the stripping21-firstpass-patches branches when they are created (currently up to Rec) and should become the same as this screenshot when complete

lhcb-stripping21-patches slot continues to pick up strippin21-patches branches, so will be hybrid (and broken) until all branches are migrated to DaVinci v39r1p1 stack (currently only done as far as Rec)

@erodrigu @cvazquez please report here when you migrate the remaining projects

Hi @cattanem, all, I did the same for Phys. All fine there. I will try and do the rest asap, though, as you know now, I'm leaving tomorrow for a 4-day long weekend …

I changed nothing to the nightlies set-up as I could not find the file you edited in https://gitlab.cern.ch/lhcb-core/LHCbNightlyConf/tree/master. Seems I'm looking at the wrong place.

@erodrigu indeed you saw nothing because I had pushed a branch with the changes but forgot to make a merge request...

Perhaps you could add me as maintainer of Phys, Analysis and DaVinci? That way there is someone who can do these sort of maintenance operations when you are away?

I've added you as maintainer for the 3 projects, @cattanem. I won't be able to do anything else until this evening so let me know if you do the swap of branches for Analysis and DaVinci as otherwise I will do it asap.

@erodrigu done for Analysis and DaVinci (note that branch DaVinci/DV4Strip21 no longer exists, the new ones are called DaVinci/stripping21-firstpass-patches and DaVinci/stripping21-patches

@cvazquez could you please do the last remaining rename/creation, for Stripping? (Stripping/stripping21-patches to be renamed Stripping/stripping21-firstpass-patches, Stripping/stripping21-patches to be created from Stripping/v10r1p1) - if you prefer, I can do it, if you make me maintainer of Stripping Gitlab project.

@cattanem Sorry, I've been sick these days and yesterday I was completely out. I prefer to do it myself (just to keep track of the changes), will do so this afternoon.