Skip to content

Draft: Update for snakemake8

Matthew David Monk requested to merge mmonk_snakemake8 into master

This introduces (breaking) changes so that apd can be used with Snakemake>=8 which changed how user interact with remote storage (XRootD in this case).

The changes are not backwards compatible with Snakemake versions below 8 as a result of Snakemake itself not being backwards compatibible in between those versions due to the move to using "storage plugins".

Note, if you try this out yourself you may see something like this in the snakemake output:

Building DAG of jobs...
Retrieving .snakemake/storage/apd/os/lhcb/grid/prod/lhcb/anaprod/lhcb/MC/Dev/NTUPLE.ROOT/00236385/0000/00236385_00000002_1.ntuple.root from storage.
Retrieving .snakemake/storage/apd/os/lhcb/grid/prod/lhcb/anaprod/lhcb/MC/Dev/NTUPLE.ROOT/00236385/0000/00236385_00000001_1.ntuple.root from storage.

This does NOT necessarily mean that it is actually downloading the remote file, indeed in my case if I then printed the input files in the python script I passed the files to it correctly has the PFN. I think this is probably a bug in Snakemake (by default Snakemake assumes you want to download remote files which we usually do not want).

TODO:

  • Check it works with a "usual" setup (where a token is not needed)
  • Add a test workflow
  • Check the test workflow works locally
  • Add a CI job for the test workflow
  • Check it works with using a token in the URL for the authentication
  • Add a check for the snakemake version so if it is <8 it uses the old code otherwise this new stuff
Edited by Matthew David Monk

Merge request reports