Skip to content

Use XRootD if possible and process files in parallel

Chris Burr requested to merge cburr/LbNightlyTools:faster-lbninstall into master

Modifies lbninstall to:

  • Try to copy using xrdcp if possible, falls back to HTTP if that fails (requires credentials unfortunately)
  • Download and unpack all files in parallel. Using joblib for now, but I could use something else if preferred?

This makes installations on CVMFS of large projects a lot faster. I'll open a JIRA task shortly with more thoughts about making installations faster.

I'm leaving this as WIP for now to see if this is an avenue worth pursuing.

Edited by Chris Burr

Merge request reports