Skip to content

fix(cvmfs): fetch pilot.tar from s3 and do not extract it

Alexandre Franck Boyer requested to merge alboyer/lbtaskrun:main_FIX_pilot-tar into master

We still have an issue with the pilot files on CVMFS:

Trying file:/cvmfs/lhcb.cern.ch/lhcbdirac/pilot
2024-05-22 11:18:09 UTC ERROR    file:/cvmfs/lhcb.cern.ch/lhcbdirac/pilot unreacheable (this is normal!)
2024-05-22 11:18:09 UTC ERROR    <urlopen error [Errno 2] No such file or directory: '/cvmfs/lhcb.cern.ch/lhcbdirac/pilot/pilot.tar'>
Traceback (most recent call last):
  File "<stdin>", line 105, in <module>
  File "/usr/lib64/python2.7/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib64/python2.7/urllib2.py", line 431, in open
    response = self._open(req, data)
  File "/usr/lib64/python2.7/urllib2.py", line 449, in _open
    '_open', req)
  File "/usr/lib64/python2.7/urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "/usr/lib64/python2.7/urllib2.py", line 1353, in file_open
    return self.open_local_file(req)
  File "/usr/lib64/python2.7/urllib2.py", line 1393, in open_local_file
    raise URLError(msg)
URLError: <urlopen error [Errno 2] No such file or directory: '/cvmfs/lhcb.cern.ch/lhcbdirac/pilot/pilot.tar'>

This is because pilot.tar is required by the pilot wrapper: https://github.com/DIRACGrid/DIRAC/blob/integration/src/DIRAC/WorkloadManagementSystem/Utilities/PilotWrapper.py#L260

But in lbtaskrun we extract its content and we don't save pilot.tar by itself: https://gitlab.cern.ch/lhcb-core/tasks/lbtaskrun/-/blob/4a54723cfc13f2f4af7236cfcad4a62b6ec88be0/src/lbtaskrun/cvmfs/init.py#L630-644

The easiest solution is likely to include pilot.tar along with pilot.json and checksums.sha512 in cvmfs. Any opinion?

Merge request reports