Skip to content

Avoid a race in DIRAC which causes job creation to fail

Chris Burr requested to merge use-dirac-v7r3 into master

Testing productions can fail with this surprising stack trace:

  File "/miniconda/envs/analysis-productions/lib/python3.8/site-packages/dirac_prod/classes/input_dataset.py", line 32, in replicas
    for se, pfn in dw(Dirac().getReplicas([str(self)], active=True))[str(self)].items():
  File "/miniconda/envs/analysis-productions/lib/python3.8/site-packages/DIRAC/Interfaces/API/Dirac.py", line 858, in getReplicas
    repsResult = dm.getReplicas(lfns, active=active, preferDisk=preferDisk, diskOnly=diskOnly)
  File "/miniconda/envs/analysis-productions/lib/python3.8/site-packages/DIRAC/DataManagementSystem/Client/DataManager.py", line 1742, in getReplicas
    succPfn = seObj.getURL(se_lfn[se],
  File "/miniconda/envs/analysis-productions/lib/python3.8/site-packages/DIRAC/Resources/Storage/StorageElement.py", line 969, in getURL
    result = self.__executeMethod(lfn, protocols=protocols)
  File "/miniconda/envs/analysis-productions/lib/python3.8/site-packages/DIRAC/Resources/Storage/StorageElement.py", line 1240, in __executeMethod
    storageParameters = storage.getParameters()
  File "/miniconda/envs/analysis-productions/lib/python3.8/site-packages/DIRAC/Resources/Storage/StorageBase.py", line 119, in getParameters
    parameterDict['URLBase'] = self.getURLBase().get('Value', '')
  File "/miniconda/envs/analysis-productions/lib/python3.8/site-packages/DIRAC/Resources/Storage/GFAL2_XROOTStorage.py", line 109, in getURLBase
    return self.__addDoubleSlash(super(GFAL2_XROOTStorage, self).getURLBase(withWSUrl=withWSUrl))
TypeError: super(type, obj): obj must be an instance or subtype of type

It turns out this is due to DIRAC importing the storage class twice. To reproduce it more simply, create a file named reproducer.py containing:

class A:
    def example(self):
        pass

class B(A):
    def example(self):
        super(B, self).example()

Then run

import imp
import reproducer

obj1 = reproducer.B()
obj1.example()

reproducer = imp.load_module("reproducer", *imp.find_module("reproducer"))
obj2 = reproducer.B()

obj2.example()
obj1.example()

This results in:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-1-c14935bd165d> in <module>
      9
     10 obj2.example()
---> 11 obj1.example()

~/Development/DPA/LbAnalysisProductions/reproducer.py in example(self)
      5 class B(A):
      6     def example(self):
----> 7         super(B, self).example()

TypeError: super(type, obj): obj must be an instance or subtype of type

It turns out I already fixed this for DIRAC 7.3 as part of other refactoring.

Edited by Chris Burr

Merge request reports