Failure when running CRAB jobs at CERN

When I tried to send crab jobs with mkNtuples at CERN, I got the below error message:

$ mkNtuples -c FastSim.json 
Opening FastSim.json
latest commit: Merge branch 'bugfix/HTC_CERN' into 'master'

Bugfix/htc cern

See merge request Proto/Darwin!69               by Louis Moureaux (a30e901bffe1fdf)
+ das-cmssw el8 mkNtuples -c FastSim_a30e901bffe1fdf.json
Opening FastSim_a30e901bffe1fdf.json
storage site: T2_CH_CERN
/QCD_Pt-15to7000_TuneCP5_Flat2018_13TeV_pythia8/RunIISummer20UL17MiniAODv2-FlatPU0to70_FSUL17_FastSimulation_106X_mc2017_realistic_v9-v2/MINIAODSIM
local output: QCD_Pt-15to7000_TuneCP5_Flat2018_13TeV_pythia8_a30e901bffe1fdf/crab_FastSim_RunIISummer20UL17MiniAODv2-FlatPU0to70_FSUL17_FastSimulation_106X_mc2017_realistic_v9-v2
input: /QCD_Pt-15to7000_TuneCP5_Flat2018_13TeV_pythia8/RunIISummer20UL17MiniAODv2-FlatPU0to70_FSUL17_FastSimulation_106X_mc2017_realistic_v9-v2/MINIAODSIM
storage: QCD_Pt-15to7000_TuneCP5_Flat2018_13TeV_pythia8/a30e901bffe1fdf_FastSim_RunIISummer20UL17MiniAODv2-FlatPU0to70_FSUL17_FastSimulation_106X_mc2017_realistic_v9-v2
Found non option element 192:00 in command line.
Process Process-1:
Traceback (most recent call last):
  File "/cvmfs/cms.cern.ch/el8_amd64_gcc10/external/python3/3.9.6-67e5cf5b4952101922f1d4c8474baa39/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/cvmfs/cms.cern.ch/el8_amd64_gcc10/external/python3/3.9.6-67e5cf5b4952101922f1d4c8474baa39/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/eos/home-e/euslan/EL9/DasAnalysisSystem/CMSSW_12_4_0/bin/el8_amd64_gcc10/mkNtuples", line 199, in submit
    return crabCommand('submit', config = config)
  File "/cvmfs/cms.cern.ch/share/cms/crab-prod/v3.240621.01/lib/CRABAPI/RawCommand.py", line 28, in crabCommand
    return execRaw(command, arguments)
  File "/cvmfs/cms.cern.ch/share/cms/crab-prod/v3.240621.01/lib/CRABAPI/RawCommand.py", line 51, in execRaw
    cmdobj = getattr(mod, command)(logger, args)
  File "/cvmfs/cms.cern.ch/share/cms/crab-prod/v3.240621.01/lib/CRABClient/Commands/submit.py", line 36, in __init__
    SubCommand.__init__(self, logger, cmdargs, disable_interspersed_args=True)
  File "/cvmfs/cms.cern.ch/share/cms/crab-prod/v3.240621.01/lib/CRABClient/Commands/SubCommand.py", line 364, in __init__
    self.handleVomsProxy(proxyOptsSetPlace)
  File "/cvmfs/cms.cern.ch/share/cms/crab-prod/v3.240621.01/lib/CRABClient/Commands/SubCommand.py", line 482, in handleVomsProxy
    proxyInfo = self.credentialHandler.createNewVomsProxy(timeLeftThreshold=720, \
  File "/cvmfs/cms.cern.ch/share/cms/crab-prod/v3.240621.01/lib/CRABClient/CredentialInteractions.py", line 103, in createNewVomsProxy
    proxy.create()
  File "/cvmfs/cms.cern.ch/share/cms/crab-prod/v3.240621.01/lib/CRABClient/ProxyInteractions.py", line 52, in create
    raise ProxyCreationException(msg)
CRABClient.ClientExceptions.ProxyCreationException: Error executing voms-proxy-init --rfc --cert ~/.globus/usercert.pem --key ~/.globus/userkey.pem --out  --valid 192:00 --voms cms:/cms:

I could reproduce the error with DoubleMuon data sets, running at CERN, whereas it works fine when running at DESY.

Seem related to the last line:

CRABClient.ClientExceptions.ProxyCreationException: Error executing voms-proxy-init --rfc --cert ~/.globus/usercert.pem --key ~/.globus/userkey.pem --out --valid 192:00 --voms cms:/cms:

and to some intermediate line, just before the error Stack from Python:

Found non option element 192:00 in command line.

However this command is incomplete :/ Need some debugging. It might just be some missing paths in the container.

added to epic &3

added Urgent changepatch statusto do typesoftware labels

Testing step by step:

% el8
Singularity> cd $DAS_BASE
Singularity> source CMS/setup
rucio 1.31.7
Singularity> cd CMSSW_12_4_0/
Singularity> cmsenv
Singularity> voms-proxy-init -voms cms
Enter GRID pass phrase:
Your identity: /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=paconnor/CN=744980/CN=Patrick Connor
Creating temporary proxy ...................................................................................................................
............. Done                                                    
Contacting  voms-cms-auth.app.cern.ch:443 [/DC=ch/DC=cern/OU=computers/CN=cms-auth.web.cern.ch] "cms" Done
Creating proxy ...........................................................................................
ERROR: Cannot write proxy to:      

$\to$

this error does not happen at DESY, where a proxy is created in my home directory

It seems that CRAB tries to run voms-proxy-init (in the apptainer) instead of picking the certificate that was activated in the shell (in EL9).

Seems that with the cmssw-el* commands, CRAB was found to work. Should take a look tomorrow...

@euslan Here is a temporary solution (I plan to automate this fix in the future).

Once per session, you need:

voms-proxy-init -voms cms -rfc -valid 192:00
export X509_USER_PROXY=`voms-proxy-info --path`

(You were probably already running the first line in the past, but probably not the second line.) Then mkNtuple -c config.json should work. Please confirm if this works for you.

assigned to @paconnor

removed Urgent label

Ok, so the issue it clarified, then I removed the urgent label and I will "resolve" previous threads.

However, I leave it open until we have automated this fix in the software.

mentioned in commit gitlab-profile@27dad26f

mentioned in merge request gitlab-profile!45 (merged)

mentioned in commit gitlab-profile@bb941c7e

mentioned in commit gitlab-profile@894f0d16

closed with merge request gitlab-profile!45 (merged)