Skip to content

Crab server error when submitting 2024 samples

When trying to produce n-tuples for 2024 AK8 Data (RunFtoI), crab server crashes printing something that looks like an html error (haven't pasted the entire error here i.e. dots):

mkNtuples -c -i el9 Run3_2024PromptAK8.json 
Opening Run3_2024PromptAK8.json
latest commit: Merge branch 'bugfix/hold' into 'master'

fix kerberos

See merge request Proto/Darwin!112               by Patrick Louis S Connor (a6b717c3d9d1f20)
+ das-cmssw el9 mkNtuples -c Run3_2024PromptAK8_a6b717c3d9d1f20.json
Opening Run3_2024PromptAK8_a6b717c3d9d1f20.json
storage site: T3_CH_CERN
/JetMET0/Run2024F-PromptReco-v1/MINIAOD
local output: JetMET0_a6b717c3d9d1f20/crab_Run3_2024PromptAK8_Run2024F-PromptReco-v1
input: /JetMET0/Run2024F-PromptReco-v1/MINIAOD
storage: JetMET0/a6b717c3d9d1f20_Run3_2024PromptAK8_Run2024F-PromptReco-v1
Importing CMSSW configuration /eos/home-a/aziaka/DAS_MCRun3/CMSSW_14_0_14/src/Core/Ntupliser/python/Ntupliser_cfg.py
Finished importing CMSSW configuration /eos/home-a/aziaka/DAS_MCRun3/CMSSW_14_0_14/src/Core/Ntupliser/python/Ntupliser_cfg.py
Sending the request to the server at cmsweb.cern.ch
Fatal error trying to connect to https://cmsweb.cern.ch:8443/crabserver/prod/workflow using dryrun=0&workflow=crab_Run3_2024PromptAK8_Run2024F-PromptReco-v1&saveoutput=1&savelogsflag=1&inputdata=%2FJetMET0%2FRun2024F-PromptReco-v1%2FMINIAOD&nonvaliddata=0&dbsurl=https%3A%2F%2Fcmsweb.cern.ch%2Fdbs%2Fprod%2Fglobal%2FDBSReader&usepar
..........
416%2C420%2C592&lumis=35%2C671%2C673%2C1185%2C1193%2C1825&lumis=1%2C172&lumis=33%2C846&lumis=32%2C567%2C569%2C602%2C604%2C694.
exit code from curl = 0
HTTP code/reason = 400/Bad Request .  stdout:
<!DOCTYPE html PUBLIC
"-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html>
<head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8"></meta>
    <title>400 Bad Request</title>
    <style type="text/css">
    #powered_by {
        margin-top: 20px;
        border-top: 2px solid black;
        font-style: italic;
    }

    #traceback {
        color: red;
    }
    </style>
</head>
    <body>
        <h2>400 Bad Request</h2>
        <p>Invalid input parameter</p>
        <pre id="traceback"></pre>
    <div id="powered_by">
      <span>
        Powered by <a href="http://www.cherrypy.dev">CherryPy 18.8.0</a>
      </span>
    </div>
    </body>
</html>

Process Process-1:
Traceback (most recent call last):
  File "/cvmfs/cms.cern.ch/el9_amd64_gcc12/external/python3/3.9.14-8e02587b42992e07ed46b00eca9dfc3a/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/cvmfs/cms.cern.ch/el9_amd64_gcc12/external/python3/3.9.14-8e02587b42992e07ed46b00eca9dfc3a/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/eos/home-a/aziaka/DAS_MCRun3/CMSSW_14_0_14/bin/el9_amd64_gcc12/mkNtuples", line 207, in submit
    return crabCommand('submit', config = config)
  File "/cvmfs/cms.cern.ch/share/cms/crab-prod/v3.250409.00/lib/CRABAPI/RawCommand.py", line 28, in crabCommand
    return execRaw(command, arguments)
  File "/cvmfs/cms.cern.ch/share/cms/crab-prod/v3.250409.00/lib/CRABAPI/RawCommand.py", line 52, in execRaw
    res = cmdobj()
  File "/cvmfs/cms.cern.ch/share/cms/crab-prod/v3.250409.00/lib/CRABClient/Commands/submit.py", line 138, in __call__
    dictresult, status, reason = server.put(api=self.defaultApi, data=self.configreq_encoded)
  File "/cvmfs/cms.cern.ch/share/cms/crab-prod/v3.250409.00/lib/CRABClient/RestInterfaces.py", line 287, in put
    return self.server.put(uri, data)
  File "/cvmfs/cms.cern.ch/share/cms/crab-prod/v3.250409.00/lib/CRABClient/RestInterfaces.py", line 133, in put
    return self.makeRequest(uri=uri, data=data, verb='PUT')
  File "/cvmfs/cms.cern.ch/share/cms/crab-prod/v3.250409.00/lib/CRABClient/RestInterfaces.py", line 225, in makeRequest
    raise RESTInterfaceException(stderr)
CRABClient.ClientExceptions.RESTInterfaceException:   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 188.184.75.82:8443...
* Connected to cmsweb.cern.ch (188.184.75.82) port 8443 (#0)
* ALPN, offering http/1.1
*  CAfile: /etc/pki/tls/certs/ca-bundle.crt
*  CApath: /etc/grid-security/certificates/
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.2 (IN), TLS header, Certificate Status (22):
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
............

I use the installed CMSSW version in DAS dedicated to Run3 i.e. CMSSW_14_1_0 and after setting the suggested path for JAVA (issue #41).

The same error occurs also for AK4.

Edited by Argyro Ziaka