fix(workflow): UploadMC TypeError exception with DateTime objects
I randomly found an issue in the UploadMC
module while reading the logs of the PushJobAgent
:
File "/opt/dirac/versions/v11.0.46-1723623328/Linux-x86_64/lib/python3.11/site-packages/LHCbDIRAC/Workflow/Modules/UploadMC.py", line 176, in execute
output.write(str(json.dumps(jsonData)))
^^^^^^^^^^^^^^^^^^^^
File "/opt/dirac/versions/v11.0.46-1723623328/Linux-x86_64/lib/python3.11/json/__init__.py", line 231, in dumps
return _default_encoder.encode(obj)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/dirac/versions/v11.0.46-1723623328/Linux-x86_64/lib/python3.11/json/encoder.py", line 200, in encode
chunks = self.iterencode(o, _one_shot=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/dirac/versions/v11.0.46-1723623328/Linux-x86_64/lib/python3.11/json/encoder.py", line 258, in iterencode
return _iterencode(o, 0)
^^^^^^^^^^^^^^^^^
File "/opt/dirac/versions/v11.0.46-1723623328/Linux-x86_64/lib/python3.11/json/encoder.py", line 180, in default
raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type datetime is not JSON serializable
The UploadMC
module tries to serialize a datetime
object in a JSON file.
- Note1: from what I can see, this object was introduced a few months ago in 3740f91e, and current tests did not spot the issue. The workflow tests I introduced in
devel
would have spotted it from what I can see. Should we backport the tests tomaster
? - Note2: I tried to spot similar issues in other MC simulation jobs that were running on "classic" sites (not managed by the
PushJobAgent
), and I noticed that there was no log related to theUploadMC
step (std.out
generally stops atUploadOutputData
). Is this expected?
BEGINRELEASENOTES
*Workflow FIX: UploadMC TypeError exception with datetime objects serialized
ENDRELEASENOTES
Merge request reports
Activity
added alsoTargeting:devel label
Thank you for the fix. On
Note2: I tried to spot similar issues in other MC simulation jobs that were running on "classic" sites (not managed by the
PushJobAgent
), and I noticed that there was no log related to theUploadMC
step (std.out
generally stops atUploadOutputData
). Is this expected?The
UploadMC
module is normally not run:GaudiStep_Modules = GaudiApplication GaudiStep_Modules += AnalyseXMLSummary GaudiStep_Modules += ErrorLogging GaudiStep_Modules += BookkeepingReport GaudiStep_Modules += StepAccounting
mentioned in commit 1424db01
added sweep:done label
added sweep:failed label
Sweep summary
Sweep ran in https://gitlab.cern.ch/lhcb-dirac/LHCbDIRAC/-/jobs/43336894
Failed:- devel
The
UploadMC
module is normally not runThe
UploadMC
is found in theJobFinalization
step, not theGaudi
step.It seems to be part of the default values of the step: https://gitlab.cern.ch/lhcb-dirac/LHCbDIRAC/-/blob/master/src/LHCbDIRAC/ProductionManagementSystem/Client/Production.py#L393-409