MC Job Options issueshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues2024-02-01T14:46:57+01:00https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/225logParser failed due to missing platform info2024-02-01T14:46:57+01:00Yang LiulogParser failed due to missing platform infoHi @sargyrop , as we discussed in [Fixing automatic determination of release for CI runs (!2861) · Merge requests · atlas-physics / pmg / MC Job Options · GitLab (cern.ch)](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_re...Hi @sargyrop , as we discussed in [Fixing automatic determination of release for CI runs (!2861) · Merge requests · atlas-physics / pmg / MC Job Options · GitLab (cern.ch)](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/2861). It seems the added line to extract the platform info will cause problem for some of the jobs.
[Here](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/pipelines/6821117) is one example.
Many thanks for your time to help.
Cheers
Yanghttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/224Enforce saveProcDir=False in MG joboptions2024-02-21T14:55:47+01:00Spyros ArgyropoulosEnforce saveProcDir=False in MG joboptionsin the Madgraph JOs there is the possibility of saving the process directory or delete it. While keeping it is very useful for debugging, we try to enforce to have it to False for official production as it might give issues for some proc...in the Madgraph JOs there is the possibility of saving the process directory or delete it. While keeping it is very useful for debugging, we try to enforce to have it to False for official production as it might give issues for some processes in grid jobs. Jan asked that if we are trying to enforce it systematically, it should be part of the logParser. We have discussed it a bit and were thinking whether it can be added a check in the logParser to make sure that it is set to saveProcDir=False and let the CI crash in case it is changed in any bug fix push to gitlab.Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/223Block the usage of xAOD filters in HepMC3 releases2024-01-26T23:10:32+01:00Spyros ArgyropoulosBlock the usage of xAOD filters in HepMC3 releasesSee [this talk](https://indico.cern.ch/event/1372411/contributions/5770199/attachments/2788275/4861850/rel23_Jan2_24.pdf) for a description of the problem.
We want to block all jO that use HepMC3 (all r23.6 for the moment - to be fine t...See [this talk](https://indico.cern.ch/event/1372411/contributions/5770199/attachments/2788275/4861850/rel23_Jan2_24.pdf) for a description of the problem.
We want to block all jO that use HepMC3 (all r23.6 for the moment - to be fine tuned later) that use xAOD filters.Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/222Can I pass LHE container to inputGeneratorFile?2023-12-20T20:13:01+01:00Yiming AbulaitiCan I pass LHE container to inputGeneratorFile?Hi,
In log.generate.short, Can I pass a container to inputGeneratorFile?
for example:
nEventsPerJob = 5000
- Requested output events = 10000
- transform = Gen_tf
- inputFilesPerJob = 99
- inputGeneratorFile = </afs/cern.ch/work/som...Hi,
In log.generate.short, Can I pass a container to inputGeneratorFile?
for example:
nEventsPerJob = 5000
- Requested output events = 10000
- transform = Gen_tf
- inputFilesPerJob = 99
- inputGeneratorFile = </afs/cern.ch/work/somepath.../mc15_13TeV.345054.PowhegPythia8EvtGen_NNPDF3_AZNLO_WpH125J_MINLO_lvbb_VpT.evgen.TXT.e5706/TXT.10406367._003729.tar.gz.1> .... and more file-path like this.
Is it possible to use this:
- inputGeneratorFile = mc15_13TeV.345054.PowhegPythia8EvtGen_NNPDF3_AZNLO_WpH125J_MINLO_lvbb_VpT.evgen.TXT.e5706
And let rucio download 99 files? It is practically inconvenient to edit log.generate.short and put 99 file names.
Cheers,
Ablethttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/221CI has to run on AlamLinux9 environment2024-01-31T07:55:48+01:00Yiming AbulaitiCI has to run on AlamLinux9 environmentHi All,
The new release 23.6.20 is available only on the AlamLinux9 environment.
But GIT CI don't support AL9 yet.
So in order to test new JOs with 23.6.20 CI has to run on AL9.
Depends on https://gitlab.cern.ch/atlas/athena/-/merge_re...Hi All,
The new release 23.6.20 is available only on the AlamLinux9 environment.
But GIT CI don't support AL9 yet.
So in order to test new JOs with 23.6.20 CI has to run on AL9.
Depends on https://gitlab.cern.ch/atlas/athena/-/merge_requests/67225Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/220Query ProdSys instead of rucio when checking for registered files2023-11-09T15:57:22+01:00Spyros ArgyropoulosQuery ProdSys instead of rucio when checking for registered filesCurrently when `[skip modfiles]` is used `notify.sh` is checking on rucio if samples with a given DSID exist.
When a sample is obsoleted it apparently takes very long for the information to propagate so we should check if we can use Prod...Currently when `[skip modfiles]` is used `notify.sh` is checking on rucio if samples with a given DSID exist.
When a sample is obsoleted it apparently takes very long for the information to propagate so we should check if we can use ProdSys instead.Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/219logParser run failed.2023-10-30T17:39:36+01:00Yiming AbulaitilogParser run failed.logParser run failed due to the unprotected parameter "nEventsPerJob_fromJO". See the error message bellow
'''
- Number of input LHE events: 65000
Traceback (most recent call last):
File "./scripts/logParser.py", line 782, in <module>...logParser run failed due to the unprotected parameter "nEventsPerJob_fromJO". See the error message bellow
'''
- Number of input LHE events: 65000
Traceback (most recent call last):
File "./scripts/logParser.py", line 782, in <module>
main()
File "./scripts/logParser.py", line 683, in main
if expected_EVNT_out > 2 * nEventsPerJob_fromJO:
TypeError: unsupported operand type(s) for *: 'int' and 'NoneType'
'''
In line https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/blob/master/scripts/logParser.py?ref_type=heads#L683
The variable "nEventsPerJob_fromJO" is used but it can be None type when the neventsPerjob is not specified in JO file.
You could just you "nEventsPerJob" variable since it is already overwritten by "nEventsPerJob_fromJO" or set to 10000 if "nEventsPerJob_fromJO" is None.
see line: https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/blame/master/scripts/logParser.py#L518
For test, you can download a log.generate file here: https://cernbox.cern.ch/s/U86AjY5bTjTACwy
Cheers,
AbletSpyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/2185360 GeV shoudl be an allowed energy in the CI2023-10-20T11:15:28+02:00Jan Kretzschmar5360 GeV shoudl be an allowed energy in the CIhttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/2669
Run 3 HI and reference run energy will be 5360 GeV, so the CI should not fail with
"ERROR: unknown ecmEnergy: 5360.0"
Full log file also on lxplus in ~jkretz/pu...https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/2669
Run 3 HI and reference run energy will be 5360 GeV, so the CI should not fail with
"ERROR: unknown ecmEnergy: 5360.0"
Full log file also on lxplus in ~jkretz/public/log.generateSpyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/217How can I run the CI longer then 24 hours?2023-10-04T17:34:07+02:00Yiming AbulaitiHow can I run the CI longer then 24 hours?Hi,
Some metrics element calculation in MG takes longer than 24 hours. But the CI time limit is 24 hours.
Is it allowed to run the CI longer than 24 hours?
Cheers.
AbletHi,
Some metrics element calculation in MG takes longer than 24 hours. But the CI time limit is 24 hours.
Is it allowed to run the CI longer than 24 hours?
Cheers.
Ablethttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/214Assigning same DSID twice2023-09-15T19:06:30+02:00Yiming AbulaitiAssigning same DSID twiceWhen I try to generate new DSID, the same DSID assigned twice "524xxx/524547"
Bellow is the output from "./scripts/commit_new_dsid -m "Commit message" --dry-run ../100xxx/*"
Will move ../100xxx/100003 to 524xxx/524546.
...When I try to generate new DSID, the same DSID assigned twice "524xxx/524547"
Bellow is the output from "./scripts/commit_new_dsid -m "Commit message" --dry-run ../100xxx/*"
Will move ../100xxx/100003 to 524xxx/524546.
Will move ../100xxx/100005 to 524xxx/524547.
Will move ../100xxx/100000 to 524xxx/524546.
Will move ../100xxx/100001 to 524xxx/524547.
Will move ../100xxx/100007 to 602xxx/602485.
Will move ../100xxx/100008 to 602xxx/602486.
Cheers,
AbletSpyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/213Job Failed #314314072023-07-31T17:48:45+02:00Yanlin LiuJob Failed #31431407Job [#31431407](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/31431407) failed for 2afb6c436741b53d42df6790447447250439f02d:
Dear expert, I'm not quite sure what's this error about? Any insights?
Thanks,
YanlinJob [#31431407](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/31431407) failed for 2afb6c436741b53d42df6790447447250439f02d:
Dear expert, I'm not quite sure what's this error about? Any insights?
Thanks,
YanlinSpyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/212Allow running `mc??_valid` input files2023-07-31T09:40:55+02:00Spyros ArgyropoulosAllow running `mc??_valid` input filesCurrently the CI cannot run on input files named like `mc15_valid.602232.Ph_ttj_MiNNLO_scale5_LHE.evgen.TXT.e8531/TXT.34045098._000002.tar.gz.1`
The `valid` replaces the COM energy so it is potentially a not well defined naming conventi...Currently the CI cannot run on input files named like `mc15_valid.602232.Ph_ttj_MiNNLO_scale5_LHE.evgen.TXT.e8531/TXT.34045098._000002.tar.gz.1`
The `valid` replaces the COM energy so it is potentially a not well defined naming convention, since the COM energy is taken automatically from `ecmEnergy` from `log.generate` which is [directly printed from the transform](https://gitlab.cern.ch/atlas/athena/-/blob/main/Generators/EvgenJobTransforms/share/skel.GENtoEVGEN.py#L105) - from the command line arguments.
One solution is to ignore the `ecmEnergy` completely if we see that the input filed is named as `mc??_valid` however I think that's problematic because an LHE should correspond to a given COM energy and we should not allow it to be used indepndently of the `ecmEnergy`.
@katharin @mgignac @dhirschSpyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/211Job Failed #309375482023-07-11T15:39:23+02:00Yiming AbulaitiJob Failed #30937548Job [#30937548](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/30937548) failed for af0d387553a5529f8514e942f2756bd1e17166e1:
How can I pass this?
If I use full container name "mc15_13TeV.345054.PowhegPythia8EvtGen_NNPDF3_...Job [#30937548](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/30937548) failed for af0d387553a5529f8514e942f2756bd1e17166e1:
How can I pass this?
If I use full container name "mc15_13TeV.345054.PowhegPythia8EvtGen_NNPDF3_AZNLO_WpH125J_MINLO_lvbb_VpT.evgen.TXT.e
5706" then the CI job try to download 500K files.
If I use this "mc15_13TeV:TXT.10406367._041965.tar.gz.1, ...." in log.generate.short then some script add one extra "mc15_13TeV". Then the download failed because rucio can not download "mc15_13TeV:mc15_13TeV:TXT.10406367._041965.tar.gz.1".
If I use only "TXT.10406367._041965.tar.gz.1, ...." format as inputs then CI job failed saying "could not find scope". see the link above.
Any solution for this kind of thing?
I tested locally that "rucio download TXT.10406367._041965.tar.gz.1" works fine, but CI script complain about scope.
Cheers,
AbletSpyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/210Make some jo checks stricter?2024-02-18T07:23:36+01:00Spyros ArgyropoulosMake some jo checks stricter?```
runArgs.inputGeneratorFile = outputDS.replace('tar.gz', 'events')
```
passes the check since when running the jO outside the transform it leads to an undefined object.
Maybe we need another way to avoid such issues.```
runArgs.inputGeneratorFile = outputDS.replace('tar.gz', 'events')
```
passes the check since when running the jO outside the transform it leads to an undefined object.
Maybe we need another way to avoid such issues.Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/209commit_new_dsid: Symbolic link is not updated2023-05-20T16:26:07+02:00Yiming Abulaiticommit_new_dsid: Symbolic link is not updatedHi,
When I use "commit_new_dsid" to copy JO directory to GIT. Symbolic links are not updated with newly assigned DSIDS.
I used this command to move to directory to GIT area:
```
./scripts/commit_new_dsid -m "Commit message" -n ../100xx...Hi,
When I use "commit_new_dsid" to copy JO directory to GIT. Symbolic links are not updated with newly assigned DSIDS.
I used this command to move to directory to GIT area:
```
./scripts/commit_new_dsid -m "Commit message" -n ../100xxx/*
OK: ../100xxx/100000 is dir
OK: ../100xxx/100001 is dir
```
Then I checked the new directory(DSIDs).
```
[yabulait@lxplus750 mcjoboptions]$ ls -l 523xxx/523102/
lrwxrwxrwx. 0 yabulait zp 41 May 18 15:53 DirectPhotonFilter.py -> ../../100xxx/100000/DirectPhotonFilter.py
-rw-r--r--. 1 yabulait zp 163 May 18 15:53 log.generate.short
lrwxrwxrwx. 0 yabulait zp 66 May 18 15:53 MadGraphControl_MGPy8EG_DMS1_dijetgamma_pta.py -> ../../100xxx/100000/MadGraphControl_MGPy8EG_DMS1_dijetgamma_pta.py
```
Here you can see that the symbolic links are still "../../100xxx/100000/*", but it should be "../../523xxx/523101".
This happened few times recently. So I reported here.
Cheers,
AbletSpyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/208Speed up retrieval of free DSIDs2023-05-20T16:26:07+02:00Spyros ArgyropoulosSpeed up retrieval of free DSIDsSpyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/207Job Failed #281910882023-05-11T07:28:09+02:00Yanlin LiuJob Failed #28191088Job [#28191088](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/28191088) failed for edb66c3a931427b434623f6c819b32e1bf786625:
Dear @sargyrop , @dhirsch and @mgignac,
Sorry to bother you. Could you help to advice on how to...Job [#28191088](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/28191088) failed for edb66c3a931427b434623f6c819b32e1bf786625:
Dear @sargyrop , @dhirsch and @mgignac,
Sorry to bother you. Could you help to advice on how to resolve this issue?
Thanks,
Yanlinhttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/206Investigate whether the comment from mcgensvc in the notify job can be made i...2023-03-09T12:39:28+01:00Spyros ArgyropoulosInvestigate whether the comment from mcgensvc in the notify job can be made in a way that needs resolvingSpyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/205inputFilesPerJob check in logParser2023-03-14T16:19:16+01:00Spyros ArgyropoulosinputFilesPerJob check in logParserHi Spyros Argyropoulos
we would need an additional check added to JOs parser,
all JOs that are run with Gen_tf.py parameter --inputGeneratorFile should contain the setup of inputFilesPerJob, as done here:
https://gitlab.cern.ch/atlas-ph...Hi Spyros Argyropoulos
we would need an additional check added to JOs parser,
all JOs that are run with Gen_tf.py parameter --inputGeneratorFile should contain the setup of inputFilesPerJob, as done here:
https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/blob/master/601xxx/601755/mc.PhH7EG_PDF4LHC21_ggZH125_Zincl_gammagamma.py
but forgotten here:
https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/blob/master/601xxx/601757/mc.PhH7EG_PDF4LHC21_VBFH125_gammagamma.py
In fact transform sets in such situation default inputFilesPerPerJob=1, but ProdSys is not able to check it and does not want to start the job if this parameter is missing in the JOs.
Thanks,
ewelinA
AGENE-2179Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/204Apply basic tests also to fragments included from top jO2023-12-20T18:58:13+01:00Spyros ArgyropoulosApply basic tests also to fragments included from top jOVia check_jo_content probablyVia check_jo_content probablySpyros ArgyropoulosSpyros Argyropoulos