MC Job Options issueshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues2023-10-30T17:39:36+01:00https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/219logParser run failed.2023-10-30T17:39:36+01:00Yiming AbulaitilogParser run failed.logParser run failed due to the unprotected parameter "nEventsPerJob_fromJO". See the error message bellow
'''
- Number of input LHE events: 65000
Traceback (most recent call last):
File "./scripts/logParser.py", line 782, in <module>...logParser run failed due to the unprotected parameter "nEventsPerJob_fromJO". See the error message bellow
'''
- Number of input LHE events: 65000
Traceback (most recent call last):
File "./scripts/logParser.py", line 782, in <module>
main()
File "./scripts/logParser.py", line 683, in main
if expected_EVNT_out > 2 * nEventsPerJob_fromJO:
TypeError: unsupported operand type(s) for *: 'int' and 'NoneType'
'''
In line https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/blob/master/scripts/logParser.py?ref_type=heads#L683
The variable "nEventsPerJob_fromJO" is used but it can be None type when the neventsPerjob is not specified in JO file.
You could just you "nEventsPerJob" variable since it is already overwritten by "nEventsPerJob_fromJO" or set to 10000 if "nEventsPerJob_fromJO" is None.
see line: https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/blame/master/scripts/logParser.py#L518
For test, you can download a log.generate file here: https://cernbox.cern.ch/s/U86AjY5bTjTACwy
Cheers,
AbletSpyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/194Notify changes assigns MR to wrong person2022-11-20T17:08:34+01:00Spyros ArgyropoulosNotify changes assigns MR to wrong personneed to assign to convenersneed to assign to convenersSpyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/193Turn off branch pipelines2023-01-03T08:22:29+01:00Spyros ArgyropoulosTurn off branch pipelinesIf someone opens a MR when a branch pipeline is still running 2 concurrent pipelines are created.
One will fail since the one that finishes first will push to the branch and then the last CI job will try to push to a branch that is beh...If someone opens a MR when a branch pipeline is still running 2 concurrent pipelines are created.
One will fail since the one that finishes first will push to the branch and then the last CI job will try to push to a branch that is behind.
![Screenshot_2022-11-17_at_16.36.11](/uploads/afe31bf16cbf3496ccb1bf6f2703d7ac/Screenshot_2022-11-17_at_16.36.11.png)
We should turn off all branch pipelines.Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/192setup_athena pipeline failing2022-11-17T10:18:51+01:00Spyros Argyropoulossetup_athena pipeline failingSee !2152
The branch pipeline succeeds: https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/pipelines/4772445
Merge request pipeline succeeds when `log.generate.short` is present: https://gitlab.cern.ch/atlas-physics/pmg/mcjobopti...See !2152
The branch pipeline succeeds: https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/pipelines/4772445
Merge request pipeline succeeds when `log.generate.short` is present: https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/25883448
Merge request pipeline fails when `log.generate.short` is not present: https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/25883547
Branch pipeline also fails when `log.generate.short` is not present: https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/25883631Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/190Remove possibility to skip all pipelines2022-10-08T10:37:24+02:00Spyros ArgyropoulosRemove possibility to skip all pipelinesWe should remove the `[skip all]` option since it is abused with no reason.
Need to think how to redesign the pipeline to make this happen.We should remove the `[skip all]` option since it is abused with no reason.
Need to think how to redesign the pipeline to make this happen.Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/180Bug in check for non-reachable files2022-06-29T10:45:47+02:00Spyros ArgyropoulosBug in check for non-reachable filesATLMCPROD-10024
```
grep include mc.Ph_PDF4LHC21_WpH125J_Wincl_MINLO_LHE.py
include("PowhegControl/PowhegControl_HWj_Common.py")
include("Pythia8_i/Pythia8_A14_NNPDF23LO_EvtGen_Common.py")
include("Pythia8_i/Pythia8_Powheg_Main31.py"...ATLMCPROD-10024
```
grep include mc.Ph_PDF4LHC21_WpH125J_Wincl_MINLO_LHE.py
include("PowhegControl/PowhegControl_HWj_Common.py")
include("Pythia8_i/Pythia8_A14_NNPDF23LO_EvtGen_Common.py")
include("Pythia8_i/Pythia8_Powheg_Main31.py")
include("Pythia8_SMHiggs125_inc.py")
```
This appears as a bug because `PowhegControl` and `Pythia8_i` are known to `Gen_tf` but are obviously not present in the DSID directory. To do this properly one would actually need to run `Gen_tf` (where `Gen_tf` looks for the jO is based on what is in the cmake file which might change).
So basically all tests should be removed. Perhaps one which can stay is to check if there is any include pointing to `afs` but this only happened once in 1500 MRs, so I would prefer to completely remove this check.
@mgignac any objection?Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/167Fix logParser bug when running in CI2022-04-20T22:28:20+02:00Spyros ArgyropoulosFix logParser bug when running in CI![Screenshot_2022-04-20_at_21.08.27](/uploads/ed6760a1cbe977212c0904faf484fd7a/Screenshot_2022-04-20_at_21.08.27.png)![Screenshot_2022-04-20_at_21.08.27](/uploads/ed6760a1cbe977212c0904faf484fd7a/Screenshot_2022-04-20_at_21.08.27.png)Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/166Add checks for python2/3 compatibility of jO2022-04-21T17:21:24+02:00Spyros ArgyropoulosAdd checks for python2/3 compatibility of jOan example DSID is 830099: https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/blob/master/830xxx/830099/mc.H7EG_jetjet_72_Cluster_JZ1.py
R21: 21.6.85
R22: You can try 22.6.13 (later releases have issues with EvtGen_i — should be ...an example DSID is 830099: https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/blob/master/830xxx/830099/mc.H7EG_jetjet_72_Cluster_JZ1.py
R21: 21.6.85
R22: You can try 22.6.13 (later releases have issues with EvtGen_i — should be fixed soon).Spyros ArgyropoulosSpyros Argyropoulos2022-04-25https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/144Pipelines failing when only links are included?2021-06-21T16:50:31+02:00Spyros ArgyropoulosPipelines failing when only links are included?The following discussion from !1225 should be addressed:
- [ ] @jshahini started a [discussion](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/1225#note_4588898): (+1 comment)
> Hi @cgutscho
>
> I...The following discussion from !1225 should be addressed:
- [ ] @jshahini started a [discussion](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/1225#note_4588898): (+1 comment)
> Hi @cgutscho
>
> Indeed it is a duplicate, but this is by design in order to clear the CI. To give some context, these JOs are for a SUSY grid expansion.
>
> I originally tried to upload everything using only symlinks to that control file, but the CI pipelines were failing, claiming that the jobs couldn't find ```MadGraphControl_SimplifiedModel_GG_directRPVLQD.py```
>
> So I duplicated the control file you pointed to and included it in this MR so that the pipelines would succeed. After the MR gets accepted, I was going to make another one where I change all the control files to be symlinks to ```/502xxx/502416/MadGraphControl_SimplifiedModel_GG_directRPVLQD.py```. That way, there would be no duplicated control files floating around.
>
> I realize this is remarkably convoluted, so I'm more than happy to hear other ideas about preparing the JOs for grid expansions in R21.
>
> Cheers,
> Jeff
Failed pipeline: https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/pipelines/2741834
![Screenshot_2021-06-21_at_14.50.38](/uploads/1b1ebf50941d6c15803a23b2ad2bcd32/Screenshot_2021-06-21_at_14.50.38.png)S1.2021Spyros ArgyropoulosSpyros Argyropoulos2021-06-27https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/139Handle EVNT->EVNT jobs in CI and logParser2021-06-17T13:30:43+02:00Spyros ArgyropoulosHandle EVNT->EVNT jobs in CI and logParserThese jobs produce a `log.afterburn` instead of `log.generate`.
- [x] I would need an example to see how to treat this
- [x] How can we identify that it's an EVNT->EVNT job from the log?
- [x] Do we need to modify the Gen_tf command?
-...These jobs produce a `log.afterburn` instead of `log.generate`.
- [x] I would need an example to see how to treat this
- [x] How can we identify that it's an EVNT->EVNT job from the log?
- [x] Do we need to modify the Gen_tf command?
- [x] Test with `700267`S1.2021Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/133scripts/commit_new_dsid.sh crashes when reading directories2021-01-15T11:20:37+01:00Petr Jackascripts/commit_new_dsid.sh crashes when reading directories```
./scripts/commit_new_dsid.sh Var3c/* -m="Message" --dry-run
```
It fails when it tries to convert JOs directories inside Var3c directory with the message:
```
Traceback (most recent call last):
File "scripts/jo_utils.py", line 8...```
./scripts/commit_new_dsid.sh Var3c/* -m="Message" --dry-run
```
It fails when it tries to convert JOs directories inside Var3c directory with the message:
```
Traceback (most recent call last):
File "scripts/jo_utils.py", line 87, in <module>
_parse(args.DSIDs)
File "scripts/jo_utils.py", line 10, in _parse
dsids = [ int(d) for d in dsids ] # turn strings to integers
File "scripts/jo_utils.py", line 10, in <listcomp>
dsids = [ int(d) for d in dsids ] # turn strings to integers
ValueError: invalid literal for int() with base 10: 'Var3c/py8_yprod_var3cDown'
```
This issue was introduced in this commit: https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/commit/2546cd6015fd7a1b95ebfeafa31613c1645e421a
It is still possible to run the script when directories are renamed into dummy dsid numbers
./scripts/commit_new_dsid.sh -d=100000,100001 -m="Adding ttgamma MG+Py8 Var3c variation samples" --dry-run
I attached a tar file with Var3c directory.
[Var3c.tar.gz](/uploads/2d6676234f4a093b0b06806f8e4e3196/Var3c.tar.gz)S1.2021Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/130Support for Centos7 releases2021-01-10T17:33:29+01:00Christian GutschowSupport for Centos7 releasesStarting with release 21.6.51, the releases are built for Centos7 machines and so we should not be using SLC6 containers in the CI for those anymore (and gridpacks prepared on C7 machines are fine to use for those releases).Starting with release 21.6.51, the releases are built for Centos7 machines and so we should not be using SLC6 containers in the CI for those anymore (and gridpacks prepared on C7 machines are fine to use for those releases).S1.2021Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/129Improve handling of madgraph checks2021-01-01T18:32:48+01:00Spyros ArgyropoulosImprove handling of madgraph checksInstead of reading the whole file for the madgraphchecks make use of appropriate dictionary, where values can be overwritten.
ATLMCPROD-8252Instead of reading the whole file for the madgraphchecks make use of appropriate dictionary, where values can be overwritten.
ATLMCPROD-8252S1.2021Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/127ERROR Directory <dsid> does not exist2020-11-17T10:07:49+01:00Frank SiegertERROR Directory <dsid> does not existWith the attached [mcjoboptions.tar.gz](/uploads/f37339d16c15c97b60c40ca97f0aeb54/mcjoboptions.tar.gz) I have problems adding a new setup using the commit script. The `--dry-run` works fine, but the `-n` (or identically without `-n`) err...With the attached [mcjoboptions.tar.gz](/uploads/f37339d16c15c97b60c40ca97f0aeb54/mcjoboptions.tar.gz) I have problems adding a new setup using the commit script. The `--dry-run` works fine, but the `-n` (or identically without `-n`) errors out as follows:
```
[15:29 tauruslogin3: mcjoboptions]$ ./scripts/commit_new_dsid.sh wip/testForSpyros -m='Sherpa 2.2.10 test for Spyros' --dry-run
INFO: will use following remote for pushing: origin
Will use branch: dsid_fsiegert_wiptestForSpyros...
Will create new branch: dsid_fsiegert_wiptestForSpyros
Checking jO consistency and DSID ranges ...
Will move wip/testForSpyros to 700xxx/700119
New DSID directory: wip/testForSpyros ...
OK: log.generate file found.
OK: log.generate file contains no errors
OK: CI job expected to last less than 1h - time estimate: 0.09 hours
Will now add files to git commit
File: wip/testForSpyros/log.generate cannot be added to the commit. Skipping.
Will add: wip/testForSpyros/log.generate.short
Will add: wip/testForSpyros/mc_13TeV.Sh_2210_tttt_muQHT2.GRID.tar.gz
Will add: wip/testForSpyros/mc.Sh_2210_testForSpyros.py
[15:30 tauruslogin3: mcjoboptions]$ ./scripts/commit_new_dsid.sh wip/testForSpyros -m='Sherpa 2.2.10 test for Spyros' -n
INFO: will use following remote for pushing: origin
Will use branch: dsid_fsiegert_wiptestForSpyros...
Will create new branch: dsid_fsiegert_wiptestForSpyros
Checking jO consistency and DSID ranges ...
ERROR: Directory 700119 does not exist
```S2.2020Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/123logParser picks out wrong COM energy2020-09-21T16:08:05+02:00Christian GutschowlogParser picks out wrong COM energySee e.g. !676 where it extracted `ecmEnergy = 13000` even though the `log.generate` was for 8 TeV:
```
/afs/cern.ch/user/c/cgutscho/public/forSpyros/log.generate
```
Why though?See e.g. !676 where it extracted `ecmEnergy = 13000` even though the `log.generate` was for 8 TeV:
```
/afs/cern.ch/user/c/cgutscho/public/forSpyros/log.generate
```
Why though?S2.2020Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/122Bug: handling of jobs with external LHE file in logParser step2020-09-06T13:46:23+02:00Spyros ArgyropoulosBug: handling of jobs with external LHE file in logParser stepWhen external LHE files are used `log.generate.short` is added to the commit but `run_athena` just skips the job without producing any `log.generate_ci` file. Then the `check_logParser` job thinks this is a bug because if `log.generate.s...When external LHE files are used `log.generate.short` is added to the commit but `run_athena` just skips the job without producing any `log.generate_ci` file. Then the `check_logParser` job thinks this is a bug because if `log.generate.short` is present `log.generate_ci` should also be present as well at this point in the CI and complains see !652S2.2020Spyros ArgyropoulosSpyros Argyropoulos2020-09-04https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/109logParser fails in CI when run on MadGraph due to nevents check2020-05-17T14:24:09+02:00Spyros ArgyropouloslogParser fails in CI when run on MadGraph due to nevents checkAs seen in !412 when running a jO with:
```
evgenConfig.nEventsPerJob = 10000
nevents = runArgs.maxEvents1.2 if runArgs.maxEvents>0 else 1.1evgenConfig.nEventsPerJob
```
`logParser` fails with
```
ERROR: Increase nevents to be gener...As seen in !412 when running a jO with:
```
evgenConfig.nEventsPerJob = 10000
nevents = runArgs.maxEvents1.2 if runArgs.maxEvents>0 else 1.1evgenConfig.nEventsPerJob
```
`logParser` fails with
```
ERROR: Increase nevents to be generated in MG from 120 to 11000
```S1.2020Spyros ArgyropoulosSpyros Argyropoulos2020-05-16https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/108logParser crash in CI not handled correctly2020-05-05T20:25:13+02:00Spyros ArgyropouloslogParser crash in CI not handled correctlySee https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/8197685See https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/8197685S1.2020Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/107logParser crash due to double print of nevents in MG2020-12-09T10:09:13+01:00Spyros ArgyropouloslogParser crash due to double print of nevents in MGlogParser failing again because of double print-out of nevents:
```
22:32:58 Py:MadGraphUtils INFO Setting nevents = 11000.
22:33:05 Py:MadGraphUtils INFO "nevents" = 11000
```
The first printout seemed to be the old implementa...logParser failing again because of double print-out of nevents:
```
22:32:58 Py:MadGraphUtils INFO Setting nevents = 11000.
22:33:05 Py:MadGraphUtils INFO "nevents" = 11000
```
The first printout seemed to be the old implementation before the restructuring in rel. 21.6.23, however I don't understand why both printouts are printed now. Is this expected @zmarshal @hmildner @mcfayden ?
The jO is attached - provided by @ewelina - this was run in 21.6.27.
[mc.MGPy8EG_A14NNPDF23_tWgamma.py](/uploads/66b17b0604410f93d826969cc504c7ef/mc.MGPy8EG_A14NNPDF23_tWgamma.py)
Just to say if this is expected we can easily change the behaviour to parse lines containing `"nevents"` (with quotes) currently it tries to find lines containing `nevents` (without quotes) and since the printout is different (trailing dot) the first print-out is not parsed correctly. S1.2020Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/104Harmonise whitelist with Gen_tf2021-01-04T15:27:52+01:00Spyros ArgyropoulosHarmonise whitelist with Gen_tfCurrently the transform allows setups which are explicitly excluded in the whitelist, e.g. `DSID/dat/*.dat` which is excluded here: https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/blob/master/scripts/whitelist.sh#L9 as discussed ...Currently the transform allows setups which are explicitly excluded in the whitelist, e.g. `DSID/dat/*.dat` which is excluded here: https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/blob/master/scripts/whitelist.sh#L9 as discussed in !298
I no longer remember why we excluded some cases but we should definitely harmonise what is done in the transform and what is done in the CI.
@ewelina could you go through the whitelist and let me know what is treated differently there and in `Gen_tf` so that we harmonise?
Tag @cgutscho @fsiegertS1.2021