MC Job Options issueshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues2020-11-14T13:51:07+01:00https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/125JO shouldn't hardcode ATHENA_PROC_NUMBER2020-11-14T13:51:07+01:00Christian GutschowJO shouldn't hardcode ATHENA_PROC_NUMBERThe environment variable for multi-threading `ATHENA_PROC_NUMBER` should be set by prodsys, not the JOs.
Can we make the CI fail if the JOs try to assign a value to that? (The JO are free to ask if this environment variable exists and w...The environment variable for multi-threading `ATHENA_PROC_NUMBER` should be set by prodsys, not the JOs.
Can we make the CI fail if the JOs try to assign a value to that? (The JO are free to ask if this environment variable exists and what it's value is (e.g. to pass it into Madgraph), but they shouldn't try to overwrite its value
See e.g. MR !745 where this had to be corrected, but e.g. [this JO](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/blob/master/421xxx/421006/mc.MGPy8EG_A14NNPDF23_tWgamma_art.py) where it's used in an acceptable way.S2.2020Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/124New Pythia 8 checks for changing parameters2023-10-26T16:13:33+02:00Spyros ArgyropoulosNew Pythia 8 checks for changing parametersImplement code to use new developments by Giancarlo mentioned in AGENE-1915.
- [ ] To be seen which of these should result in an error and which should be a warning.
- [ ] Also check if this catches the bug reported in ATLMCPROD-7723Implement code to use new developments by Giancarlo mentioned in AGENE-1915.
- [ ] To be seen which of these should result in an error and which should be a warning.
- [ ] Also check if this catches the bug reported in ATLMCPROD-7723S1.2021Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/123logParser picks out wrong COM energy2020-09-21T16:08:05+02:00Christian GutschowlogParser picks out wrong COM energySee e.g. !676 where it extracted `ecmEnergy = 13000` even though the `log.generate` was for 8 TeV:
```
/afs/cern.ch/user/c/cgutscho/public/forSpyros/log.generate
```
Why though?See e.g. !676 where it extracted `ecmEnergy = 13000` even though the `log.generate` was for 8 TeV:
```
/afs/cern.ch/user/c/cgutscho/public/forSpyros/log.generate
```
Why though?S2.2020Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/122Bug: handling of jobs with external LHE file in logParser step2020-09-06T13:46:23+02:00Spyros ArgyropoulosBug: handling of jobs with external LHE file in logParser stepWhen external LHE files are used `log.generate.short` is added to the commit but `run_athena` just skips the job without producing any `log.generate_ci` file. Then the `check_logParser` job thinks this is a bug because if `log.generate.s...When external LHE files are used `log.generate.short` is added to the commit but `run_athena` just skips the job without producing any `log.generate_ci` file. Then the `check_logParser` job thinks this is a bug because if `log.generate.short` is present `log.generate_ci` should also be present as well at this point in the CI and complains see !652S2.2020Spyros ArgyropoulosSpyros Argyropoulos2020-09-04https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/121logParser rejects logs with nEventsPerJob > 10k2020-08-28T16:51:47+02:00Christian GutschowlogParser rejects logs with nEventsPerJob > 10kFollowing the successful test in ATLMCPROD-8659, we should allow cases where `nEventsPerJob` is a multiple of 10k.
Currently it fails saying
```
- CountHepMC Events passing all checks and written = 20000 <-- ERROR: Not an acceptable n...Following the successful test in ATLMCPROD-8659, we should allow cases where `nEventsPerJob` is a multiple of 10k.
Currently it fails saying
```
- CountHepMC Events passing all checks and written = 20000 <-- ERROR: Not an acceptable number of events for production (1, 2, 5, 10, 20, 50, 100, 200, 500, 1000, 2000, 5000, 10000)
```S2.2020Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/120Allow runArgs to be referred to in JOs but not to be overwritten by JOs2020-08-22T13:08:09+02:00Christian GutschowAllow runArgs to be referred to in JOs but not to be overwritten by JOsSee !631 for an example.See !631 for an example.S2.2020Spyros ArgyropoulosSpyros Argyropoulos2020-08-14https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/119Mentioning ATLMCPROD ticket in MR doesn't push link to Jira any longer2020-07-31T10:47:09+02:00Christian GutschowMentioning ATLMCPROD ticket in MR doesn't push link to Jira any longer... not sure there's much we can do about this though?
Any ideas anyone?... not sure there's much we can do about this though?
Any ideas anyone?https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/118Add checks for `inputfilecheck` and `inputGeneratorFile`2020-08-03T10:25:32+02:00Christian GutschowAdd checks for `inputfilecheck` and `inputGeneratorFile`Please see this test commit: 52aa8087
which has the following two lines in the JO:
```
evgenConfig.inputfilecheck = 'PhPy8EG_NNPDF30LO_EWK_ZZeeee'
runArgs.inputGeneratorFile = 'PhPy8EG_NNPDF30LO_EWK_ZZeeee._00052.events.tar.gz'
```
Th...Please see this test commit: 52aa8087
which has the following two lines in the JO:
```
evgenConfig.inputfilecheck = 'PhPy8EG_NNPDF30LO_EWK_ZZeeee'
runArgs.inputGeneratorFile = 'PhPy8EG_NNPDF30LO_EWK_ZZeeee._00052.events.tar.gz'
```
The first one I thought the CI would already be catching [along with `inputconfcheck`, no?] and the second one is clearly a problem for central production.
Can we catch these? I guess the logParser should already throw an error before the files are even committed to gitlab.S2.2020Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/117Check number of files in gridpack2023-03-01T07:42:53+01:00Christian GutschowCheck number of files in gridpackThe number of files in a gridpack shouldn't exceed 80k, otherwise some grid sites will crash. This has happened a number of times recently, e.g. for the FxFx job where the gridpack contained several files per Feynman diagram. MadGraph co...The number of files in a gridpack shouldn't exceed 80k, otherwise some grid sites will crash. This has happened a number of times recently, e.g. for the FxFx job where the gridpack contained several files per Feynman diagram. MadGraph control cleans up logs and .o files in the latest release, but for older releases it would be good to have a dedicated pipeline step that throws an error if the number of files in the gridpack is larger than 80k. Probably something like `tar -ztvf *.tgz *.tar.gz` could work?S2.2020Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/116Don't commit emacs backup files2020-06-23T15:59:46+02:00Christian GutschowDon't commit emacs backup filesCurrently the files ending in `blah~` seem to be included by the commit scripts see e.g. [here](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/tree/master/500xxx/500908).
Cheers,
ChrisCurrently the files ending in `blah~` seem to be included by the commit scripts see e.g. [here](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/tree/master/500xxx/500908).
Cheers,
ChrisSpyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/115Wrong printing of branches using a DSID2020-08-01T16:41:21+02:00Spyros ArgyropoulosWrong printing of branches using a DSIDI had a wrong error message when I tried to commit JOs for 421332:
the message I got was that dsid_jveatch_600076 already uses this DSID.
I have checked this branch and it was not the case.
I found that this DSID was used in one of the e...I had a wrong error message when I tried to commit JOs for 421332:
the message I got was that dsid_jveatch_600076 already uses this DSID.
I have checked this branch and it was not the case.
I found that this DSID was used in one of the earlier branches awaiting approval.
I think the problem is that the list of branches
https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/blob/master/scripts/check_jo_consistency.py#L118
is ordered from the newest branch to the oldest and when a new branch is submitted for merging it is updated for the changes that were introduced in other branches awaiting the approval - this way always the newest one will be pointed as the one using already a given DSID (in case of conflict).S2.2020Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/114Limit of inputFilesPerJob2021-01-21T18:13:58+01:00Xiaohu SunLimit of inputFilesPerJobIn this ticket https://its.cern.ch/jira/browse/ATLMCPROD-8583 the request needs external LHE files.
In order to have 10000 events per job, we have to set inputFilesPerJob to 200 for some of the JOs. But this tiggers an error in logparse...In this ticket https://its.cern.ch/jira/browse/ATLMCPROD-8583 the request needs external LHE files.
In order to have 10000 events per job, we have to set inputFilesPerJob to 200 for some of the JOs. But this tiggers an error in logparser checks that inputFilesPerJob is limited up to 100.
Well, we cannot cut 10000 events to 5000 to allow inputFilesPerJob back in the limit, because that will touch the CPU hour limit (5000 in the JO takes <1 hour to finish in this case).
Do you suggest how to proceed?
Thanks!Christian GutschowChristian Gutschowhttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/113Improve `check_modified_files` behaviour2020-08-04T10:02:29+02:00Spyros ArgyropoulosImprove `check_modified_files` behaviourDo a local rebase before checking what changed to avoid failed pipelines for commits that are behind master.Do a local rebase before checking what changed to avoid failed pipelines for commits that are behind master.S2.2020Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/112CI/logParser addition: maximum value for inputFilesPerJob2020-05-19T10:36:36+02:00Christian GutschowCI/logParser addition: maximum value for inputFilesPerJobThe maximum number of input LHE/EVNT files is `inputFilesPerJob=1000`.
Could this be added to the CI/logParser (whichever is best)?The maximum number of input LHE/EVNT files is `inputFilesPerJob=1000`.
Could this be added to the CI/logParser (whichever is best)?Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/111commit_new_dsid.sh creates wrong links at the -n step2020-06-03T09:54:09+02:00Judita Mamuziccommit_new_dsid.sh creates wrong links at the -n stepDear Experts,
I would like to upload the new JO and the control file for a case when many DSIDs use a same control file, in SUSY. The folder structure is:
```
Dir1:
mc.1.py -> Control.py
Control.py
Dir2:
mc.2.py -> ../Dir1/Control.py
`...Dear Experts,
I would like to upload the new JO and the control file for a case when many DSIDs use a same control file, in SUSY. The folder structure is:
```
Dir1:
mc.1.py -> Control.py
Control.py
Dir2:
mc.2.py -> ../Dir1/Control.py
```
The jobs run successfully, and the first step of checks with commit_new_dsid.sh using --dry-run is also successful. However, when the option -n is used in the second step like:
```
./scripts/commit_new_dsid.sh -d=100001-100082 -n -m="SUSY direct stau, TFilt."
```
the linked files become wrong.
Initial input:
```
ls -lah 100xxx/100001/mc* 100xxx/100002/mc*
100xxx/100001/mc.MGPy8EG_StauStauDirect_120p0_1p0_TFilt.py -> SUSY_SimplifiedModel_StauStauDirect.py
100xxx/100002/mc.MGPy8EG_StauStauDirect_160p0_1p0_TFilt.py -> ../100001/SUSY_SimplifiedModel_StauStauDirect.py
```
After step -n:
```
ls -lah 501xxx/501047/mc* 501xxx/501048/mc*
501xxx/501047/mc.MGPy8EG_StauStauDirect_120p0_1p0_TFilt.py -> SUSY_SimplifiedModel_StauStauDirect.py
501xxx/501048/mc.MGPy8EG_StauStauDirect_160p0_1p0_TFilt.py -> ../../501xxx/501047/mc.MGPy8EG_StauStauDirect_160p0_1p0_TFilt.py
```
where the last file should be:
```
501xxx/501048/mc.MGPy8EG_StauStauDirect_160p0_1p0_TFilt.py -> ../501047/SUSY_SimplifiedModel_StauStauDirect.py
```
It seems there is a problem with the copy of files with a soft link.
I attach here the reduced example.
Many thanks for your help.
Cheers,
Judita
/cc @gstark , @sargyrop , @wfawcett , @cgutscho
[100xxx_short.tar.gz](/uploads/2ac3b683ca55e00c418bc56071864798/100xxx_short.tar.gz)Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/110unwarranted logParser fail at commit-script stage?2020-05-11T15:51:37+02:00Christian Gutschowunwarranted logParser fail at commit-script stage?From @mgignac:
The commit script complained on [this line](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/blob/master/scripts/logParser.py#L222), but I'm not sure that the logic is correct in the logParser. In my log file, I ha...From @mgignac:
The commit script complained on [this line](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/blob/master/scripts/logParser.py#L222), but I'm not sure that the logic is correct in the logParser. In my log file, I have a single message that's being flagged:
`Matrix_Element_Handler::GenerateOneEvent(): Point for '2_3__u__u__W+__d__u' exceeds maximum by 15.4543.`
And when the above line fails, it's dividing by zero (e.g. `nEventsRequested` is not set).
```
Traceback (most recent call last):
File "scripts/logParser.py", line 624, in <module>
main()
File "scripts/logParser.py", line 485, in main
sherpaChecks(opts.INPUT_FILE)
File "scripts/logParser.py", line 223, in sherpaChecks
logwarn("","WARNING: be aware of: "+str(numexceeds*100./nEventsRequested)+"% of the event weights exceed the maximum by a factor of ten")
ZeroDivisionError: float division by zero
```https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/109logParser fails in CI when run on MadGraph due to nevents check2020-05-17T14:24:09+02:00Spyros ArgyropouloslogParser fails in CI when run on MadGraph due to nevents checkAs seen in !412 when running a jO with:
```
evgenConfig.nEventsPerJob = 10000
nevents = runArgs.maxEvents1.2 if runArgs.maxEvents>0 else 1.1evgenConfig.nEventsPerJob
```
`logParser` fails with
```
ERROR: Increase nevents to be gener...As seen in !412 when running a jO with:
```
evgenConfig.nEventsPerJob = 10000
nevents = runArgs.maxEvents1.2 if runArgs.maxEvents>0 else 1.1evgenConfig.nEventsPerJob
```
`logParser` fails with
```
ERROR: Increase nevents to be generated in MG from 120 to 11000
```S1.2020Spyros ArgyropoulosSpyros Argyropoulos2020-05-16https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/108logParser crash in CI not handled correctly2020-05-05T20:25:13+02:00Spyros ArgyropouloslogParser crash in CI not handled correctlySee https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/8197685See https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/8197685S1.2020Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/107logParser crash due to double print of nevents in MG2020-12-09T10:09:13+01:00Spyros ArgyropouloslogParser crash due to double print of nevents in MGlogParser failing again because of double print-out of nevents:
```
22:32:58 Py:MadGraphUtils INFO Setting nevents = 11000.
22:33:05 Py:MadGraphUtils INFO "nevents" = 11000
```
The first printout seemed to be the old implementa...logParser failing again because of double print-out of nevents:
```
22:32:58 Py:MadGraphUtils INFO Setting nevents = 11000.
22:33:05 Py:MadGraphUtils INFO "nevents" = 11000
```
The first printout seemed to be the old implementation before the restructuring in rel. 21.6.23, however I don't understand why both printouts are printed now. Is this expected @zmarshal @hmildner @mcfayden ?
The jO is attached - provided by @ewelina - this was run in 21.6.27.
[mc.MGPy8EG_A14NNPDF23_tWgamma.py](/uploads/66b17b0604410f93d826969cc504c7ef/mc.MGPy8EG_A14NNPDF23_tWgamma.py)
Just to say if this is expected we can easily change the behaviour to parse lines containing `"nevents"` (with quotes) currently it tries to find lines containing `nevents` (without quotes) and since the printout is different (trailing dot) the first print-out is not parsed correctly. S1.2020Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/106Job Failed #81849032020-04-29T21:17:04+02:00Xiaohu SunJob Failed #8184903Job [#8184903](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/8184903) failed for 127510e74ddbca868a29efecbd1b8c6144bf63b8:Job [#8184903](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/8184903) failed for 127510e74ddbca868a29efecbd1b8c6144bf63b8: