MC Job Options issueshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues2022-08-25T09:35:17+02:00https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/186Automatically merging MR from pipeline2022-08-25T09:35:17+02:00Spyros ArgyropoulosAutomatically merging MR from pipelineWhen running `merge_request_api.sh -m` from the pipeline we get a method not allowed because the pipeline is still running.
Need to fix this in order to enable an automatic merging - perhaps turn it into a "merge when pipeline succeeds...When running `merge_request_api.sh -m` from the pipeline we get a method not allowed because the pipeline is still running.
Need to fix this in order to enable an automatic merging - perhaps turn it into a "merge when pipeline succeeds"
See !2001Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/185Rebasing branching for automatically created MR when merging2024-02-18T07:24:20+01:00Spyros ArgyropoulosRebasing branching for automatically created MR when mergingWhen trying to merge automatically created MR we can get
```
./scripts/merge_request_api.sh -m
MR: 1993 - approvals left: 0
Merging 1993
{"message":"Branch cannot be merged"}
```
when previous MRs were merged in between.
One would have...When trying to merge automatically created MR we can get
```
./scripts/merge_request_api.sh -m
MR: 1993 - approvals left: 0
Merging 1993
{"message":"Branch cannot be merged"}
```
when previous MRs were merged in between.
One would have to rebase:
![Screenshot_2022-08-23_at_09.34.00](/uploads/d2d8219eaf65ec850b6169a63c62b2c8/Screenshot_2022-08-23_at_09.34.00.png)
from the CI which will launch another pipeline.
Need to see how to treat this in the pipeline.Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/184Job Failed #240867952022-08-22T12:01:09+02:00Yanlin LiuJob Failed #24086795Hi @aivina, there seems an error for the pipeline after you merged the JOs. I'm not sure what caused the issue. Do you have any idea?
Also tagging @sargyrop in case he has any insights on this.
Job [#24086795](https://gitlab.cern.ch/atl...Hi @aivina, there seems an error for the pipeline after you merged the JOs. I'm not sure what caused the issue. Do you have any idea?
Also tagging @sargyrop in case he has any insights on this.
Job [#24086795](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/24086795) failed for e886f2393fba727fbd83c8c44eb0db19f88926cc:https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/183Automatic creation of MR2022-08-23T08:37:22+02:00Spyros ArgyropoulosAutomatic creation of MR- [x] Create script that adds checks and creates MR
- [x] Approve MR
- [x] Merge open MR
- [x] Incorporate the script into CI
- [x] Add relevant parts in commit script- [x] Create script that adds checks and creates MR
- [x] Approve MR
- [x] Merge open MR
- [x] Incorporate the script into CI
- [x] Add relevant parts in commit scriptSpyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/182Job Failed #230315882022-07-05T20:49:55+02:00Yanlin LiuJob Failed #23031588Job [#23031588](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/23031588) failed for b97c45079ee2667cd494db53c8f56fcaee597831:
Hi @sargyrop ,
Can I bother you with another question related to this error? It seems that the...Job [#23031588](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/23031588) failed for b97c45079ee2667cd494db53c8f56fcaee597831:
Hi @sargyrop ,
Can I bother you with another question related to this error? It seems that the event generation of pipeline succeeded, but the log parser step failed and it says "that Number of input LHE events: 174 <-- Needs to be higher than 233". I wonder what's the motivation behind for 233 events here.
Tagging the requester @dai .
Thank you!
Best,
Yanlinhttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/181Job Failed #230158082022-07-05T15:02:11+02:00Yanlin LiuJob Failed #23015808Dear experts,
I got an error for the pipeline:
Job [#23015808](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/23015808) failed for ad8e70db868cd7e4408ea26a93d08be00e2bc4a7:
Actually I used the softlink to the control file...Dear experts,
I got an error for the pipeline:
Job [#23015808](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/23015808) failed for ad8e70db868cd7e4408ea26a93d08be00e2bc4a7:
Actually I used the softlink to the control file previously registered in 502xxx/502547/MGCtrl_Py8EG_NNPDF30nlo_Leptophilicmutau_2muZp_4mu_4pt2.py. I'm wondering what is the cause here.
Thank you!
Best,
YanlinSpyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/180Bug in check for non-reachable files2022-06-29T10:45:47+02:00Spyros ArgyropoulosBug in check for non-reachable filesATLMCPROD-10024
```
grep include mc.Ph_PDF4LHC21_WpH125J_Wincl_MINLO_LHE.py
include("PowhegControl/PowhegControl_HWj_Common.py")
include("Pythia8_i/Pythia8_A14_NNPDF23LO_EvtGen_Common.py")
include("Pythia8_i/Pythia8_Powheg_Main31.py"...ATLMCPROD-10024
```
grep include mc.Ph_PDF4LHC21_WpH125J_Wincl_MINLO_LHE.py
include("PowhegControl/PowhegControl_HWj_Common.py")
include("Pythia8_i/Pythia8_A14_NNPDF23LO_EvtGen_Common.py")
include("Pythia8_i/Pythia8_Powheg_Main31.py")
include("Pythia8_SMHiggs125_inc.py")
```
This appears as a bug because `PowhegControl` and `Pythia8_i` are known to `Gen_tf` but are obviously not present in the DSID directory. To do this properly one would actually need to run `Gen_tf` (where `Gen_tf` looks for the jO is based on what is in the cmake file which might change).
So basically all tests should be removed. Perhaps one which can stay is to check if there is any include pointing to `afs` but this only happened once in 1500 MRs, so I would prefer to completely remove this check.
@mgignac any objection?Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/179run_athena.sh: ERROR: could not find scope2022-06-24T16:59:38+02:00Yiming Abulaitirun_athena.sh: ERROR: could not find scopeHi,
When I try to commit new JOs which use LHE files as input. I got ERROR: could not find scope from run_athena.sh scripts.
I tried different formats in log.generate.short file:
inputGeneratorFile = mc15_13TeV:TXT.16395453._000001.tar....Hi,
When I try to commit new JOs which use LHE files as input. I got ERROR: could not find scope from run_athena.sh scripts.
I tried different formats in log.generate.short file:
inputGeneratorFile = mc15_13TeV:TXT.16395453._000001.tar.gz.1
or
inputGeneratorFile = TXT.16395453._000001.tar.gz.1
or
inputGeneratorFile = TXT.16395453._000001.events
None of the these worked. The run_athena.sh is searching `*"mc15"*` in inputGeneratorFile file name, but if I add "mc15_13TeV:" to the inputGeneratorFile then latter it complain that it can not downlond "mc15_13TeV:mc15_13TeV:<files>" because of two scop.
In the past this "inputGeneratorFile = TXT.16395453._000001.tar.gz.1" works fine. But now I am not sure how to pass the LHE file names.
The original JIRA request: https://its.cern.ch/jira/browse/ATLMCPROD-10013
one of the failed pipeline: https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/22771311
my commit branch: dsid_yabulait_601385
Please let me know if I missed something or if there is a potential bug in the run_athena.sh.
Cheers,
AbletSpyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/178Catch users trying to include files in private paths2022-06-27T20:18:19+02:00Matthew GignacCatch users trying to include files in private pathsI doubt this happens again in the future, so probably pretty low priority, but it would be nice to introduce some checks into the CI to ensure users aren't doing something crazy, like trying to import from user defined afs paths...
See...I doubt this happens again in the future, so probably pretty low priority, but it would be nice to introduce some checks into the CI to ensure users aren't doing something crazy, like trying to import from user defined afs paths...
See: https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/1885
Cheers,
Matthewhttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/177Allow Pythia8_ShowerWeights to be used with Pythia >= 8.3072022-06-06T07:01:42+02:00Spyros ArgyropoulosAllow Pythia8_ShowerWeights to be used with Pythia >= 8.307See AGENE-1478See AGENE-1478Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/176Job Failed #220519692022-05-26T00:34:53+02:00Jan KretzschmarJob Failed #22051969Job [#22051969](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/22051969) failed for 6964c28821e9ec0e33f4548a73191e02293e6c5b:
Hi @sargyrop , this pipeline appears to throw a couple of warnings and errors eventually. This i...Job [#22051969](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/22051969) failed for 6964c28821e9ec0e33f4548a73191e02293e6c5b:
Hi @sargyrop , this pipeline appears to throw a couple of warnings and errors eventually. This is using a pretty old SL6 release, so maybe it cannot be fixed...? Maybe you can just have a quick look and let us know.
thanks, JanSpyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/175Follow-up from "Allow to run mc21_13p6 E2E jobs in CI"2022-05-17T11:08:25+02:00Spyros ArgyropoulosFollow-up from "Allow to run mc21_13p6 E2E jobs in CI"Use grep to extract scope of EVNT files to download in E2E jobs instead of hardcoding
The following discussion from !1831 should be addressed:
- [ ] @cgutscho started a [discussion](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions...Use grep to extract scope of EVNT files to download in E2E jobs instead of hardcoding
The following discussion from !1831 should be addressed:
- [ ] @cgutscho started a [discussion](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/1831#note_5560365): (+8 comments)
> The energy might not be in the log, but it is in the file, no? If we grep the log for the release and set it up we could run `checkMetaSG.py` on the file, e.g.
> ```
> checkMetaSG EVNT.24960681._002005.pool.root.1 | grep beam_energy
> beam_energy | [6500000.0]
> | beam_energy | int | 6500000
> ```
> which is a bit more faff but should work "in general"?https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/174Arithmetic expression error in check_logParser.sh2022-05-17T16:23:22+02:00Spyros ArgyropoulosArithmetic expression error in check_logParser.shSee https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/21845100See https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/21845100https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/173logParser crash when using gridpacks2022-05-17T10:36:18+02:00Dominic HirschbuehllogParser crash when using gridpacksWhen running the logParser for a log.generate which used a gridpack, it crashes with:
"/afs/cern.ch/work/d/dhirsch/stop13TeV/mcjoboptions/scripts/logParser.py", line 368, in powhegChecks
if not glob.glob(f"{os.path.dirname(logFile)}/...When running the logParser for a log.generate which used a gridpack, it crashes with:
"/afs/cern.ch/work/d/dhirsch/stop13TeV/mcjoboptions/scripts/logParser.py", line 368, in powhegChecks
if not glob.glob(f"{os.path.dirname(logFile)}/mc_*TeV.*.GRID.tar.gz"):
File "/cvmfs/atlas.cern.ch/repo/sw/software/22.6/sw/lcg/releases/LCG_101_ATLAS_18/Python/3.9.6/x86_64-centos7-gcc11-opt/lib/python3.9/posixpath.py", line 152, in dirname
p = os.fspath(p)
TypeError: expected str, bytes or os.PathLike object, not list
The problem is that the function
powhegChecks(logFile)
expects a filename, but gets the file content.
A similar problem should happen for Madgraph, since the function is defined properly
with def madgraphChecks(logContent), but the variable logFile used to check the gridpack
is nowwhere defined.
Cheers
DominicSpyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/172EVNTtoEVNT and ECM: fails to download input EVNT in CI2022-05-17T16:03:17+02:00Matthew GignacEVNTtoEVNT and ECM: fails to download input EVNT in CIHi,
A few of the CI jobs fail for some E2E jOs in mc21: https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/21697381
Looks like the $ECMENERGY environment variable is not correctly set. In this type of transform, the ECM is no...Hi,
A few of the CI jobs fail for some E2E jOs in mc21: https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/21697381
Looks like the $ECMENERGY environment variable is not correctly set. In this type of transform, the ECM is not required and not written into the log, which is probably the underlying issue. Any ideas on how to detect this?
As an aside, I noticed that the log.generate.short is claiming ecmEnergy of 13000 GeV, though the transform was run with 13600 TeV (not that it matters...). I guess the 13000 is a default somewhere, which maybe is causing some issues?
Cheers,
MatthewSpyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/171CI download to check for grid input LHEs with mc21_13p6TeV scope2022-05-10T07:28:54+02:00Jan KretzschmarCI download to check for grid input LHEs with mc21_13p6TeV scopeIn https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/1824 I hit the problem, that Grid input LHEs are always trying to be downloaded with mc15_13TeV or mc16_13TeV scope, I think the origin of this is here:
https://gi...In https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/1824 I hit the problem, that Grid input LHEs are always trying to be downloaded with mc15_13TeV or mc16_13TeV scope, I think the origin of this is here:
https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/blob/master/scripts/run_athena.sh#L115-138
First, it would probably be preferable, if we get the CME string from the CME value rather than fixing it to 13 TeV.
Second, we now also need to allow mc21
As shortest patch we could just keep extending the list of allowed scopes, here mc21_13p6TeV being the obvious one to add immediately.
Many thanks, Janhttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/170New commit script failing for directories with arbitrary names2022-05-06T12:59:13+02:00Spyros ArgyropoulosNew commit script failing for directories with arbitrary names![Screenshot_2022-05-06_at_11.16.03](/uploads/a87f3ca9c98c1d19745c55cc15380114/Screenshot_2022-05-06_at_11.16.03.png)
This is because of:
```
newDSID += parseDSIDList(args.DSID)
```
seems to be adding an empty item in the list![Screenshot_2022-05-06_at_11.16.03](/uploads/a87f3ca9c98c1d19745c55cc15380114/Screenshot_2022-05-06_at_11.16.03.png)
This is because of:
```
newDSID += parseDSIDList(args.DSID)
```
seems to be adding an empty item in the listChristian GutschowChristian Gutschowhttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/169CI job to check if EVNTs have been obsoleted to run for modified jO2022-11-16T14:32:57+01:00Spyros ArgyropoulosCI job to check if EVNTs have been obsoleted to run for modified jO- if a jO is modified
- check if EVNT containers exist & have > 0 associated files, if so throw an error
e.g. https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/1795- if a jO is modified
- check if EVNT containers exist & have > 0 associated files, if so throw an error
e.g. https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/1795Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/168logParser unsupported locale setting2022-04-21T20:53:54+02:00Spyros ArgyropouloslogParser unsupported locale settingFull error message:
INFO: New DSID directory: 100xxx/100001 ...
OK: log.generate file found.
Traceback (most recent call last):
File "scripts/logParser.py", line 8, in <module>
locale.setlocale(locale.LC_CTYPE, f'{lang}.UTF-8'...Full error message:
INFO: New DSID directory: 100xxx/100001 ...
OK: log.generate file found.
Traceback (most recent call last):
File "scripts/logParser.py", line 8, in <module>
locale.setlocale(locale.LC_CTYPE, f'{lang}.UTF-8')
File "/usr/lib64/python3.6/locale.py", line 598, in setlocale
return _setlocale(category, locale)
locale.Error: unsupported locale setting
ERROR: logParser run failed.
Need output of
- locale
- locale -a
- env
- which machine you are running on
@yanlin @nishuSpyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/167Fix logParser bug when running in CI2022-04-20T22:28:20+02:00Spyros ArgyropoulosFix logParser bug when running in CI![Screenshot_2022-04-20_at_21.08.27](/uploads/ed6760a1cbe977212c0904faf484fd7a/Screenshot_2022-04-20_at_21.08.27.png)![Screenshot_2022-04-20_at_21.08.27](/uploads/ed6760a1cbe977212c0904faf484fd7a/Screenshot_2022-04-20_at_21.08.27.png)Spyros ArgyropoulosSpyros Argyropoulos