MC Job Options issueshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues2019-09-07T11:55:17+02:00https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/16Update run_athena CI job to use Gen_tf2019-09-07T11:55:17+02:00Spyros ArgyropoulosUpdate run_athena CI job to use Gen_tf```
asetup 21.6.6,AthGeneration
Gen_tf.py --ecmEnergy=13000 --jobConfig=421002 --outputEVNTFile=test.EVNT.pool.root
``````
asetup 21.6.6,AthGeneration
Gen_tf.py --ecmEnergy=13000 --jobConfig=421002 --outputEVNTFile=test.EVNT.pool.root
```Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/14Protection for CI jobs triggered by mistake (gitlab bugs)2019-07-03T15:14:27+02:00Spyros ArgyropoulosProtection for CI jobs triggered by mistake (gitlab bugs)Currently the CI job scripts depend on the correct triggering of the jobs by the logic coded into the `gitlab-ci.yml` file.
If a job gets triggered by mistake it will return a failure, because the `grep` commands will not find any mat...Currently the CI job scripts depend on the correct triggering of the jobs by the logic coded into the `gitlab-ci.yml` file.
If a job gets triggered by mistake it will return a failure, because the `grep` commands will not find any matching file. In principle there is no need for such a triggered job to return failure.
This can be solved by using
```
set -e
grep ...
set +e
```
#11Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/13Test CI with multiple simultaneous commits2019-07-03T15:14:28+02:00Spyros ArgyropoulosTest CI with multiple simultaneous commitsNeed to add at least two DSID directories simultaneously and
* [x] check that the jobs run, going through all the new DSID directories
* [x] check with steering scriptNeed to add at least two DSID directories simultaneously and
* [x] check that the jobs run, going through all the new DSID directories
* [x] check with steering scriptSpyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/11Wrong job triggering2019-07-03T15:14:26+02:00Spyros ArgyropoulosWrong job triggeringhttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/commit/e4d5928c74e5d3f0ca6c00e6304b6845844b07de
Adding a non jO file triggered several jO-related jobs.
Need to investigate and fix.
* [x] Check if it is fixed by c2bb3e51770d5780e...https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/commit/e4d5928c74e5d3f0ca6c00e6304b6845844b07de
Adding a non jO file triggered several jO-related jobs.
Need to investigate and fix.
* [x] Check if it is fixed by c2bb3e51770d5780ead48ad82fc0c295d34f9afe
This is now fixed as discussed in #9 and also in #14 (will close this automatically through the MR)Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/10Fix history2019-06-13T17:53:57+02:00Spyros ArgyropoulosFix historyCurrently for every MR we get all the history of the repositor.
See below where for a simple MR with 5 commits there are 277 commits that show up.
This is very problematic since quickly merging will becomes extremely painful and there ...Currently for every MR we get all the history of the repositor.
See below where for a simple MR with 5 commits there are 277 commits that show up.
This is very problematic since quickly merging will becomes extremely painful and there are already strange artifacts, e.g. the fact that every MR tries to close a JIRA ticket which was mentioned in a commit message 200 commits ago...
Perhaps this needs a rebasing or so - need to understand how to fix it.
![Screenshot_2019-06-08_at_17.07.00](/uploads/8846fb2fce4d039e28bab2076ef860cd/Screenshot_2019-06-08_at_17.07.00.png)Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/8Develop CI checks for jO that require new base fragments2019-07-01T13:17:58+02:00Spyros ArgyropoulosDevelop CI checks for jO that require new base fragmentsAn example can be found in `421xxx/421002`.
Perhaps for the moment we don't need anything more than the `run_athena` job implemented in the `run_athena` branch, but this will not catch the problematic case described below.
Someone has ...An example can be found in `421xxx/421002`.
Perhaps for the moment we don't need anything more than the `run_athena` job implemented in the `run_athena` branch, but this will not catch the problematic case described below.
Someone has 421002 locally where they override the common file with their local one and then they forget to push it to the repo.
The athena job in this case still runs (see attachment below) but probably produces different results from what the local job produced. This is impossible to catch with the current checks.
Maybe one way would be to run athena with exactly the same settings (random numbers etc, extracted from log.generate) and then perhaps write a basic rivet routine (that the uploader should also provide for their local run). Maybe it's a bit too much but we should see how to avoid the above problem.
* [x] Check 421002 with new automatic script
* [x] Check what happens when 421002 is added without the `Sherpa_i` directory
=> Athena runs with no error
* [ ] Develop logic to avoid the situation reported here
[log.generate_without_Sherpa_i](/uploads/2972af7f71bb85defeb491437cdae3f6/log.generate_without_Sherpa_i)[log.generate_with_Sherpa_i](/uploads/fbc9fd01b84211061628ecf2a972d07f/log.generate_with_Sherpa_i)
@fsiegert can you tell from the log.generate files above which one was using `Sherpa_i/2.2.6_BaseFragment.py`?
And how are we supposed to know if a user should or shouldn't upload an additional directory in a DSID directory?https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/225logParser failed due to missing platform info2024-02-01T14:46:57+01:00Yang LiulogParser failed due to missing platform infoHi @sargyrop , as we discussed in [Fixing automatic determination of release for CI runs (!2861) · Merge requests · atlas-physics / pmg / MC Job Options · GitLab (cern.ch)](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_re...Hi @sargyrop , as we discussed in [Fixing automatic determination of release for CI runs (!2861) · Merge requests · atlas-physics / pmg / MC Job Options · GitLab (cern.ch)](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/2861). It seems the added line to extract the platform info will cause problem for some of the jobs.
[Here](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/pipelines/6821117) is one example.
Many thanks for your time to help.
Cheers
Yanghttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/2185360 GeV shoudl be an allowed energy in the CI2023-10-20T11:15:28+02:00Jan Kretzschmar5360 GeV shoudl be an allowed energy in the CIhttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/2669
Run 3 HI and reference run energy will be 5360 GeV, so the CI should not fail with
"ERROR: unknown ecmEnergy: 5360.0"
Full log file also on lxplus in ~jkretz/pu...https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/2669
Run 3 HI and reference run energy will be 5360 GeV, so the CI should not fail with
"ERROR: unknown ecmEnergy: 5360.0"
Full log file also on lxplus in ~jkretz/public/log.generateSpyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/214Assigning same DSID twice2023-09-15T19:06:30+02:00Yiming AbulaitiAssigning same DSID twiceWhen I try to generate new DSID, the same DSID assigned twice "524xxx/524547"
Bellow is the output from "./scripts/commit_new_dsid -m "Commit message" --dry-run ../100xxx/*"
Will move ../100xxx/100003 to 524xxx/524546.
...When I try to generate new DSID, the same DSID assigned twice "524xxx/524547"
Bellow is the output from "./scripts/commit_new_dsid -m "Commit message" --dry-run ../100xxx/*"
Will move ../100xxx/100003 to 524xxx/524546.
Will move ../100xxx/100005 to 524xxx/524547.
Will move ../100xxx/100000 to 524xxx/524546.
Will move ../100xxx/100001 to 524xxx/524547.
Will move ../100xxx/100007 to 602xxx/602485.
Will move ../100xxx/100008 to 602xxx/602486.
Cheers,
AbletSpyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/209commit_new_dsid: Symbolic link is not updated2023-05-20T16:26:07+02:00Yiming Abulaiticommit_new_dsid: Symbolic link is not updatedHi,
When I use "commit_new_dsid" to copy JO directory to GIT. Symbolic links are not updated with newly assigned DSIDS.
I used this command to move to directory to GIT area:
```
./scripts/commit_new_dsid -m "Commit message" -n ../100xx...Hi,
When I use "commit_new_dsid" to copy JO directory to GIT. Symbolic links are not updated with newly assigned DSIDS.
I used this command to move to directory to GIT area:
```
./scripts/commit_new_dsid -m "Commit message" -n ../100xxx/*
OK: ../100xxx/100000 is dir
OK: ../100xxx/100001 is dir
```
Then I checked the new directory(DSIDs).
```
[yabulait@lxplus750 mcjoboptions]$ ls -l 523xxx/523102/
lrwxrwxrwx. 0 yabulait zp 41 May 18 15:53 DirectPhotonFilter.py -> ../../100xxx/100000/DirectPhotonFilter.py
-rw-r--r--. 1 yabulait zp 163 May 18 15:53 log.generate.short
lrwxrwxrwx. 0 yabulait zp 66 May 18 15:53 MadGraphControl_MGPy8EG_DMS1_dijetgamma_pta.py -> ../../100xxx/100000/MadGraphControl_MGPy8EG_DMS1_dijetgamma_pta.py
```
Here you can see that the symbolic links are still "../../100xxx/100000/*", but it should be "../../523xxx/523101".
This happened few times recently. So I reported here.
Cheers,
AbletSpyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/197check_jo_consistency failed2022-12-19T16:20:55+01:00Yiming Abulaiticheck_jo_consistency failedHi,
When I commit a second time after fix something in Control file, the consistency check failed.
The consistency is checking some other files that is not a part of my commits (I am trying to register 518405-518446 range).
But job is fa...Hi,
When I commit a second time after fix something in Control file, the consistency check failed.
The consistency is checking some other files that is not a part of my commits (I am trying to register 518405-518446 range).
But job is failed due to some errors related to 421xxx/421100/..
So How can I fix this?
Cheers,
Ablet
Error part:
OK: No generator full name is found
Generators used: ['Py8', 'EG']
ERROR: file /builds/atlas-physics/pmg/mcjoboptions/scripts/../421xxx/421100/mc.Py8EG_A14NNPDF23LO_Ztautau.py contains includes pointing to MC15JobOptions
Failed Job
https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/26543765Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/189Job Failed #249610542022-10-05T09:10:18+02:00Yang LiuJob Failed #24961054Job [#24961054](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/24961054) failed for 492434176f479e1ec018b7891f3b211cd8387eb0:
Hi @sargyrop, it seems sometimes ago, when uploading the squark version of the 2step decay via s...Job [#24961054](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/24961054) failed for 492434176f479e1ec018b7891f3b211cd8387eb0:
Hi @sargyrop, it seems sometimes ago, when uploading the squark version of the 2step decay via sleptons the corresponding person skip the checks of the modifies causing the CO is duplicated in 505622 and 505945.
While this time, I need to upload the CO and JO files for the extended points of the GG_N2_2LN1 which have the same CO as in the 505622 and 505945.
The ideal way to do this is to fix the duplication existing before by redirecting the CO in 505945 to the one in 505622 as this commit did.
But as I can see, changing the existing file need some extra approvement.
So I'm contacting u here to see what I should do to make this happen.
Many thanks
Cheers
YangSpyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/176Job Failed #220519692022-05-26T00:34:53+02:00Jan KretzschmarJob Failed #22051969Job [#22051969](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/22051969) failed for 6964c28821e9ec0e33f4548a73191e02293e6c5b:
Hi @sargyrop , this pipeline appears to throw a couple of warnings and errors eventually. This i...Job [#22051969](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/22051969) failed for 6964c28821e9ec0e33f4548a73191e02293e6c5b:
Hi @sargyrop , this pipeline appears to throw a couple of warnings and errors eventually. This is using a pretty old SL6 release, so maybe it cannot be fixed...? Maybe you can just have a quick look and let us know.
thanks, JanSpyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/174Arithmetic expression error in check_logParser.sh2022-05-17T16:23:22+02:00Spyros ArgyropoulosArithmetic expression error in check_logParser.shSee https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/21845100See https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/21845100https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/173logParser crash when using gridpacks2022-05-17T10:36:18+02:00Dominic HirschbuehllogParser crash when using gridpacksWhen running the logParser for a log.generate which used a gridpack, it crashes with:
"/afs/cern.ch/work/d/dhirsch/stop13TeV/mcjoboptions/scripts/logParser.py", line 368, in powhegChecks
if not glob.glob(f"{os.path.dirname(logFile)}/...When running the logParser for a log.generate which used a gridpack, it crashes with:
"/afs/cern.ch/work/d/dhirsch/stop13TeV/mcjoboptions/scripts/logParser.py", line 368, in powhegChecks
if not glob.glob(f"{os.path.dirname(logFile)}/mc_*TeV.*.GRID.tar.gz"):
File "/cvmfs/atlas.cern.ch/repo/sw/software/22.6/sw/lcg/releases/LCG_101_ATLAS_18/Python/3.9.6/x86_64-centos7-gcc11-opt/lib/python3.9/posixpath.py", line 152, in dirname
p = os.fspath(p)
TypeError: expected str, bytes or os.PathLike object, not list
The problem is that the function
powhegChecks(logFile)
expects a filename, but gets the file content.
A similar problem should happen for Madgraph, since the function is defined properly
with def madgraphChecks(logContent), but the variable logFile used to check the gridpack
is nowwhere defined.
Cheers
DominicSpyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/172EVNTtoEVNT and ECM: fails to download input EVNT in CI2022-05-17T16:03:17+02:00Matthew GignacEVNTtoEVNT and ECM: fails to download input EVNT in CIHi,
A few of the CI jobs fail for some E2E jOs in mc21: https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/21697381
Looks like the $ECMENERGY environment variable is not correctly set. In this type of transform, the ECM is no...Hi,
A few of the CI jobs fail for some E2E jOs in mc21: https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/21697381
Looks like the $ECMENERGY environment variable is not correctly set. In this type of transform, the ECM is not required and not written into the log, which is probably the underlying issue. Any ideas on how to detect this?
As an aside, I noticed that the log.generate.short is claiming ecmEnergy of 13000 GeV, though the transform was run with 13600 TeV (not that it matters...). I guess the 13000 is a default somewhere, which maybe is causing some issues?
Cheers,
MatthewSpyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/171CI download to check for grid input LHEs with mc21_13p6TeV scope2022-05-10T07:28:54+02:00Jan KretzschmarCI download to check for grid input LHEs with mc21_13p6TeV scopeIn https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/1824 I hit the problem, that Grid input LHEs are always trying to be downloaded with mc15_13TeV or mc16_13TeV scope, I think the origin of this is here:
https://gi...In https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/1824 I hit the problem, that Grid input LHEs are always trying to be downloaded with mc15_13TeV or mc16_13TeV scope, I think the origin of this is here:
https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/blob/master/scripts/run_athena.sh#L115-138
First, it would probably be preferable, if we get the CME string from the CME value rather than fixing it to 13 TeV.
Second, we now also need to allow mc21
As shortest patch we could just keep extending the list of allowed scopes, here mc21_13p6TeV being the obvious one to add immediately.
Many thanks, Janhttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/170New commit script failing for directories with arbitrary names2022-05-06T12:59:13+02:00Spyros ArgyropoulosNew commit script failing for directories with arbitrary names![Screenshot_2022-05-06_at_11.16.03](/uploads/a87f3ca9c98c1d19745c55cc15380114/Screenshot_2022-05-06_at_11.16.03.png)
This is because of:
```
newDSID += parseDSIDList(args.DSID)
```
seems to be adding an empty item in the list![Screenshot_2022-05-06_at_11.16.03](/uploads/a87f3ca9c98c1d19745c55cc15380114/Screenshot_2022-05-06_at_11.16.03.png)
This is because of:
```
newDSID += parseDSIDList(args.DSID)
```
seems to be adding an empty item in the listChristian GutschowChristian Gutschowhttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/168logParser unsupported locale setting2022-04-21T20:53:54+02:00Spyros ArgyropouloslogParser unsupported locale settingFull error message:
INFO: New DSID directory: 100xxx/100001 ...
OK: log.generate file found.
Traceback (most recent call last):
File "scripts/logParser.py", line 8, in <module>
locale.setlocale(locale.LC_CTYPE, f'{lang}.UTF-8'...Full error message:
INFO: New DSID directory: 100xxx/100001 ...
OK: log.generate file found.
Traceback (most recent call last):
File "scripts/logParser.py", line 8, in <module>
locale.setlocale(locale.LC_CTYPE, f'{lang}.UTF-8')
File "/usr/lib64/python3.6/locale.py", line 598, in setlocale
return _setlocale(category, locale)
locale.Error: unsupported locale setting
ERROR: logParser run failed.
Need output of
- locale
- locale -a
- env
- which machine you are running on
@yanlin @nishuSpyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/165Decoding issue in commit script / logParser ?2022-04-10T17:20:38+02:00Spyros ArgyropoulosDecoding issue in commit script / logParser ?![Screenshot_2022-04-10_at_11.44.33](/uploads/8e97d70a5350e335d2005d379cc8d393/Screenshot_2022-04-10_at_11.44.33.png)
appears with python 3.6.8 (same as on lxplus) - content at /afs/cern.ch/user/a/aivina/public/JO_TEST_DIR
Not reproduc...![Screenshot_2022-04-10_at_11.44.33](/uploads/8e97d70a5350e335d2005d379cc8d393/Screenshot_2022-04-10_at_11.44.33.png)
appears with python 3.6.8 (same as on lxplus) - content at /afs/cern.ch/user/a/aivina/public/JO_TEST_DIR
Not reproducible on lxplus or locally
@aivinaSpyros ArgyropoulosSpyros Argyropoulos