MC Job Options issueshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues2022-04-21T20:53:54+02:00https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/168logParser unsupported locale setting2022-04-21T20:53:54+02:00Spyros ArgyropouloslogParser unsupported locale settingFull error message:
INFO: New DSID directory: 100xxx/100001 ...
OK: log.generate file found.
Traceback (most recent call last):
File "scripts/logParser.py", line 8, in <module>
locale.setlocale(locale.LC_CTYPE, f'{lang}.UTF-8'...Full error message:
INFO: New DSID directory: 100xxx/100001 ...
OK: log.generate file found.
Traceback (most recent call last):
File "scripts/logParser.py", line 8, in <module>
locale.setlocale(locale.LC_CTYPE, f'{lang}.UTF-8')
File "/usr/lib64/python3.6/locale.py", line 598, in setlocale
return _setlocale(category, locale)
locale.Error: unsupported locale setting
ERROR: logParser run failed.
Need output of
- locale
- locale -a
- env
- which machine you are running on
@yanlin @nishuSpyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/167Fix logParser bug when running in CI2022-04-20T22:28:20+02:00Spyros ArgyropoulosFix logParser bug when running in CI![Screenshot_2022-04-20_at_21.08.27](/uploads/ed6760a1cbe977212c0904faf484fd7a/Screenshot_2022-04-20_at_21.08.27.png)![Screenshot_2022-04-20_at_21.08.27](/uploads/ed6760a1cbe977212c0904faf484fd7a/Screenshot_2022-04-20_at_21.08.27.png)Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/166Add checks for python2/3 compatibility of jO2022-04-21T17:21:24+02:00Spyros ArgyropoulosAdd checks for python2/3 compatibility of jOan example DSID is 830099: https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/blob/master/830xxx/830099/mc.H7EG_jetjet_72_Cluster_JZ1.py
R21: 21.6.85
R22: You can try 22.6.13 (later releases have issues with EvtGen_i — should be ...an example DSID is 830099: https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/blob/master/830xxx/830099/mc.H7EG_jetjet_72_Cluster_JZ1.py
R21: 21.6.85
R22: You can try 22.6.13 (later releases have issues with EvtGen_i — should be fixed soon).Spyros ArgyropoulosSpyros Argyropoulos2022-04-25https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/165Decoding issue in commit script / logParser ?2022-04-10T17:20:38+02:00Spyros ArgyropoulosDecoding issue in commit script / logParser ?![Screenshot_2022-04-10_at_11.44.33](/uploads/8e97d70a5350e335d2005d379cc8d393/Screenshot_2022-04-10_at_11.44.33.png)
appears with python 3.6.8 (same as on lxplus) - content at /afs/cern.ch/user/a/aivina/public/JO_TEST_DIR
Not reproduc...![Screenshot_2022-04-10_at_11.44.33](/uploads/8e97d70a5350e335d2005d379cc8d393/Screenshot_2022-04-10_at_11.44.33.png)
appears with python 3.6.8 (same as on lxplus) - content at /afs/cern.ch/user/a/aivina/public/JO_TEST_DIR
Not reproducible on lxplus or locally
@aivinaSpyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/164Some improvements for commit script2022-04-10T13:43:26+02:00Spyros ArgyropoulosSome improvements for commit script- The removal of the `.sh` extension means that 1) the OS thinks it's an executable and will try to run it instead of opening it with an editor 2) the editor doesn't recognise that it's python so there is no highlighting etc. Generally I...- The removal of the `.sh` extension means that 1) the OS thinks it's an executable and will try to run it instead of opening it with an editor 2) the editor doesn't recognise that it's python so there is no highlighting etc. Generally I see no reason for having removed the extension and I think we should put it back (also same for other scripts where extensions have been removed)
- [L164](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/blob/master/scripts/commit_new_dsid#L164) seems to be a debug statement which was forgotten to be removed, I don't see the reason for having a long dump of dsids that are going to be processed and this is more clearly visible from the following printout statements which are clearer
![Screenshot_2022-04-10_at_11.12.20](/uploads/eeabafaa609b285bd55e375886126cd8/Screenshot_2022-04-10_at_11.12.20.png)Christian GutschowChristian Gutschowhttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/163ecmEnergy in logParser fixed to 13000 GeV?2022-04-06T11:25:01+02:00Jan KretzschmarecmEnergy in logParser fixed to 13000 GeV?I have the impression that logParser fixes the ecmEnergy to 13000 GeV in the log.generate.short irrespective what the given log file gives
https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/blob/master/scripts/logParser.py#L471
Thi...I have the impression that logParser fixes the ecmEnergy to 13000 GeV in the log.generate.short irrespective what the given log file gives
https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/blob/master/scripts/logParser.py#L471
This impacted https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/1748
(I can fix that MR myself now)https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/162Job Failed #208263692022-04-04T13:40:29+02:00Christian GutschowJob Failed #20826369Job [#20826369](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/20826369) failed for 06fa2b7f0bbd6b1dd30f9dd4e073b3b2206ff634:Job [#20826369](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/20826369) failed for 06fa2b7f0bbd6b1dd30f9dd4e073b3b2206ff634:https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/161logParser expects randomSeed arg even for E2E jobs2022-03-25T14:51:01+01:00Jan KretzschmarlogParser expects randomSeed arg even for E2E jobsHi,
in https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/1725 I think I 'discovered' an issue that the logParser requires the argument 'randomSeed' to be present also for EVNT-to-EVNT (afterburner filtering) jobs. T...Hi,
in https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/1725 I think I 'discovered' an issue that the logParser requires the argument 'randomSeed' to be present also for EVNT-to-EVNT (afterburner filtering) jobs. The problem is: even if you give the argument, it is not printed to the log file. Not sure if this is an issue of the transform, but in any case these jobs are not really supposed to need ranfdom numbers in most case.
Thanks, JanSpyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/160check_unique_controlFile too loose2022-04-10T13:51:04+02:00Spyros Argyropouloscheck_unique_controlFile too looseI notice in !1694 that due to mistakes in previous MRs there are spurious errors, check https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/20185882
This is because https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/blob/...I notice in !1694 that due to mistakes in previous MRs there are spurious errors, check https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/20185882
This is because https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/blob/master/507xxx/507035/MadGraphControl_Py8EG_2HDMa_monoH_common.py is a file rather than a link.
This means that all the future MRs that will point to this control file (and it's almost certain we'll have more) will all have the same spurious errors.
In fact this is going to happen for all files that should be links and weren't so the situation is going to become worse with time. I would therefore propose to make the test stricter, i.e. we should not allow two physical files with the same name and the same content. If the content is different then the name should be adjusted. And we should not allow the CI job to fail.
Is there any objection @jkretz @mgignac @cgutscho ?Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/159Allow run_athena to run 13p6 TeV2022-05-10T11:04:58+02:00Spyros ArgyropoulosAllow run_athena to run 13p6 TeVCurrently only `mc15_13TeV` and `mc16_13TeV` external LHE files are handled in run_athena. Should be extended to `mc16_13p6TeV` and `mc21_13p6TeV`.
Check if we can automate this so that we don't need to hard-code things as done now.Currently only `mc15_13TeV` and `mc16_13TeV` external LHE files are handled in run_athena. Should be extended to `mc16_13p6TeV` and `mc21_13p6TeV`.
Check if we can automate this so that we don't need to hard-code things as done now.Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/158Handle purple list in logParser2022-05-05T16:15:51+02:00Spyros ArgyropoulosHandle purple list in logParserTBD if this is something that we can catch and report e.g. in the parser
See !1673TBD if this is something that we can catch and report e.g. in the parser
See !1673Spyros ArgyropoulosSpyros Argyropoulos2022-04-01https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/157logParser.py didn't grep inputGeneratorFile2022-03-02T09:41:23+01:00Yiming AbulaitilogParser.py didn't grep inputGeneratorFileHi
---- This is for LHE files as inputs -----
Bellow is the content of log.generate.short, this log file is generated with 21.6.75
---------
- estimated CPU for CI job = 0.00 hrs
- using release = AthGeneration-21.6.75
- ecmEnergy = ...Hi
---- This is for LHE files as inputs -----
Bellow is the content of log.generate.short, this log file is generated with 21.6.75
---------
- estimated CPU for CI job = 0.00 hrs
- using release = AthGeneration-21.6.75
- ecmEnergy = 13000.0
- inputGeneratorFile = 09:20:14 Py:Gen_tf INFO inputGeneratorFile = TXT.440365._000001.tar.gz
- randomSeed = 1234
- EVNT to EVNT = False
- LHEonly = False
---------------
The inputGeneratorFile field is messed up here.
But the logParser.py works fine with old releases like 21.6.56. The reason is that athena print out changed in new release (this is what I observed):
Print out from 21.6.56:
--> 16:06:18 Py:Gen_tf INFO inputGeneratorFile used TXT.440329._000001.tar.gz
Print out from 21.6.75:
--> 09:09:41 Py:Gen_tf INFO inputGeneratorFile = TXT.440363._000001.tar.gz
You can see the changed from "used" to "=".
The line [L159](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/blob/master/scripts/logParser.py#L159) has to check both "used" and "=" to accommodate changes made in athena.
Cheers,
AbletSpyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/156commit_new_dsid.sh replace correct symbolic links2022-02-20T19:39:51+01:00Yiming Abulaiticommit_new_dsid.sh replace correct symbolic linksDear,
The commit_new_dsid.sh script replaces the DSID path in the symbolic event if the symbolic link is correct.
for example, my symbolic link is "../../510xxx/510250/file". When the script copy 100xx/* to DSID directory the "../../510...Dear,
The commit_new_dsid.sh script replaces the DSID path in the symbolic event if the symbolic link is correct.
for example, my symbolic link is "../../510xxx/510250/file". When the script copy 100xx/* to DSID directory the "../../510xxx/510250/" is replaced by the first new DSID (for example "../../511xxx/511424/").
Is it possible to let commit_new_dsid.sh keep the symbolic links when it is in 500xx-999xx range?
Cheers,
AbletSpyros ArgyropoulosSpyros Argyropoulos2022-02-20https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/155MR not being linked on JIRAs2021-12-03T14:57:38+01:00Matthew GignacMR not being linked on JIRAsIn some recent requests, it was noticed that the MRs are not being linked on JIRA. For example see: https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/1510In some recent requests, it was noticed that the MRs are not being linked on JIRA. For example see: https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/1510Spyros ArgyropoulosSpyros Argyropoulos2021-12-05https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/154CI cannot deal with integration grids for several ECM being present2021-11-16T21:10:54+01:00Jan KretzschmarCI cannot deal with integration grids for several ECM being presentI'm trying to commit new jO that come with integration grids. https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/1485
As we want to run it at different sqrt(s) value, we need several of those. the Gen_tf.py transform ...I'm trying to commit new jO that come with integration grids. https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/1485
As we want to run it at different sqrt(s) value, we need several of those. the Gen_tf.py transform works this out correctly.
However, the CI bails on trying to copy the GRID file, because it expects just a single file https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/blob/master/scripts/run_athena.sh#L136
Probably the best solution would be to copy the right file for the srt(s) value in question that would be sth like `mc_${sqrts}TeV.*.GRID.tar.gz` instead of `*.GRID.tar.gz`.
And ${sqrts} should be either an integer like 5,7,8,13,14 or 8p16 or 13p6 for special values.Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/153Do not throw warning from dummy CI job if a jO has not been committed2021-12-08T11:47:38+01:00Spyros ArgyropoulosDo not throw warning from dummy CI job if a jO has not been committedSpyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/152Skip the check for 10% extra LHE events in case of LHE-only jobs2021-11-10T11:51:30+01:00Jan KretzschmarSkip the check for 10% extra LHE events in case of LHE-only jobsHi,
In preparing some LHE-only jobs with MG (e.g. https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/1462) I hit the issue that this check https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/blob/master/scripts/lo...Hi,
In preparing some LHE-only jobs with MG (e.g. https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/1462) I hit the issue that this check https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/blob/master/scripts/logParser.py#L307-316 demands "10% more" LHE events. Obviously, in an LHE-only job this can never be passed, as the number of events requested is the number of events in the LHE file. Can this check be disabled? I guess the condition would be sth like "--outputTXTFile" present and "--outputEVNTFile" absent.
Looking at this again, I wonder if skipping this check in case of externally supplied LHE files actually makes sense as stated in the comment `# This check only makes sense if no external LHE inputs are used` - you'd normally want this to be checked also in this case?
Thanks, Jan
PS: for the above MR I circumvented the issue by hacking logParser locally to remove these lines(!)Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/151Support for multiple tarballs with different COM energy2021-10-12T15:31:16+02:00Christian GutschowSupport for multiple tarballs with different COM energyThis should not have crashed:
Job [#16844556](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/16844556) failed for 9efbabb68119e3a4a2bf19113c4d8f3ffeb2b9e9:
It used to be working, so not sure what's changed?This should not have crashed:
Job [#16844556](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/16844556) failed for 9efbabb68119e3a4a2bf19113c4d8f3ffeb2b9e9:
It used to be working, so not sure what's changed?Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/150Make CI job that sends email to conveners2021-09-30T13:43:46+02:00Spyros ArgyropoulosMake CI job that sends email to conveners- when commit message contains [skip modfiles]
- also when files are actually modified ? (we probably want this as well - some people add skip modfiles when there's no reason to)- when commit message contains [skip modfiles]
- also when files are actually modified ? (we probably want this as well - some people add skip modfiles when there's no reason to)Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/149NLHE check not working well in CI2022-03-12T09:33:06+01:00Christian GutschowNLHE check not working well in CIAs seen in !1304.
Maybe this check could be based on the extrapolated number of events?
> For a sufficiently large `nEventsPerJob` the 10% is more conservative than 4 sigma, so no harm done, albeit a bit more inefficient. For low `nEve...As seen in !1304.
Maybe this check could be based on the extrapolated number of events?
> For a sufficiently large `nEventsPerJob` the 10% is more conservative than 4 sigma, so no harm done, albeit a bit more inefficient. For low `nEventsPerJob` 4sigma is more conservative and people would probably anyway notice this in their local runs and choose a larger margin (e.g. factors of 25 rather than 1.1). So arguably the 4sigma thing is probably not needed in practice, provided people actually test their setups locally...
Maybe @jkretz should indicate what his preference is for the long run, but at the moment this new rule is causing the CI to crash frequently for no good reason, so we need to patch somehow.
Another option is to remove the 4sigma rule and go back to the previous check with 10% extra events, or that we have the 4sigma rule only for jobs that have a small number of requested output events (although it's actually impossible to know if people just test with a low number of events)