MC Job Options issueshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues2022-05-10T07:28:54+02:00https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/171CI download to check for grid input LHEs with mc21_13p6TeV scope2022-05-10T07:28:54+02:00Jan KretzschmarCI download to check for grid input LHEs with mc21_13p6TeV scopeIn https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/1824 I hit the problem, that Grid input LHEs are always trying to be downloaded with mc15_13TeV or mc16_13TeV scope, I think the origin of this is here:
https://gi...In https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/1824 I hit the problem, that Grid input LHEs are always trying to be downloaded with mc15_13TeV or mc16_13TeV scope, I think the origin of this is here:
https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/blob/master/scripts/run_athena.sh#L115-138
First, it would probably be preferable, if we get the CME string from the CME value rather than fixing it to 13 TeV.
Second, we now also need to allow mc21
As shortest patch we could just keep extending the list of allowed scopes, here mc21_13p6TeV being the obvious one to add immediately.
Many thanks, Janhttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/170New commit script failing for directories with arbitrary names2022-05-06T12:59:13+02:00Spyros ArgyropoulosNew commit script failing for directories with arbitrary names![Screenshot_2022-05-06_at_11.16.03](/uploads/a87f3ca9c98c1d19745c55cc15380114/Screenshot_2022-05-06_at_11.16.03.png)
This is because of:
```
newDSID += parseDSIDList(args.DSID)
```
seems to be adding an empty item in the list![Screenshot_2022-05-06_at_11.16.03](/uploads/a87f3ca9c98c1d19745c55cc15380114/Screenshot_2022-05-06_at_11.16.03.png)
This is because of:
```
newDSID += parseDSIDList(args.DSID)
```
seems to be adding an empty item in the listChristian GutschowChristian Gutschowhttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/158Handle purple list in logParser2022-05-05T16:15:51+02:00Spyros ArgyropoulosHandle purple list in logParserTBD if this is something that we can catch and report e.g. in the parser
See !1673TBD if this is something that we can catch and report e.g. in the parser
See !1673Spyros ArgyropoulosSpyros Argyropoulos2022-04-01https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/168logParser unsupported locale setting2022-04-21T20:53:54+02:00Spyros ArgyropouloslogParser unsupported locale settingFull error message:
INFO: New DSID directory: 100xxx/100001 ...
OK: log.generate file found.
Traceback (most recent call last):
File "scripts/logParser.py", line 8, in <module>
locale.setlocale(locale.LC_CTYPE, f'{lang}.UTF-8'...Full error message:
INFO: New DSID directory: 100xxx/100001 ...
OK: log.generate file found.
Traceback (most recent call last):
File "scripts/logParser.py", line 8, in <module>
locale.setlocale(locale.LC_CTYPE, f'{lang}.UTF-8')
File "/usr/lib64/python3.6/locale.py", line 598, in setlocale
return _setlocale(category, locale)
locale.Error: unsupported locale setting
ERROR: logParser run failed.
Need output of
- locale
- locale -a
- env
- which machine you are running on
@yanlin @nishuSpyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/166Add checks for python2/3 compatibility of jO2022-04-21T17:21:24+02:00Spyros ArgyropoulosAdd checks for python2/3 compatibility of jOan example DSID is 830099: https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/blob/master/830xxx/830099/mc.H7EG_jetjet_72_Cluster_JZ1.py
R21: 21.6.85
R22: You can try 22.6.13 (later releases have issues with EvtGen_i — should be ...an example DSID is 830099: https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/blob/master/830xxx/830099/mc.H7EG_jetjet_72_Cluster_JZ1.py
R21: 21.6.85
R22: You can try 22.6.13 (later releases have issues with EvtGen_i — should be fixed soon).Spyros ArgyropoulosSpyros Argyropoulos2022-04-25https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/167Fix logParser bug when running in CI2022-04-20T22:28:20+02:00Spyros ArgyropoulosFix logParser bug when running in CI![Screenshot_2022-04-20_at_21.08.27](/uploads/ed6760a1cbe977212c0904faf484fd7a/Screenshot_2022-04-20_at_21.08.27.png)![Screenshot_2022-04-20_at_21.08.27](/uploads/ed6760a1cbe977212c0904faf484fd7a/Screenshot_2022-04-20_at_21.08.27.png)Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/165Decoding issue in commit script / logParser ?2022-04-10T17:20:38+02:00Spyros ArgyropoulosDecoding issue in commit script / logParser ?![Screenshot_2022-04-10_at_11.44.33](/uploads/8e97d70a5350e335d2005d379cc8d393/Screenshot_2022-04-10_at_11.44.33.png)
appears with python 3.6.8 (same as on lxplus) - content at /afs/cern.ch/user/a/aivina/public/JO_TEST_DIR
Not reproduc...![Screenshot_2022-04-10_at_11.44.33](/uploads/8e97d70a5350e335d2005d379cc8d393/Screenshot_2022-04-10_at_11.44.33.png)
appears with python 3.6.8 (same as on lxplus) - content at /afs/cern.ch/user/a/aivina/public/JO_TEST_DIR
Not reproducible on lxplus or locally
@aivinaSpyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/160check_unique_controlFile too loose2022-04-10T13:51:04+02:00Spyros Argyropouloscheck_unique_controlFile too looseI notice in !1694 that due to mistakes in previous MRs there are spurious errors, check https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/20185882
This is because https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/blob/...I notice in !1694 that due to mistakes in previous MRs there are spurious errors, check https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/20185882
This is because https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/blob/master/507xxx/507035/MadGraphControl_Py8EG_2HDMa_monoH_common.py is a file rather than a link.
This means that all the future MRs that will point to this control file (and it's almost certain we'll have more) will all have the same spurious errors.
In fact this is going to happen for all files that should be links and weren't so the situation is going to become worse with time. I would therefore propose to make the test stricter, i.e. we should not allow two physical files with the same name and the same content. If the content is different then the name should be adjusted. And we should not allow the CI job to fail.
Is there any objection @jkretz @mgignac @cgutscho ?Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/164Some improvements for commit script2022-04-10T13:43:26+02:00Spyros ArgyropoulosSome improvements for commit script- The removal of the `.sh` extension means that 1) the OS thinks it's an executable and will try to run it instead of opening it with an editor 2) the editor doesn't recognise that it's python so there is no highlighting etc. Generally I...- The removal of the `.sh` extension means that 1) the OS thinks it's an executable and will try to run it instead of opening it with an editor 2) the editor doesn't recognise that it's python so there is no highlighting etc. Generally I see no reason for having removed the extension and I think we should put it back (also same for other scripts where extensions have been removed)
- [L164](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/blob/master/scripts/commit_new_dsid#L164) seems to be a debug statement which was forgotten to be removed, I don't see the reason for having a long dump of dsids that are going to be processed and this is more clearly visible from the following printout statements which are clearer
![Screenshot_2022-04-10_at_11.12.20](/uploads/eeabafaa609b285bd55e375886126cd8/Screenshot_2022-04-10_at_11.12.20.png)Christian GutschowChristian Gutschowhttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/163ecmEnergy in logParser fixed to 13000 GeV?2022-04-06T11:25:01+02:00Jan KretzschmarecmEnergy in logParser fixed to 13000 GeV?I have the impression that logParser fixes the ecmEnergy to 13000 GeV in the log.generate.short irrespective what the given log file gives
https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/blob/master/scripts/logParser.py#L471
Thi...I have the impression that logParser fixes the ecmEnergy to 13000 GeV in the log.generate.short irrespective what the given log file gives
https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/blob/master/scripts/logParser.py#L471
This impacted https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/1748
(I can fix that MR myself now)https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/162Job Failed #208263692022-04-04T13:40:29+02:00Christian GutschowJob Failed #20826369Job [#20826369](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/20826369) failed for 06fa2b7f0bbd6b1dd30f9dd4e073b3b2206ff634:Job [#20826369](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/20826369) failed for 06fa2b7f0bbd6b1dd30f9dd4e073b3b2206ff634:https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/147Support for 7-digit DSIDs2022-04-01T15:44:37+02:00Christian GutschowSupport for 7-digit DSIDsDraft proposal discussed in [GIT meeting](https://indico.cern.ch/event/1061701/#23-support-for-7-digit-dsids-i):
* Stick to current repo, look into sparse checkouts in case response time becomes unsatisfactory
* keep 6-digit structure ...Draft proposal discussed in [GIT meeting](https://indico.cern.ch/event/1061701/#23-support-for-7-digit-dsids-i):
* Stick to current repo, look into sparse checkouts in case response time becomes unsatisfactory
* keep 6-digit structure as is, group 7-digits in millions, then in thousands
* Retain grouping by generator (?) for 7-digits (`15xxxxx` Madgraph, `16xxxxx` Powheg etc.)
* can use `10xxxxx` - `14xxxxx` for large BSM scans for instance
* keep using 6-digits for now, but introduce simple mechanism to disable 6-digit booking altogether at some pointChristian GutschowChristian Gutschowhttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/161logParser expects randomSeed arg even for E2E jobs2022-03-25T14:51:01+01:00Jan KretzschmarlogParser expects randomSeed arg even for E2E jobsHi,
in https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/1725 I think I 'discovered' an issue that the logParser requires the argument 'randomSeed' to be present also for EVNT-to-EVNT (afterburner filtering) jobs. T...Hi,
in https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/1725 I think I 'discovered' an issue that the logParser requires the argument 'randomSeed' to be present also for EVNT-to-EVNT (afterburner filtering) jobs. The problem is: even if you give the argument, it is not printed to the log file. Not sure if this is an issue of the transform, but in any case these jobs are not really supposed to need ranfdom numbers in most case.
Thanks, JanSpyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/149NLHE check not working well in CI2022-03-12T09:33:06+01:00Christian GutschowNLHE check not working well in CIAs seen in !1304.
Maybe this check could be based on the extrapolated number of events?
> For a sufficiently large `nEventsPerJob` the 10% is more conservative than 4 sigma, so no harm done, albeit a bit more inefficient. For low `nEve...As seen in !1304.
Maybe this check could be based on the extrapolated number of events?
> For a sufficiently large `nEventsPerJob` the 10% is more conservative than 4 sigma, so no harm done, albeit a bit more inefficient. For low `nEventsPerJob` 4sigma is more conservative and people would probably anyway notice this in their local runs and choose a larger margin (e.g. factors of 25 rather than 1.1). So arguably the 4sigma thing is probably not needed in practice, provided people actually test their setups locally...
Maybe @jkretz should indicate what his preference is for the long run, but at the moment this new rule is causing the CI to crash frequently for no good reason, so we need to patch somehow.
Another option is to remove the 4sigma rule and go back to the previous check with 10% extra events, or that we have the 4sigma rule only for jobs that have a small number of requested output events (although it's actually impossible to know if people just test with a low number of events)https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/63logParser: document warnings and errors2022-03-12T09:24:59+01:00Spyros ArgyropouloslogParser: document warnings and errorsI've been thinking that it would help with the approval process to document somewhere (I'd say in the README - to keep everything together) all the cases where `logParser` throws WARNING or ERROR, so that approvers can easily skim throug...I've been thinking that it would help with the approval process to document somewhere (I'd say in the README - to keep everything together) all the cases where `logParser` throws WARNING or ERROR, so that approvers can easily skim through the CI job output and understand if something should be followed up on.Futurehttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/157logParser.py didn't grep inputGeneratorFile2022-03-02T09:41:23+01:00Yiming AbulaitilogParser.py didn't grep inputGeneratorFileHi
---- This is for LHE files as inputs -----
Bellow is the content of log.generate.short, this log file is generated with 21.6.75
---------
- estimated CPU for CI job = 0.00 hrs
- using release = AthGeneration-21.6.75
- ecmEnergy = ...Hi
---- This is for LHE files as inputs -----
Bellow is the content of log.generate.short, this log file is generated with 21.6.75
---------
- estimated CPU for CI job = 0.00 hrs
- using release = AthGeneration-21.6.75
- ecmEnergy = 13000.0
- inputGeneratorFile = 09:20:14 Py:Gen_tf INFO inputGeneratorFile = TXT.440365._000001.tar.gz
- randomSeed = 1234
- EVNT to EVNT = False
- LHEonly = False
---------------
The inputGeneratorFile field is messed up here.
But the logParser.py works fine with old releases like 21.6.56. The reason is that athena print out changed in new release (this is what I observed):
Print out from 21.6.56:
--> 16:06:18 Py:Gen_tf INFO inputGeneratorFile used TXT.440329._000001.tar.gz
Print out from 21.6.75:
--> 09:09:41 Py:Gen_tf INFO inputGeneratorFile = TXT.440363._000001.tar.gz
You can see the changed from "used" to "=".
The line [L159](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/blob/master/scripts/logParser.py#L159) has to check both "used" and "=" to accommodate changes made in athena.
Cheers,
AbletSpyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/156commit_new_dsid.sh replace correct symbolic links2022-02-20T19:39:51+01:00Yiming Abulaiticommit_new_dsid.sh replace correct symbolic linksDear,
The commit_new_dsid.sh script replaces the DSID path in the symbolic event if the symbolic link is correct.
for example, my symbolic link is "../../510xxx/510250/file". When the script copy 100xx/* to DSID directory the "../../510...Dear,
The commit_new_dsid.sh script replaces the DSID path in the symbolic event if the symbolic link is correct.
for example, my symbolic link is "../../510xxx/510250/file". When the script copy 100xx/* to DSID directory the "../../510xxx/510250/" is replaced by the first new DSID (for example "../../511xxx/511424/").
Is it possible to let commit_new_dsid.sh keep the symbolic links when it is in 500xx-999xx range?
Cheers,
AbletSpyros ArgyropoulosSpyros Argyropoulos2022-02-20https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/69Checks for madgraph mode and integration time2022-02-18T15:51:06+01:00Spyros ArgyropoulosChecks for madgraph mode and integration time* if multicore mode then throw ERROR and print that they should use single core
* if integration time very long then ERROR and print that they should create a gridpack
https://prodtask-dev.cern.ch/prodtask/inputlist_with_request/26924...* if multicore mode then throw ERROR and print that they should use single core
* if integration time very long then ERROR and print that they should create a gridpack
https://prodtask-dev.cern.ch/prodtask/inputlist_with_request/26924/
https://its.cern.ch/jira/browse/ATLMCPROD-7731FutureHannes MildnerHannes Mildnerhttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/34Lower limit on amount of run_athena jobs that should be run2022-02-18T15:50:50+01:00Spyros ArgyropoulosLower limit on amount of run_athena jobs that should be runAs shown in [this pipeline](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/pipelines/1093023) we can completely miss cases where `log.generate` files are not provided.
Frank suggested the following check that can be implemented i...As shown in [this pipeline](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/pipelines/1093023) we can completely miss cases where `log.generate` files are not provided.
Frank suggested the following check that can be implemented in e.g. the `run_athena` job:
```
if Nlogfiles < roundup(0.01*N_newDSIDs)
return 1
```Futurehttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/153Do not throw warning from dummy CI job if a jO has not been committed2021-12-08T11:47:38+01:00Spyros ArgyropoulosDo not throw warning from dummy CI job if a jO has not been committedSpyros ArgyropoulosSpyros Argyropoulos