MC Job Options issueshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues2020-07-27T13:10:20+02:00https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/98Extraction of nEventsPerJob from file2020-07-27T13:10:20+02:00Spyros ArgyropoulosExtraction of nEventsPerJob from file### Bug
```
nEventsPerJob=5000 -> this is what the logParser reads from the file
# I can change it here
if (efficiencyLow) nEventsPerJob*=2
evgenConfig.nEventsPerJob = nEventsPerJob -> this is what the transform will use
```
### Solut...### Bug
```
nEventsPerJob=5000 -> this is what the logParser reads from the file
# I can change it here
if (efficiencyLow) nEventsPerJob*=2
evgenConfig.nEventsPerJob = nEventsPerJob -> this is what the transform will use
```
### Solution
Credits to Frank Sauerburger
Put the following in `scriptB.py`
```
import argparse
def readParamFromJO(jOpath, param):
locals = {"evgenConfig": argparse.Namespace()}
with open(jOpath) as jOFile:
for line in jOFile.readlines():
if "os.system" in line: continue # for security
try:
exec(line, {}, locals)
except:
# print(f"fail to parse {line}") # uncomment for debugging
pass
return getattr(locals["evgenConfig"], param) if hasattr(locals["evgenConfig"], param) else None
jOFile="./source/mc.scriptA.py"
nEventsPerJob=readParamFromJO(jOFile, 'nEventsPerJob')
# Check nEventsPerJob
if nEventsPerJob is None:
print(f"WARNING: evgenConfig.nEventsPerJob is not defined in the jO. Will set to default=10000")
nEventsPerJob=10000
else:
print(f"nEventsPerJob from jO={nEventsPerJob}")
# Check minEvents
if readParamFromJO(jOFile, 'minEvents') is not None:
print(f"ERROR: {jOFile} is using deprecated parameter evgenConfig.minEvents. Please switch to evgenConfig.nEventsPerJob")
```
### Testing:
Put the following in `./source/mc.scriptA.py`
```
import Sherpa_i.Sherpa_iConf
import os
import GeneratorFilters.GeneratorFiltersConf
include("./scriptA.py") # this doesn't work because python doesn't know what include is
evgenConfig.XVAR=5
filtSeq.YVAR=10
evgenConfig.nEventsPerJob=1
evgenConfig.nEventsPerJob=2
evgenConfig.nEventsPerJob*=3
evgenConfig.nEventsPerJob=os.system("rm test")
#evgenConfig.nEventsPerJob=10
#print(f"{evgenConfig.nEventsPerJob}")
```
Running `python3 scriptB.py` gives
```
fail to parse import Sherpa_i.Sherpa_iConf
fail to parse import GeneratorFilters.GeneratorFiltersConf
fail to parse include("./scriptA.py") # this doesn't work because python doesn't know what include is
fail to parse filtSeq.YVAR=10
Final Answer: nEventsPerJob=6
```
The added bonus is that if there is no `evgenConfig.nEventsPerJob` defined this would automatically throw an error.
## What is done in ProdSys
The first occurence of `evgenConfig.nEventsPerJob` is usedS2.2020Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/97Handling of jO files processed in non-Unix filesystems2020-04-17T13:57:23+02:00Spyros ArgyropoulosHandling of jO files processed in non-Unix filesystemsFrom @avroy
> calculating [nEvents](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/blob/master/scripts/run_athena.sh#L70) failed with the following errors:
```
(standard_in) 1: illegal character: ^M
(standard_in) 1: illegal c...From @avroy
> calculating [nEvents](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/blob/master/scripts/run_athena.sh#L70) failed with the following errors:
```
(standard_in) 1: illegal character: ^M
(standard_in) 1: illegal character: ^M
```
> Naively, this is due to carriage return and not uniformly processed across operating systems
## Todo
See how to handle this:
* during commit script? : probably not ideal since not everyone uses it
* doing a `dos2unix` in the CI? : might require special image - need to see if `dos2unix` is available in the images we use
* doing a `sed 's/^M//g'` as described [here](https://stackoverflow.com/questions/2658931/why-error-illegal-character-m?answertab=votes#tab-top) in all CI jobs? : @avroy can you test whether this works for you?S1.2020Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/90Job Failed #7575152 'RunArguments' object has no attribute 'inputGeneratorFile'2020-04-16T13:32:39+02:00Xiaohu SunJob Failed #7575152 'RunArguments' object has no attribute 'inputGeneratorFile'Job [#7575152](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/7575152) failed for c3035481c6ec0ec00926a14dc33427bb6b590fb1:
Dear experts,
To my understanding this relates to external LHE files. Please correct me if I am ...Job [#7575152](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/7575152) failed for c3035481c6ec0ec00926a14dc33427bb6b590fb1:
Dear experts,
To my understanding this relates to external LHE files. Please correct me if I am wrong.
The only connection between the JO / DSID and the external LHE files seems only in the spreadsheet.
The spreadsheet is not known by the CI system while uploading JOs.
Do we need some special setups to point to some afs or eos location of the external LHE files?
Thanks!
Best,
Xiaohuhttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/228Add functionality to allow creating log.generate.short without running the co...2024-03-15T15:47:38+01:00Spyros ArgyropoulosAdd functionality to allow creating log.generate.short without running the commit scriptThis can either be done by supplying an extra script or updating the commit script to handle such cases (might be better).
Standalone code is provided below.
The code below works with
`python3 makeLogShort.py ./path/to/log.generate ./p...This can either be done by supplying an extra script or updating the commit script to handle such cases (might be better).
Standalone code is provided below.
The code below works with
`python3 makeLogShort.py ./path/to/log.generate ./path/to/mc.XXX.py`
```python
#! /usr/bin/env python3
import re,sys, subprocess, shlex
def bashExec(cmd, debug = False, checkStatus = False):
if debug:
print(cmd)
return ''
p = subprocess.run(shlex.split(cmd), stdout=subprocess.PIPE)
if checkStatus:
return p.stdout.decode('utf-8').strip(), p.returncode
return p.stdout.decode('utf-8').strip()
tags = [
'estimated CPU for CI job',
'using release',
'ecmEnergy',
'inputGeneratorFile',
'inputEVNT_PreFile',
'randomSeed',
'EVNT to EVNT',
'LHEonly',
'ATHENA_PROC_NUMBER',
'platform'
]
logparser_cmd = f'python3 scripts/logParser.py -i {sys.argv[1]} -j {sys.argv[2]}'
LPO, rc = bashExec(logparser_cmd, checkStatus = True)
# Remove colours from logParserOutput and extract necessary information
LPO_noCols = re.sub(r'\x1b\[[0-9]{1,3}[mK]', '', LPO)
greps = sum((re.compile(f'.*{tag}.*').findall(LPO_noCols)[:1] for tag in tags), [])
# Pipe information into log.generate.short
with open(f'log.generate.short', 'w') as logShort:
logShort.write('\n'.join(greps))
```Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/227Establish CI to handle stale MRs and branches2024-02-19T11:14:22+01:00Spyros ArgyropoulosEstablish CI to handle stale MRs and branchesFollowing !2906 we should add a CI job that uses this functionality to handle MRs that are open for long with failed pipelines and also delete the associated branches when closing the MRsFollowing !2906 we should add a CI job that uses this functionality to handle MRs that are open for long with failed pipelines and also delete the associated branches when closing the MRsSpyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/226Automatically handle TestHepMC efficiency warnings and errors2024-02-04T12:02:36+01:00Spyros ArgyropoulosAutomatically handle TestHepMC efficiency warnings and errorsOnce https://gitlab.cern.ch/atlas/athena/-/merge_requests/68566 is merged we should change the hardcoded thresholds in `logParser` and allow the check to be performed dynamically.Once https://gitlab.cern.ch/atlas/athena/-/merge_requests/68566 is merged we should change the hardcoded thresholds in `logParser` and allow the check to be performed dynamically.Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/225logParser failed due to missing platform info2024-02-01T14:46:57+01:00Yang LiulogParser failed due to missing platform infoHi @sargyrop , as we discussed in [Fixing automatic determination of release for CI runs (!2861) · Merge requests · atlas-physics / pmg / MC Job Options · GitLab (cern.ch)](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_re...Hi @sargyrop , as we discussed in [Fixing automatic determination of release for CI runs (!2861) · Merge requests · atlas-physics / pmg / MC Job Options · GitLab (cern.ch)](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/2861). It seems the added line to extract the platform info will cause problem for some of the jobs.
[Here](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/pipelines/6821117) is one example.
Many thanks for your time to help.
Cheers
Yanghttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/224Enforce saveProcDir=False in MG joboptions2024-02-21T14:55:47+01:00Spyros ArgyropoulosEnforce saveProcDir=False in MG joboptionsin the Madgraph JOs there is the possibility of saving the process directory or delete it. While keeping it is very useful for debugging, we try to enforce to have it to False for official production as it might give issues for some proc...in the Madgraph JOs there is the possibility of saving the process directory or delete it. While keeping it is very useful for debugging, we try to enforce to have it to False for official production as it might give issues for some processes in grid jobs. Jan asked that if we are trying to enforce it systematically, it should be part of the logParser. We have discussed it a bit and were thinking whether it can be added a check in the logParser to make sure that it is set to saveProcDir=False and let the CI crash in case it is changed in any bug fix push to gitlab.Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/223Block the usage of xAOD filters in HepMC3 releases2024-01-26T23:10:32+01:00Spyros ArgyropoulosBlock the usage of xAOD filters in HepMC3 releasesSee [this talk](https://indico.cern.ch/event/1372411/contributions/5770199/attachments/2788275/4861850/rel23_Jan2_24.pdf) for a description of the problem.
We want to block all jO that use HepMC3 (all r23.6 for the moment - to be fine t...See [this talk](https://indico.cern.ch/event/1372411/contributions/5770199/attachments/2788275/4861850/rel23_Jan2_24.pdf) for a description of the problem.
We want to block all jO that use HepMC3 (all r23.6 for the moment - to be fine tuned later) that use xAOD filters.Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/222Can I pass LHE container to inputGeneratorFile?2023-12-20T20:13:01+01:00Yiming AbulaitiCan I pass LHE container to inputGeneratorFile?Hi,
In log.generate.short, Can I pass a container to inputGeneratorFile?
for example:
nEventsPerJob = 5000
- Requested output events = 10000
- transform = Gen_tf
- inputFilesPerJob = 99
- inputGeneratorFile = </afs/cern.ch/work/som...Hi,
In log.generate.short, Can I pass a container to inputGeneratorFile?
for example:
nEventsPerJob = 5000
- Requested output events = 10000
- transform = Gen_tf
- inputFilesPerJob = 99
- inputGeneratorFile = </afs/cern.ch/work/somepath.../mc15_13TeV.345054.PowhegPythia8EvtGen_NNPDF3_AZNLO_WpH125J_MINLO_lvbb_VpT.evgen.TXT.e5706/TXT.10406367._003729.tar.gz.1> .... and more file-path like this.
Is it possible to use this:
- inputGeneratorFile = mc15_13TeV.345054.PowhegPythia8EvtGen_NNPDF3_AZNLO_WpH125J_MINLO_lvbb_VpT.evgen.TXT.e5706
And let rucio download 99 files? It is practically inconvenient to edit log.generate.short and put 99 file names.
Cheers,
Ablethttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/221CI has to run on AlamLinux9 environment2024-01-31T07:55:48+01:00Yiming AbulaitiCI has to run on AlamLinux9 environmentHi All,
The new release 23.6.20 is available only on the AlamLinux9 environment.
But GIT CI don't support AL9 yet.
So in order to test new JOs with 23.6.20 CI has to run on AL9.
Depends on https://gitlab.cern.ch/atlas/athena/-/merge_re...Hi All,
The new release 23.6.20 is available only on the AlamLinux9 environment.
But GIT CI don't support AL9 yet.
So in order to test new JOs with 23.6.20 CI has to run on AL9.
Depends on https://gitlab.cern.ch/atlas/athena/-/merge_requests/67225Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/220Query ProdSys instead of rucio when checking for registered files2023-11-09T15:57:22+01:00Spyros ArgyropoulosQuery ProdSys instead of rucio when checking for registered filesCurrently when `[skip modfiles]` is used `notify.sh` is checking on rucio if samples with a given DSID exist.
When a sample is obsoleted it apparently takes very long for the information to propagate so we should check if we can use Prod...Currently when `[skip modfiles]` is used `notify.sh` is checking on rucio if samples with a given DSID exist.
When a sample is obsoleted it apparently takes very long for the information to propagate so we should check if we can use ProdSys instead.Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/219logParser run failed.2023-10-30T17:39:36+01:00Yiming AbulaitilogParser run failed.logParser run failed due to the unprotected parameter "nEventsPerJob_fromJO". See the error message bellow
'''
- Number of input LHE events: 65000
Traceback (most recent call last):
File "./scripts/logParser.py", line 782, in <module>...logParser run failed due to the unprotected parameter "nEventsPerJob_fromJO". See the error message bellow
'''
- Number of input LHE events: 65000
Traceback (most recent call last):
File "./scripts/logParser.py", line 782, in <module>
main()
File "./scripts/logParser.py", line 683, in main
if expected_EVNT_out > 2 * nEventsPerJob_fromJO:
TypeError: unsupported operand type(s) for *: 'int' and 'NoneType'
'''
In line https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/blob/master/scripts/logParser.py?ref_type=heads#L683
The variable "nEventsPerJob_fromJO" is used but it can be None type when the neventsPerjob is not specified in JO file.
You could just you "nEventsPerJob" variable since it is already overwritten by "nEventsPerJob_fromJO" or set to 10000 if "nEventsPerJob_fromJO" is None.
see line: https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/blame/master/scripts/logParser.py#L518
For test, you can download a log.generate file here: https://cernbox.cern.ch/s/U86AjY5bTjTACwy
Cheers,
AbletSpyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/2185360 GeV shoudl be an allowed energy in the CI2023-10-20T11:15:28+02:00Jan Kretzschmar5360 GeV shoudl be an allowed energy in the CIhttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/2669
Run 3 HI and reference run energy will be 5360 GeV, so the CI should not fail with
"ERROR: unknown ecmEnergy: 5360.0"
Full log file also on lxplus in ~jkretz/pu...https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/2669
Run 3 HI and reference run energy will be 5360 GeV, so the CI should not fail with
"ERROR: unknown ecmEnergy: 5360.0"
Full log file also on lxplus in ~jkretz/public/log.generateSpyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/217How can I run the CI longer then 24 hours?2023-10-04T17:34:07+02:00Yiming AbulaitiHow can I run the CI longer then 24 hours?Hi,
Some metrics element calculation in MG takes longer than 24 hours. But the CI time limit is 24 hours.
Is it allowed to run the CI longer than 24 hours?
Cheers.
AbletHi,
Some metrics element calculation in MG takes longer than 24 hours. But the CI time limit is 24 hours.
Is it allowed to run the CI longer than 24 hours?
Cheers.
Ablethttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/216Allow rucio to download files stored on datadisk2023-09-29T15:48:41+02:00Spyros ArgyropoulosAllow rucio to download files stored on datadiskIn !2641 the CI failed because the input LHE was stored on datadisk
```
rucio list-file-replicas mc23_13p6TeV:TXT.33690939._000003.tar.gz.1
| mc23_13p6TeV | TXT.33690939._000003.tar.gz.1 | 1.345 MB | 6ed2753b | FZK-LCG2_DATADISK: roo...In !2641 the CI failed because the input LHE was stored on datadisk
```
rucio list-file-replicas mc23_13p6TeV:TXT.33690939._000003.tar.gz.1
| mc23_13p6TeV | TXT.33690939._000003.tar.gz.1 | 1.345 MB | 6ed2753b | FZK-LCG2_DATADISK: root://atlasxrootd-kit.gridka.de:1094//pnfs/gridka.de/atlas/disk-only/atlasdatadisk/rucio/mc23_13p6TeV/c5/61/TXT.33690939._000003.tar.gz.1 |
```
Misha said this is not an issue for prodsys because `rucio download` can be used from everywhere.
Apparently we can request special access for `mcgensvc` from mailto:atlas-adc-ddm-support@cern.ch
(we should keep Misha in cc)
We probably also need to request a new grid certificate for `mcgensvc`Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/215Catch cases where MadSpin uses BR that don't add up to 12023-09-23T06:51:44+02:00Spyros ArgyropoulosCatch cases where MadSpin uses BR that don't add up to 1From @dhirsch
> today, we have discovered a bug in the aMC@NLO ttbar samples (see discussion on the top reco mailing list) It turned out that the branching rations for the W boson decay in the param_card didn't add up to one. I.e. the i...From @dhirsch
> today, we have discovered a bug in the aMC@NLO ttbar samples (see discussion on the top reco mailing list) It turned out that the branching rations for the W boson decay in the param_card didn't add up to one. I.e. the is a typo in two values. However MadSpin basically reported this problem with this line: INFO: Branching ratio to allowed decays: 0.984064
> We also have the case of samples generated with gridpacks. In that case the parameter card would be taken from the gridpack. This is the case for the ttH sample (I checked 346443). One can see for this sample that no BR is specified in the parameter card used; then in the header of the lhe file, I see the following values:
```
DECAY 24 2.046400e+00
# BR NDA ID1 ID2 ...
3.333659e-01 2 -1 2 # 0.68219997776
3.333659e-01 2 -3 4 # 0.68219997776
1.111171e-01 2 -11 12 # 0.22739003344
1.111171e-01 2 -13 14 # 0.22739003344
1.110340e-01 2 -15 16 # 0.2272199776
```
It's probably these two samples
```
aMC@NLO+Pythia
410465 aMcAtNloPy8EvtGen_MEN30NLO_A14N23LO_ttbar_noShWe_dil
410464 aMcAtNloPy8EvtGen_MEN30NLO_A14N23LO_ttbar_noShWe_SingleLep
```Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/214Assigning same DSID twice2023-09-15T19:06:30+02:00Yiming AbulaitiAssigning same DSID twiceWhen I try to generate new DSID, the same DSID assigned twice "524xxx/524547"
Bellow is the output from "./scripts/commit_new_dsid -m "Commit message" --dry-run ../100xxx/*"
Will move ../100xxx/100003 to 524xxx/524546.
...When I try to generate new DSID, the same DSID assigned twice "524xxx/524547"
Bellow is the output from "./scripts/commit_new_dsid -m "Commit message" --dry-run ../100xxx/*"
Will move ../100xxx/100003 to 524xxx/524546.
Will move ../100xxx/100005 to 524xxx/524547.
Will move ../100xxx/100000 to 524xxx/524546.
Will move ../100xxx/100001 to 524xxx/524547.
Will move ../100xxx/100007 to 602xxx/602485.
Will move ../100xxx/100008 to 602xxx/602486.
Cheers,
AbletSpyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/213Job Failed #314314072023-07-31T17:48:45+02:00Yanlin LiuJob Failed #31431407Job [#31431407](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/31431407) failed for 2afb6c436741b53d42df6790447447250439f02d:
Dear expert, I'm not quite sure what's this error about? Any insights?
Thanks,
YanlinJob [#31431407](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/31431407) failed for 2afb6c436741b53d42df6790447447250439f02d:
Dear expert, I'm not quite sure what's this error about? Any insights?
Thanks,
YanlinSpyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/212Allow running `mc??_valid` input files2023-07-31T09:40:55+02:00Spyros ArgyropoulosAllow running `mc??_valid` input filesCurrently the CI cannot run on input files named like `mc15_valid.602232.Ph_ttj_MiNNLO_scale5_LHE.evgen.TXT.e8531/TXT.34045098._000002.tar.gz.1`
The `valid` replaces the COM energy so it is potentially a not well defined naming conventi...Currently the CI cannot run on input files named like `mc15_valid.602232.Ph_ttj_MiNNLO_scale5_LHE.evgen.TXT.e8531/TXT.34045098._000002.tar.gz.1`
The `valid` replaces the COM energy so it is potentially a not well defined naming convention, since the COM energy is taken automatically from `ecmEnergy` from `log.generate` which is [directly printed from the transform](https://gitlab.cern.ch/atlas/athena/-/blob/main/Generators/EvgenJobTransforms/share/skel.GENtoEVGEN.py#L105) - from the command line arguments.
One solution is to ignore the `ecmEnergy` completely if we see that the input filed is named as `mc??_valid` however I think that's problematic because an LHE should correspond to a given COM energy and we should not allow it to be used indepndently of the `ecmEnergy`.
@katharin @mgignac @dhirschSpyros ArgyropoulosSpyros Argyropoulos