MC Job Options issueshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues2024-02-21T14:55:47+01:00https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/224Enforce saveProcDir=False in MG joboptions2024-02-21T14:55:47+01:00Spyros ArgyropoulosEnforce saveProcDir=False in MG joboptionsin the Madgraph JOs there is the possibility of saving the process directory or delete it. While keeping it is very useful for debugging, we try to enforce to have it to False for official production as it might give issues for some proc...in the Madgraph JOs there is the possibility of saving the process directory or delete it. While keeping it is very useful for debugging, we try to enforce to have it to False for official production as it might give issues for some processes in grid jobs. Jan asked that if we are trying to enforce it systematically, it should be part of the logParser. We have discussed it a bit and were thinking whether it can be added a check in the logParser to make sure that it is set to saveProcDir=False and let the CI crash in case it is changed in any bug fix push to gitlab.Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/185Rebasing branching for automatically created MR when merging2024-02-18T07:24:20+01:00Spyros ArgyropoulosRebasing branching for automatically created MR when mergingWhen trying to merge automatically created MR we can get
```
./scripts/merge_request_api.sh -m
MR: 1993 - approvals left: 0
Merging 1993
{"message":"Branch cannot be merged"}
```
when previous MRs were merged in between.
One would have...When trying to merge automatically created MR we can get
```
./scripts/merge_request_api.sh -m
MR: 1993 - approvals left: 0
Merging 1993
{"message":"Branch cannot be merged"}
```
when previous MRs were merged in between.
One would have to rebase:
![Screenshot_2022-08-23_at_09.34.00](/uploads/d2d8219eaf65ec850b6169a63c62b2c8/Screenshot_2022-08-23_at_09.34.00.png)
from the CI which will launch another pipeline.
Need to see how to treat this in the pipeline.Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/210Make some jo checks stricter?2024-02-18T07:23:36+01:00Spyros ArgyropoulosMake some jo checks stricter?```
runArgs.inputGeneratorFile = outputDS.replace('tar.gz', 'events')
```
passes the check since when running the jO outside the transform it leads to an undefined object.
Maybe we need another way to avoid such issues.```
runArgs.inputGeneratorFile = outputDS.replace('tar.gz', 'events')
```
passes the check since when running the jO outside the transform it leads to an undefined object.
Maybe we need another way to avoid such issues.Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/225logParser failed due to missing platform info2024-02-01T14:46:57+01:00Yang LiulogParser failed due to missing platform infoHi @sargyrop , as we discussed in [Fixing automatic determination of release for CI runs (!2861) · Merge requests · atlas-physics / pmg / MC Job Options · GitLab (cern.ch)](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_re...Hi @sargyrop , as we discussed in [Fixing automatic determination of release for CI runs (!2861) · Merge requests · atlas-physics / pmg / MC Job Options · GitLab (cern.ch)](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/2861). It seems the added line to extract the platform info will cause problem for some of the jobs.
[Here](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/pipelines/6821117) is one example.
Many thanks for your time to help.
Cheers
Yanghttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/221CI has to run on AlamLinux9 environment2024-01-31T07:55:48+01:00Yiming AbulaitiCI has to run on AlamLinux9 environmentHi All,
The new release 23.6.20 is available only on the AlamLinux9 environment.
But GIT CI don't support AL9 yet.
So in order to test new JOs with 23.6.20 CI has to run on AL9.
Depends on https://gitlab.cern.ch/atlas/athena/-/merge_re...Hi All,
The new release 23.6.20 is available only on the AlamLinux9 environment.
But GIT CI don't support AL9 yet.
So in order to test new JOs with 23.6.20 CI has to run on AL9.
Depends on https://gitlab.cern.ch/atlas/athena/-/merge_requests/67225Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/223Block the usage of xAOD filters in HepMC3 releases2024-01-26T23:10:32+01:00Spyros ArgyropoulosBlock the usage of xAOD filters in HepMC3 releasesSee [this talk](https://indico.cern.ch/event/1372411/contributions/5770199/attachments/2788275/4861850/rel23_Jan2_24.pdf) for a description of the problem.
We want to block all jO that use HepMC3 (all r23.6 for the moment - to be fine t...See [this talk](https://indico.cern.ch/event/1372411/contributions/5770199/attachments/2788275/4861850/rel23_Jan2_24.pdf) for a description of the problem.
We want to block all jO that use HepMC3 (all r23.6 for the moment - to be fine tuned later) that use xAOD filters.Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/222Can I pass LHE container to inputGeneratorFile?2023-12-20T20:13:01+01:00Yiming AbulaitiCan I pass LHE container to inputGeneratorFile?Hi,
In log.generate.short, Can I pass a container to inputGeneratorFile?
for example:
nEventsPerJob = 5000
- Requested output events = 10000
- transform = Gen_tf
- inputFilesPerJob = 99
- inputGeneratorFile = </afs/cern.ch/work/som...Hi,
In log.generate.short, Can I pass a container to inputGeneratorFile?
for example:
nEventsPerJob = 5000
- Requested output events = 10000
- transform = Gen_tf
- inputFilesPerJob = 99
- inputGeneratorFile = </afs/cern.ch/work/somepath.../mc15_13TeV.345054.PowhegPythia8EvtGen_NNPDF3_AZNLO_WpH125J_MINLO_lvbb_VpT.evgen.TXT.e5706/TXT.10406367._003729.tar.gz.1> .... and more file-path like this.
Is it possible to use this:
- inputGeneratorFile = mc15_13TeV.345054.PowhegPythia8EvtGen_NNPDF3_AZNLO_WpH125J_MINLO_lvbb_VpT.evgen.TXT.e5706
And let rucio download 99 files? It is practically inconvenient to edit log.generate.short and put 99 file names.
Cheers,
Ablethttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/204Apply basic tests also to fragments included from top jO2023-12-20T18:58:13+01:00Spyros ArgyropoulosApply basic tests also to fragments included from top jOVia check_jo_content probablyVia check_jo_content probablySpyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/6Development model, CI triggering and refining of diff-filters2023-11-30T13:05:48+01:00Spyros ArgyropoulosDevelopment model, CI triggering and refining of diff-filters* [x] Define different triggers for jobs that can be skipped
* [x] Define branches on which jobs will run
* [x] Can we allow everyone to be a developer since the master is protected?
* [x] Refine `diff-filter`
**Job triggering**
A...* [x] Define different triggers for jobs that can be skipped
* [x] Define branches on which jobs will run
* [x] Can we allow everyone to be a developer since the master is protected?
* [x] Refine `diff-filter`
**Job triggering**
Allow for different triggers for jobs that can be skipped, so that users can adjust which jobs to run. We should also see on which branches the pipelines will run.
**Development model**
Switching to a development model where everyone uses branches is necessary since some pipelines (`check_logParser` and later the athena jobs) rely on hidden variables (mcgensvc password etc) that should not be visible to normal users. This means that if a user would fork the repository the variables would not be copied and the CI jobs would fail.
Since `master` is protected probably we can allow users to be developer, as done by other ATLAS packages.
**Refining of `diff-filter`**
We currently have the following diff-filters:
* `check_unique_physicsShort`: `diff-filter=A | grep -E "mc[0-9]{2}.*py"`
* `check_modified_files`: `diff-filter=MDCRTUXB`
* `check_logParser`: `diff-filter=AMDCRTUXB` and files must be named like `mcXX.*.py`
* `check_jo_consistency`: no diff-filter but triggered by `changes: "[0-9][0-9][0-9]xxx/**/mc[0-9][0-9].*.py"`
legend:
```
# Copied (C), Deleted (D), Modified (M), Renamed (R), have their type
# (i.e. regular file, symlink, submodule, …) changed (T), are Unmerged (U),
# are Unknown (X), or have had their pairing Broken (B).
```
The following perhaps need to be revised:
* not clear to me that we need `UXB` in `check_logParser`
* for `check_jo_consistency` triggering on any change in the DSID directories is probably too much. We should perhaps restrict this to only files named like `mcXX.*.py` with a `diff-filter=AMCRT`Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/220Query ProdSys instead of rucio when checking for registered files2023-11-09T15:57:22+01:00Spyros ArgyropoulosQuery ProdSys instead of rucio when checking for registered filesCurrently when `[skip modfiles]` is used `notify.sh` is checking on rucio if samples with a given DSID exist.
When a sample is obsoleted it apparently takes very long for the information to propagate so we should check if we can use Prod...Currently when `[skip modfiles]` is used `notify.sh` is checking on rucio if samples with a given DSID exist.
When a sample is obsoleted it apparently takes very long for the information to propagate so we should check if we can use ProdSys instead.Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/219logParser run failed.2023-10-30T17:39:36+01:00Yiming AbulaitilogParser run failed.logParser run failed due to the unprotected parameter "nEventsPerJob_fromJO". See the error message bellow
'''
- Number of input LHE events: 65000
Traceback (most recent call last):
File "./scripts/logParser.py", line 782, in <module>...logParser run failed due to the unprotected parameter "nEventsPerJob_fromJO". See the error message bellow
'''
- Number of input LHE events: 65000
Traceback (most recent call last):
File "./scripts/logParser.py", line 782, in <module>
main()
File "./scripts/logParser.py", line 683, in main
if expected_EVNT_out > 2 * nEventsPerJob_fromJO:
TypeError: unsupported operand type(s) for *: 'int' and 'NoneType'
'''
In line https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/blob/master/scripts/logParser.py?ref_type=heads#L683
The variable "nEventsPerJob_fromJO" is used but it can be None type when the neventsPerjob is not specified in JO file.
You could just you "nEventsPerJob" variable since it is already overwritten by "nEventsPerJob_fromJO" or set to 10000 if "nEventsPerJob_fromJO" is None.
see line: https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/blame/master/scripts/logParser.py#L518
For test, you can download a log.generate file here: https://cernbox.cern.ch/s/U86AjY5bTjTACwy
Cheers,
AbletSpyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/124New Pythia 8 checks for changing parameters2023-10-26T16:13:33+02:00Spyros ArgyropoulosNew Pythia 8 checks for changing parametersImplement code to use new developments by Giancarlo mentioned in AGENE-1915.
- [ ] To be seen which of these should result in an error and which should be a warning.
- [ ] Also check if this catches the bug reported in ATLMCPROD-7723Implement code to use new developments by Giancarlo mentioned in AGENE-1915.
- [ ] To be seen which of these should result in an error and which should be a warning.
- [ ] Also check if this catches the bug reported in ATLMCPROD-7723S1.2021Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/2185360 GeV shoudl be an allowed energy in the CI2023-10-20T11:15:28+02:00Jan Kretzschmar5360 GeV shoudl be an allowed energy in the CIhttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/2669
Run 3 HI and reference run energy will be 5360 GeV, so the CI should not fail with
"ERROR: unknown ecmEnergy: 5360.0"
Full log file also on lxplus in ~jkretz/pu...https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/2669
Run 3 HI and reference run energy will be 5360 GeV, so the CI should not fail with
"ERROR: unknown ecmEnergy: 5360.0"
Full log file also on lxplus in ~jkretz/public/log.generateSpyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/217How can I run the CI longer then 24 hours?2023-10-04T17:34:07+02:00Yiming AbulaitiHow can I run the CI longer then 24 hours?Hi,
Some metrics element calculation in MG takes longer than 24 hours. But the CI time limit is 24 hours.
Is it allowed to run the CI longer than 24 hours?
Cheers.
AbletHi,
Some metrics element calculation in MG takes longer than 24 hours. But the CI time limit is 24 hours.
Is it allowed to run the CI longer than 24 hours?
Cheers.
Ablethttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/198Investigate usage of sparse checkout2023-09-22T08:26:33+02:00Spyros ArgyropoulosInvestigate usage of sparse checkout# Old solution
```
mkdir mcjoboptions
cd mcjoboptions
git init
git remote add -f origin ssh://git@gitlab.cern.ch:7999/atlas-physics/pmg/mcjoboptions.git
git config core.sparseCheckout true
echo scripts > .git/info/sparse-checkout
echo c...# Old solution
```
mkdir mcjoboptions
cd mcjoboptions
git init
git remote add -f origin ssh://git@gitlab.cern.ch:7999/atlas-physics/pmg/mcjoboptions.git
git config core.sparseCheckout true
echo scripts > .git/info/sparse-checkout
echo common >> .git/info/sparse-checkout
echo .gitignore >> .git/info/sparse-checkout
git pull origin master
```
This works - **notice that the `700xxx` directory is not cloned - the script automatically finds the correct DSID directory that should be created**
![Screenshot_2023-01-03_at_09.39.15](/uploads/bbf3ad31b4bdc38bf29c5e8723b1fa4b/Screenshot_2023-01-03_at_09.39.15.png)
# New solution
Needs git 2.26 or higher
```
mkdir mcjoboptions
cd mcjoboptions
git init
git remote add -f origin ssh://git@gitlab.cern.ch:7999/atlas-physics/pmg/mcjoboptions.git
git sparse-checkout init
git sparse-checkout set common scripts .gitignore
git pull origin master
```Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/214Assigning same DSID twice2023-09-15T19:06:30+02:00Yiming AbulaitiAssigning same DSID twiceWhen I try to generate new DSID, the same DSID assigned twice "524xxx/524547"
Bellow is the output from "./scripts/commit_new_dsid -m "Commit message" --dry-run ../100xxx/*"
Will move ../100xxx/100003 to 524xxx/524546.
...When I try to generate new DSID, the same DSID assigned twice "524xxx/524547"
Bellow is the output from "./scripts/commit_new_dsid -m "Commit message" --dry-run ../100xxx/*"
Will move ../100xxx/100003 to 524xxx/524546.
Will move ../100xxx/100005 to 524xxx/524547.
Will move ../100xxx/100000 to 524xxx/524546.
Will move ../100xxx/100001 to 524xxx/524547.
Will move ../100xxx/100007 to 602xxx/602485.
Will move ../100xxx/100008 to 602xxx/602486.
Cheers,
AbletSpyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/213Job Failed #314314072023-07-31T17:48:45+02:00Yanlin LiuJob Failed #31431407Job [#31431407](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/31431407) failed for 2afb6c436741b53d42df6790447447250439f02d:
Dear expert, I'm not quite sure what's this error about? Any insights?
Thanks,
YanlinJob [#31431407](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/31431407) failed for 2afb6c436741b53d42df6790447447250439f02d:
Dear expert, I'm not quite sure what's this error about? Any insights?
Thanks,
YanlinSpyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/212Allow running `mc??_valid` input files2023-07-31T09:40:55+02:00Spyros ArgyropoulosAllow running `mc??_valid` input filesCurrently the CI cannot run on input files named like `mc15_valid.602232.Ph_ttj_MiNNLO_scale5_LHE.evgen.TXT.e8531/TXT.34045098._000002.tar.gz.1`
The `valid` replaces the COM energy so it is potentially a not well defined naming conventi...Currently the CI cannot run on input files named like `mc15_valid.602232.Ph_ttj_MiNNLO_scale5_LHE.evgen.TXT.e8531/TXT.34045098._000002.tar.gz.1`
The `valid` replaces the COM energy so it is potentially a not well defined naming convention, since the COM energy is taken automatically from `ecmEnergy` from `log.generate` which is [directly printed from the transform](https://gitlab.cern.ch/atlas/athena/-/blob/main/Generators/EvgenJobTransforms/share/skel.GENtoEVGEN.py#L105) - from the command line arguments.
One solution is to ignore the `ecmEnergy` completely if we see that the input filed is named as `mc??_valid` however I think that's problematic because an LHE should correspond to a given COM energy and we should not allow it to be used indepndently of the `ecmEnergy`.
@katharin @mgignac @dhirschSpyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/211Job Failed #309375482023-07-11T15:39:23+02:00Yiming AbulaitiJob Failed #30937548Job [#30937548](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/30937548) failed for af0d387553a5529f8514e942f2756bd1e17166e1:
How can I pass this?
If I use full container name "mc15_13TeV.345054.PowhegPythia8EvtGen_NNPDF3_...Job [#30937548](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/30937548) failed for af0d387553a5529f8514e942f2756bd1e17166e1:
How can I pass this?
If I use full container name "mc15_13TeV.345054.PowhegPythia8EvtGen_NNPDF3_AZNLO_WpH125J_MINLO_lvbb_VpT.evgen.TXT.e
5706" then the CI job try to download 500K files.
If I use this "mc15_13TeV:TXT.10406367._041965.tar.gz.1, ...." in log.generate.short then some script add one extra "mc15_13TeV". Then the download failed because rucio can not download "mc15_13TeV:mc15_13TeV:TXT.10406367._041965.tar.gz.1".
If I use only "TXT.10406367._041965.tar.gz.1, ...." format as inputs then CI job failed saying "could not find scope". see the link above.
Any solution for this kind of thing?
I tested locally that "rucio download TXT.10406367._041965.tar.gz.1" works fine, but CI script complain about scope.
Cheers,
AbletSpyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/208Speed up retrieval of free DSIDs2023-05-20T16:26:07+02:00Spyros ArgyropoulosSpeed up retrieval of free DSIDsSpyros ArgyropoulosSpyros Argyropoulos