MC Job Options issueshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues2023-10-20T11:15:28+02:00https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/2185360 GeV shoudl be an allowed energy in the CI2023-10-20T11:15:28+02:00Jan Kretzschmar5360 GeV shoudl be an allowed energy in the CIhttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/2669
Run 3 HI and reference run energy will be 5360 GeV, so the CI should not fail with
"ERROR: unknown ecmEnergy: 5360.0"
Full log file also on lxplus in ~jkretz/pu...https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/2669
Run 3 HI and reference run energy will be 5360 GeV, so the CI should not fail with
"ERROR: unknown ecmEnergy: 5360.0"
Full log file also on lxplus in ~jkretz/public/log.generateSpyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/118Add checks for `inputfilecheck` and `inputGeneratorFile`2020-08-03T10:25:32+02:00Christian GutschowAdd checks for `inputfilecheck` and `inputGeneratorFile`Please see this test commit: 52aa8087
which has the following two lines in the JO:
```
evgenConfig.inputfilecheck = 'PhPy8EG_NNPDF30LO_EWK_ZZeeee'
runArgs.inputGeneratorFile = 'PhPy8EG_NNPDF30LO_EWK_ZZeeee._00052.events.tar.gz'
```
Th...Please see this test commit: 52aa8087
which has the following two lines in the JO:
```
evgenConfig.inputfilecheck = 'PhPy8EG_NNPDF30LO_EWK_ZZeeee'
runArgs.inputGeneratorFile = 'PhPy8EG_NNPDF30LO_EWK_ZZeeee._00052.events.tar.gz'
```
The first one I thought the CI would already be catching [along with `inputconfcheck`, no?] and the second one is clearly a problem for central production.
Can we catch these? I guess the logParser should already throw an error before the files are even committed to gitlab.S2.2020Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/102Add checks for input files2020-07-30T07:37:56+02:00Spyros ArgyropoulosAdd checks for input filesAdd checks:
* [ ] no `evgenConfig.inputfilecheck`
* [ ] no `evgenConfig.inputconfcheck` allowed
both are always in the top JO
Also
* [ ] Restructure checks so that everything related to reading the jO is done in one place and everyt...Add checks:
* [ ] no `evgenConfig.inputfilecheck`
* [ ] no `evgenConfig.inputconfcheck` allowed
both are always in the top JO
Also
* [ ] Restructure checks so that everything related to reading the jO is done in one place and everything related to reading the log is done in `logParser`S2.2020https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/166Add checks for python2/3 compatibility of jO2022-04-21T17:21:24+02:00Spyros ArgyropoulosAdd checks for python2/3 compatibility of jOan example DSID is 830099: https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/blob/master/830xxx/830099/mc.H7EG_jetjet_72_Cluster_JZ1.py
R21: 21.6.85
R22: You can try 22.6.13 (later releases have issues with EvtGen_i — should be ...an example DSID is 830099: https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/blob/master/830xxx/830099/mc.H7EG_jetjet_72_Cluster_JZ1.py
R21: 21.6.85
R22: You can try 22.6.13 (later releases have issues with EvtGen_i — should be fixed soon).Spyros ArgyropoulosSpyros Argyropoulos2022-04-25https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/85Add check that only 1 jO is inside a DSID directory2020-03-03T13:33:33+01:00Spyros ArgyropoulosAdd check that only 1 jO is inside a DSID directoryCheck that only one file called `mc.*.py` is in a DSID directory.Check that only one file called `mc.*.py` is in a DSID directory.S1.2020Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/95Add check to see if gridpack was used and if the grid pack is provided2020-08-04T13:43:36+02:00Spyros ArgyropoulosAdd check to see if gridpack was used and if the grid pack is provided
I was wondering how to catch such cases and avoid having pipeline jobs running for 1h and failing without apparent reason. We would need an indicator in log.generate that a gridpack was used.
I don't see PowhegConfig.gridpack printed i...
I was wondering how to catch such cases and avoid having pipeline jobs running for 1h and failing without apparent reason. We would need an indicator in log.generate that a gridpack was used.
I don't see PowhegConfig.gridpack printed in the log that Olga provided. I see
```
16:47:17 Py:PowhegControl INFO | powheginput keyword use-old-grid set to 1.0000000000000000
Does this tell us whether a gridpack was used?
```
Comment by @fsiegert
> Hi @sargyrop,
I think there are things which we'll never be able to catch if requesters modify the DSID directory before submitting but after having run the evgen test. This is not only relevant for gridpacks, but also potentially removing include files etc. So I wouldn't put too much effort into catching these cases if it's not easy.
We just need to educate users that they:
run the evgen test in a clean working directory
should not modify the DSID directory before submission
Best,
Frank
I think this is a pretty straightforward check: if ((gridpack used) && ! (gridpack present)) then ERROR So I am only asking how to specify (gridpack used)
Comment by @amoroso :
> Hi @fsiegert, @sargyrop,
I wonder if we couldn't catch case 2 within the CI. We could add a checksum to the DSID directory to the Gen_tf output, and have a pipeline check that the checksum in the attached logfile and the one recomputed by the CI are the same.
cheers, Simone
## Solution for Madgraph
GRID presence can be identified by lines like:
```
06:17:07 Py:MadGraphUtils INFO Generating events from gridpack
```S2.2020Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/76Add dat files in DSID directories in whitelist2020-02-21T10:51:10+01:00Spyros ArgyropoulosAdd dat files in DSID directories in whitelistShould add `*.dat` in whitelistShould add `*.dat` in whitelistS1.2020Spyros ArgyropoulosSpyros Argyropoulos2020-02-15https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/87Allo jO files to be links in whitelist2020-03-09T15:22:42+01:00Spyros ArgyropoulosAllo jO files to be links in whitelistAs needed in !265
`mc.*.py` should be allowed as a link tooAs needed in !265
`mc.*.py` should be allowed as a link tooS1.2020Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/64Allow CI to run with external LHE inputs2021-04-27T12:14:25+02:00Spyros ArgyropoulosAllow CI to run with external LHE inputsThe CI can currently not yet run jobs with input event files. Largely the same tasks appear for both LHE->EVNT and EVNT->EVNT files:
* [x] Upload mcgensvc grid certificate in CI and make sure `voms-proxy-init -voms atlas` works in the CI...The CI can currently not yet run jobs with input event files. Largely the same tasks appear for both LHE->EVNT and EVNT->EVNT files:
* [x] Upload mcgensvc grid certificate in CI and make sure `voms-proxy-init -voms atlas` works in the CI
* [x] Write (rucio) file that was used in local testing into `log.generate.short`, e.g. `inputGeneratorFile=TXT.<dsid>.tar.gz` for LHE or `inputEVNTFile=1231231.EVNT.pool.root` for EVNT input.
* [x] If input file is specified in `log.generate.short`, the CI `run_athena.sh` job should `rucio get` that file and add the corresponding arguments to the `Gen_tf.py` command line as described in [Twiki](https://twiki.cern.ch/twiki/bin/view/AtlasProtected/SpecialConfigurations#Using_event_input_LHE_EVNT_or_EV).
* [x] Special treatment for EVNT->EVNT jobs: Apparently such jobs do not produce a `log.generate` file but a `log.afterburn`. Disable logParser for these, or special treatment? (move to #139)
* [ ] Ideally, the CI should also check the `inputFilesPerJob` in the JO for reasonable values at this stage: it should not exceed `10GB/sizeof(downloadedTestFile)` to make sure they fit into a grid node.
* [x] Similar case that came up in #89 is running athena with `--inputGeneratorFile`
Example !203
LHE-only log: `~sargyrop/public/log.generate_LHE`S1.2021Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/46Allow GRID files to point to other links2019-11-04T18:15:00+01:00Spyros ArgyropoulosAllow GRID files to point to other linksCurrently the CI will check if a GRID file (link) points to eos or cvmfs.
We want to allow the possibility that a link points to another link.
```
while $(test -L $GRID) ; do
GRID=$(readlink $GRID) ;
done
```
Alternatively we can...Currently the CI will check if a GRID file (link) points to eos or cvmfs.
We want to allow the possibility that a link points to another link.
```
while $(test -L $GRID) ; do
GRID=$(readlink $GRID) ;
done
```
Alternatively we can replace `readlink` with `readlink -e` (if we want to test for the existence of the final file) or `readlink -f` (if we don't care about testing that the final file exists). Need to see if `-e/f` are available in the CI image.AlphaSpyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/216Allow rucio to download files stored on datadisk2023-09-29T15:48:41+02:00Spyros ArgyropoulosAllow rucio to download files stored on datadiskIn !2641 the CI failed because the input LHE was stored on datadisk
```
rucio list-file-replicas mc23_13p6TeV:TXT.33690939._000003.tar.gz.1
| mc23_13p6TeV | TXT.33690939._000003.tar.gz.1 | 1.345 MB | 6ed2753b | FZK-LCG2_DATADISK: roo...In !2641 the CI failed because the input LHE was stored on datadisk
```
rucio list-file-replicas mc23_13p6TeV:TXT.33690939._000003.tar.gz.1
| mc23_13p6TeV | TXT.33690939._000003.tar.gz.1 | 1.345 MB | 6ed2753b | FZK-LCG2_DATADISK: root://atlasxrootd-kit.gridka.de:1094//pnfs/gridka.de/atlas/disk-only/atlasdatadisk/rucio/mc23_13p6TeV/c5/61/TXT.33690939._000003.tar.gz.1 |
```
Misha said this is not an issue for prodsys because `rucio download` can be used from everywhere.
Apparently we can request special access for `mcgensvc` from mailto:atlas-adc-ddm-support@cern.ch
(we should keep Misha in cc)
We probably also need to request a new grid certificate for `mcgensvc`Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/120Allow runArgs to be referred to in JOs but not to be overwritten by JOs2020-08-22T13:08:09+02:00Christian GutschowAllow runArgs to be referred to in JOs but not to be overwritten by JOsSee !631 for an example.See !631 for an example.S2.2020Spyros ArgyropoulosSpyros Argyropoulos2020-08-14https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/159Allow run_athena to run 13p6 TeV2022-05-10T11:04:58+02:00Spyros ArgyropoulosAllow run_athena to run 13p6 TeVCurrently only `mc15_13TeV` and `mc16_13TeV` external LHE files are handled in run_athena. Should be extended to `mc16_13p6TeV` and `mc21_13p6TeV`.
Check if we can automate this so that we don't need to hard-code things as done now.Currently only `mc15_13TeV` and `mc16_13TeV` external LHE files are handled in run_athena. Should be extended to `mc16_13p6TeV` and `mc21_13p6TeV`.
Check if we can automate this so that we don't need to hard-code things as done now.Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/212Allow running `mc??_valid` input files2023-07-31T09:40:55+02:00Spyros ArgyropoulosAllow running `mc??_valid` input filesCurrently the CI cannot run on input files named like `mc15_valid.602232.Ph_ttj_MiNNLO_scale5_LHE.evgen.TXT.e8531/TXT.34045098._000002.tar.gz.1`
The `valid` replaces the COM energy so it is potentially a not well defined naming conventi...Currently the CI cannot run on input files named like `mc15_valid.602232.Ph_ttj_MiNNLO_scale5_LHE.evgen.TXT.e8531/TXT.34045098._000002.tar.gz.1`
The `valid` replaces the COM energy so it is potentially a not well defined naming convention, since the COM energy is taken automatically from `ecmEnergy` from `log.generate` which is [directly printed from the transform](https://gitlab.cern.ch/atlas/athena/-/blob/main/Generators/EvgenJobTransforms/share/skel.GENtoEVGEN.py#L105) - from the command line arguments.
One solution is to ignore the `ecmEnergy` completely if we see that the input filed is named as `mc??_valid` however I think that's problematic because an LHE should correspond to a given COM energy and we should not allow it to be used indepndently of the `ecmEnergy`.
@katharin @mgignac @dhirschSpyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/188Allow to skip Nfiles check in gridpack2022-09-06T20:27:53+02:00Spyros ArgyropoulosAllow to skip Nfiles check in gridpackThe following discussion from !2014 should be addressed:
- [ ] @narayan started a [discussion](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/2014#note_5975654): (+3 comments)
> Hi @sargyrop
>
> S...The following discussion from !2014 should be addressed:
- [ ] @narayan started a [discussion](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/2014#note_5975654): (+3 comments)
> Hi @sargyrop
>
> Shall I do ``[skip Athena]`` in order to get this merged ?
>
> Cheers
> RohinSpyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/204Apply basic tests also to fragments included from top jO2023-12-20T18:58:13+01:00Spyros ArgyropoulosApply basic tests also to fragments included from top jOVia check_jo_content probablyVia check_jo_content probablySpyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/174Arithmetic expression error in check_logParser.sh2022-05-17T16:23:22+02:00Spyros ArgyropoulosArithmetic expression error in check_logParser.shSee https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/21845100See https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/21845100https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/186Automatically merging MR from pipeline2022-08-25T09:35:17+02:00Spyros ArgyropoulosAutomatically merging MR from pipelineWhen running `merge_request_api.sh -m` from the pipeline we get a method not allowed because the pipeline is still running.
Need to fix this in order to enable an automatic merging - perhaps turn it into a "merge when pipeline succeeds...When running `merge_request_api.sh -m` from the pipeline we get a method not allowed because the pipeline is still running.
Need to fix this in order to enable an automatic merging - perhaps turn it into a "merge when pipeline succeeds"
See !2001Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/183Automatic creation of MR2022-08-23T08:37:22+02:00Spyros ArgyropoulosAutomatic creation of MR- [x] Create script that adds checks and creates MR
- [x] Approve MR
- [x] Merge open MR
- [x] Incorporate the script into CI
- [x] Add relevant parts in commit script- [x] Create script that adds checks and creates MR
- [x] Approve MR
- [x] Merge open MR
- [x] Incorporate the script into CI
- [x] Add relevant parts in commit scriptSpyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/40Automatic script not picking up links to GRID files2019-10-06T17:12:35+02:00Spyros ArgyropoulosAutomatic script not picking up links to GRID files!60!60Alpha