MC Job Options issueshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues2020-01-15T13:14:35+01:00https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/60Fix behaviour of pipelines when [skip ci] is used2020-01-15T13:14:35+01:00Spyros ArgyropoulosFix behaviour of pipelines when [skip ci] is usedApparently there has been a change in the gitlab policy where, if we enforce that pipelines must succeed there must always be a CI job that runs successfully, so we might have to implement something like https://docs.gitlab.com/ee/user/p...Apparently there has been a change in the gitlab policy where, if we enforce that pipelines must succeed there must always be a CI job that runs successfully, so we might have to implement something like https://docs.gitlab.com/ee/user/project/merge_requests/merge_when_pipeline_succeeds.html (see Limitations).
https://gitlab.cern.ch/help/user/project/merge_requests/merge_when_pipeline_succeeds.md#only-allow-merge-requests-to-be-merged-if-the-pipeline-succeeds
Associated gitlab issues describing the policy change:
https://gitlab.com/gitlab-org/gitlab/issues/14791
https://gitlab.com/gitlab-org/gitlab-foss/issues/66271
Affects !180 see also !184BetaSpyros ArgyropoulosSpyros Argyropoulos2020-01-17https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/61Potential harmonisation of evgen transform checks with CI2020-03-20T09:37:59+01:00Spyros ArgyropoulosPotential harmonisation of evgen transform checks with CIThe checks implemented here: https://gitlab.cern.ch/atlas/athena/blob/21.6/Generators/EvgenJobTransforms/share/skel.GENtoEVGEN.py
are supposed to be the same as the checks we use in the CI: https://gitlab.cern.ch/atlas-physics/pmg/mcjob...The checks implemented here: https://gitlab.cern.ch/atlas/athena/blob/21.6/Generators/EvgenJobTransforms/share/skel.GENtoEVGEN.py
are supposed to be the same as the checks we use in the CI: https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/blob/master/scripts/check_jo_consistency.py, however given that they are coded in 2 completely independent parts they are bound to go out of sync.
We should see if there's a way to tie them together so that they never go out of sync.
Carrying over from #2S1.2020https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/62Use 1 event as minimum in `run_athena`2020-01-22T10:03:04+01:00Spyros ArgyropoulosUse 1 event as minimum in `run_athena`When xsec check is downgraded to WARNING (#59) we should switch back to using 1 event as minimum in `run_athena` instead of 10.When xsec check is downgraded to WARNING (#59) we should switch back to using 1 event as minimum in `run_athena` instead of 10.S1.2020Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/63logParser: document warnings and errors2022-03-12T09:24:59+01:00Spyros ArgyropouloslogParser: document warnings and errorsI've been thinking that it would help with the approval process to document somewhere (I'd say in the README - to keep everything together) all the cases where `logParser` throws WARNING or ERROR, so that approvers can easily skim throug...I've been thinking that it would help with the approval process to document somewhere (I'd say in the README - to keep everything together) all the cases where `logParser` throws WARNING or ERROR, so that approvers can easily skim through the CI job output and understand if something should be followed up on.Futurehttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/64Allow CI to run with external LHE inputs2021-04-27T12:14:25+02:00Spyros ArgyropoulosAllow CI to run with external LHE inputsThe CI can currently not yet run jobs with input event files. Largely the same tasks appear for both LHE->EVNT and EVNT->EVNT files:
* [x] Upload mcgensvc grid certificate in CI and make sure `voms-proxy-init -voms atlas` works in the CI...The CI can currently not yet run jobs with input event files. Largely the same tasks appear for both LHE->EVNT and EVNT->EVNT files:
* [x] Upload mcgensvc grid certificate in CI and make sure `voms-proxy-init -voms atlas` works in the CI
* [x] Write (rucio) file that was used in local testing into `log.generate.short`, e.g. `inputGeneratorFile=TXT.<dsid>.tar.gz` for LHE or `inputEVNTFile=1231231.EVNT.pool.root` for EVNT input.
* [x] If input file is specified in `log.generate.short`, the CI `run_athena.sh` job should `rucio get` that file and add the corresponding arguments to the `Gen_tf.py` command line as described in [Twiki](https://twiki.cern.ch/twiki/bin/view/AtlasProtected/SpecialConfigurations#Using_event_input_LHE_EVNT_or_EV).
* [x] Special treatment for EVNT->EVNT jobs: Apparently such jobs do not produce a `log.generate` file but a `log.afterburn`. Disable logParser for these, or special treatment? (move to #139)
* [ ] Ideally, the CI should also check the `inputFilesPerJob` in the JO for reasonable values at this stage: it should not exceed `10GB/sizeof(downloadedTestFile)` to make sure they fit into a grid node.
* [x] Similar case that came up in #89 is running athena with `--inputGeneratorFile`
Example !203
LHE-only log: `~sargyrop/public/log.generate_LHE`S1.2021Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/65Running pipelines in custom swarm runner2021-06-17T13:04:50+02:00Spyros ArgyropoulosRunning pipelines in custom swarm runnerInstructions from Lukas here: https://clouddocs.web.cern.ch/containers/tutorials/swarmgitlab.html
Idea would be that if we set this up we could run with the full number of events exactly as in production.
The instructions below work. ...Instructions from Lukas here: https://clouddocs.web.cern.ch/containers/tutorials/swarmgitlab.html
Idea would be that if we set this up we could run with the full number of events exactly as in production.
The instructions below work. Wo have to understand whether this is what we want:
* [ ] does it have access to cvmfs? If not how would we set it up so that it has?
* [ ] does it buy us anything from using the shared runners?
* [ ] how tough would the maintenance be?
* [ ] is it better to just set up a dedicated machine? Maybe we should ask someone from the CERN IT to do it?Futurehttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/66Modify logParser/CI to handle cases with no TestHepMC results2020-08-17T10:40:40+02:00Spyros ArgyropoulosModify logParser/CI to handle cases with no TestHepMC resultsAs mentioned here https://its.cern.ch/jira/browse/ATLHI-297 there might be cases where the requirement of TestHepMC results in logParser blocks a production.
@olszewsk I would need a log.generate file to provide a solutionAs mentioned here https://its.cern.ch/jira/browse/ATLHI-297 there might be cases where the requirement of TestHepMC results in logParser blocks a production.
@olszewsk I would need a log.generate file to provide a solutionS2.2020Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/67Example Job Options are missing2020-01-28T17:53:01+01:00Jonas NeundorfExample Job Options are missingHi all,
I am following the MadGraph examples from this twiki [1], which contains some links to example job options, e.g. [2]. When I try to access them, I get the error message `"421xxx/421402/mc.MG_ttbar_LHAPDFTest.py" did not exist on...Hi all,
I am following the MadGraph examples from this twiki [1], which contains some links to example job options, e.g. [2]. When I try to access them, I get the error message `"421xxx/421402/mc.MG_ttbar_LHAPDFTest.py" did not exist on "master"`. This seems to be true for all of the MadGraph examples (DSID starting afaik with 4214). I am pretty sure they were still accessible last week.
Please either restore them or update the twiki with the new links.
Best regards,
Jonas Neundorf
[1] https://twiki.cern.ch/twiki/bin/view/AtlasProtected/MadGraph5aMCatNLOForAtlas
[2] https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/blob/master/421xxx/421402/mc.MG_ttbar_LHAPDFTest.pyZach MarshallZach Marshallhttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/68Clearer commit messages from commit script2020-02-06T14:17:11+01:00Spyros ArgyropoulosClearer commit messages from commit scriptThe complaint was that the default message from the commit script `"Adding DSID ..."` doesn't help when browsing the jO files.
We could make the `-m` option obligatory (now it's optional) and then it would be up to the user to use usef...The complaint was that the default message from the commit script `"Adding DSID ..."` doesn't help when browsing the jO files.
We could make the `-m` option obligatory (now it's optional) and then it would be up to the user to use useful messages.
In MRs with multiple commits this is also now easily fixed by changing the commit message in the web interface before merging.
�Tagging @fsiegertS1.2020Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/70Extending whitelist to cover parameter cards and other common includes2020-02-12T13:58:13+01:00Spyros ArgyropoulosExtending whitelist to cover parameter cards and other common includesIssue arose in !214
This case had files like:
```
500004/MadGraphControl_*.py
500004/MadGraph*.dat
500004/mc.*.py
```
for Sherpa, in the cases where you were adding python files these were like this `500005/Sherpa_i/*.py`
Should we al...Issue arose in !214
This case had files like:
```
500004/MadGraphControl_*.py
500004/MadGraph*.dat
500004/mc.*.py
```
for Sherpa, in the cases where you were adding python files these were like this `500005/Sherpa_i/*.py`
Should we allow both? And also allow any dat files inside the DSID directory?
## Things to add in whitelist
* [x] `common/SUSYControl/*.py`
* [x] `common/SUSYControl/*.dat`
Tag @amoroso @cgutscho @fsiegertS1.2020Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/71Propagation of generator names from jO to skeleton and to log.generate2020-02-04T05:47:14+01:00Spyros ArgyropoulosPropagation of generator names from jO to skeleton and to log.generateThere was some confusion about how generator names are propagated from the jO file to `log.generate` and what `logParser` uses in the end.
looking at log.generate from a MG sample:
```
11:32:32 MetaData: generatorName = MadGraph(v.2.6...There was some confusion about how generator names are propagated from the jO file to `log.generate` and what `logParser` uses in the end.
looking at log.generate from a MG sample:
```
11:32:32 MetaData: generatorName = MadGraph(v.2.6.5)+Pythia8(v.240)+EvtGen(v.1.6.0)
```
so I think it’s the non abbreviated names there, because I think we get these directly from the interfaces,
e.g. for P8 I think we get it from here: https://gitlab.cern.ch/atlas/athena/blob/21.6/Generators/Pythia8_i/share/common/Pythia8_Base_Fragment.py#L4
BTW I am thinking now: what if someone puts
```
evgenConfig.generators = [“Unknown”]
```
I just did that and `logParser` doesn’t complain at all.
So I also need to understand whether my assumption is correct that there is actually no control over what goes into
MetaData: generatorName
and we completley rely on what the user puts in evgenConfig.generators, which might be nonsense.
If that’s correct then we have to devise a test to catch things which are not allowed or enforce it in the skeleton?
* [ ] try to run one of the jO but giving an invalid name => does the transform complain? If yes then nothing more to do?
* [ ] if the transform doesn't complain we should make it so that it complains
* [ ] why is `MetaData: generatorName` not the abbreviated form?S1.2020https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/72CI: how to check for modification of files2020-02-04T11:55:18+01:00Spyros ArgyropoulosCI: how to check for modification of filesShould we restrict `check_modified_file.sh` to only look for changes in `common` and DSID directories?
## Pros
* easier to use for developers (they don't have to commit with `[skip modfiles]`)
## Cons
* More unsafe (e.g. commit of jO...Should we restrict `check_modified_file.sh` to only look for changes in `common` and DSID directories?
## Pros
* easier to use for developers (they don't have to commit with `[skip modfiles]`)
## Cons
* More unsafe (e.g. commit of jO with high priority comes in with checks in scripts that have been commented out due to failures, gets merged in master and as a result the checks are disabled for everyone)S1.2020Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/73Read nEventsPerJob from JO instead of logfile2020-03-04T10:14:12+01:00Frank SiegertRead nEventsPerJob from JO instead of logfileCurrently the logParser checks whether `nEventsPerJob` is set correctly to fulfil the runtime limits of the grid job. It uses the `nEventsPerJob` written out to the `log.generate`, but that might not be the correct one in cases where `nE...Currently the logParser checks whether `nEventsPerJob` is set correctly to fulfil the runtime limits of the grid job. It uses the `nEventsPerJob` written out to the `log.generate`, but that might not be the correct one in cases where `nEventsPerJob` had to be changed after the test job. Instead the value in the JO should be used.
These checks should probably be removed from the logParser and moved into an individual check of the JO (together with the `log.generate.short`?).
I'm attaching one [log.generate](/uploads/467a185ec3c90313d200154de32fd980/log.generate) file for a [recent merge request (DSID 700008)](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/merge_requests/222) which without modifications leads to:
```
- CPU = 0.88 hrs <-- ERROR: Too low CPU time - should be between 6-12h. Adjust nEventsPerJob!
```S1.2020Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/74Remove TestHepMC efficiency checks from logParser since it's already tested i...2020-02-07T14:34:15+01:00Andrzej OlszewskiRemove TestHepMC efficiency checks from logParser since it's already tested in athenaHi @sargyrop ,
as described in https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/commit/f184726125dc94bf9f3cc85a71494f077914c0fb TestHepMC checks on athena run on small number of events issue error while on a full stat run only a wa...Hi @sargyrop ,
as described in https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/commit/f184726125dc94bf9f3cc85a71494f077914c0fb TestHepMC checks on athena run on small number of events issue error while on a full stat run only a warning. Please see log in e.g.:
/afs/cern.ch/user/o/olszewsk/workspace/public/evgen/tests/800022PbPb5TeV/.
Maybe check is too restrictive? Maybe athena check in git could be improved?
Cheers, AndrzejS1.2020Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/75Blacklisting jO control files2020-02-13T14:14:54+01:00Spyros ArgyropoulosBlacklisting jO control filesAs discussed today we would want a way to identify control files that should not be used for physics.
This could be implemented by adding an identifier in the control files like
`evgenConfig.obsolete`
that could be identified by `...As discussed today we would want a way to identify control files that should not be used for physics.
This could be implemented by adding an identifier in the control files like
`evgenConfig.obsolete`
that could be identified by `Gen_tf` and/or `logParser` (I think the first is better because no point to waste CPU for an obsolete job) and the transform can throw a `FATAL` (or `logParser` would return an `ERROR`).
@ewelina would that be possible? (if you are busy I could try to implement it in Gen_tf, just to learn how to do such things :wink:)
Also @amoroso @cgutscho @fsiegertS1.2020Ewelina Maria LobodzinskaEwelina Maria Lobodzinskahttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/76Add dat files in DSID directories in whitelist2020-02-21T10:51:10+01:00Spyros ArgyropoulosAdd dat files in DSID directories in whitelistShould add `*.dat` in whitelistShould add `*.dat` in whitelistS1.2020Spyros ArgyropoulosSpyros Argyropoulos2020-02-15https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/77Remove 95*xxx from check_modified_files2020-02-23T10:34:18+01:00Spyros ArgyropoulosRemove 95*xxx from check_modified_filesRemove 95*xxx from check_modified_filesRemove 95*xxx from check_modified_filesS1.2020Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/78commit script broken2020-02-22T16:06:55+01:00Christian Gutschowcommit script brokenLook like the recent MR !252 broke the commit script, since it's doesn't accept JOs and gridpacks apparently:
```
Checking jO consistency and DSID ranges ...
950xxx/950031/mc.Sh_228_ttbar_dilepton_FusingFragFix_valid.py in correct DSI...Look like the recent MR !252 broke the commit script, since it's doesn't accept JOs and gridpacks apparently:
```
Checking jO consistency and DSID ranges ...
950xxx/950031/mc.Sh_228_ttbar_dilepton_FusingFragFix_valid.py in correct DSID range
New DSID directory: 950xxx/950031 ...
OK: log.generate file found.
OK: log.generate file contains no errors
OK: CI job expected to last less than 1h - time estimate: 0.05 hours
Will now add files to git commit
File: log.generate cannot be added to the commit. Skipping.
File: log.generate.short cannot be added to the commit. Skipping.
File: mc_13TeV.Sh_228_ttbar_AllHadronic_EnhMaxHTavrgTopPT.GRID.tar.gz cannot be added to the commit. Skipping.
File: mc.Sh_228_ttbar_dilepton_FusingFragFix_valid.py cannot be added to the commit. Skipping.
Added: Sherpa_i
```
This was found in !256.S1.2020Spyros ArgyropoulosSpyros Argyropoulos2020-02-22https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/79Recursive extraction of GRID links2020-02-24T10:11:18+01:00Spyros ArgyropoulosRecursive extraction of GRID linksThis is causing the failure in !256.
It seems that the scripts are for some reason not using enough steps to resolve
```
950xxx/950031/mc_13TeV.Sh_228_ttbar_AllHadronic_EnhMaxHTavrgTopPT.GRID.tar.gz -> ../../700xxx/700050/mc_13TeV.Sh_2...This is causing the failure in !256.
It seems that the scripts are for some reason not using enough steps to resolve
```
950xxx/950031/mc_13TeV.Sh_228_ttbar_AllHadronic_EnhMaxHTavrgTopPT.GRID.tar.gz -> ../../700xxx/700050/mc_13TeV.Sh_228_ttbar_AllHadronic_EnhMaxHTavrgTopPT.GRID.tar.gz
```
back to
```
/eos/user/c/cgutscho/mc/700047/mc_13TeV.Sh_228_ttbar_AllHadronic_EnhMaxHTavrgTopPT.GRID.tar.gz
```
By the way @cgutscho @fsiegert isn't this workflow problematic? You have multiple files pointing to `/eos/user/c/cgutscho/mc/700047/mc_13TeV.Sh_228_ttbar_AllHadronic_EnhMaxHTavrgTopPT.GRID.tar.gz`, which has already been transferred to cvmfs, so in principle Chris can decide to remove this file from his eos area and CI again would fail.S1.2020Spyros ArgyropoulosSpyros Argyropoulos2020-02-22https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/80Improve following of recursive links in eos and run_athena CI jobs2020-03-01T09:26:02+01:00Spyros ArgyropoulosImprove following of recursive links in eos and run_athena CI jobsSuggestion from Frank: use `readlink -f`
Need to see if this is accessible in the CI bash version or perhaps find a bash version where it is (if it's small enough).Suggestion from Frank: use `readlink -f`
Need to see if this is accessible in the CI bash version or perhaps find a bash version where it is (if it's small enough).S1.2020Spyros ArgyropoulosSpyros Argyropoulos2020-03-01