MC Job Options merge requestshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests2019-12-17T15:40:23+01:00https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/131Sherpa 2.2.8 V+gamma2019-12-17T15:40:23+01:00Frank SiegertSherpa 2.2.8 V+gamma## JIRA ticket
ATLMCPROD-8008
## Checklist for requesters
- [x] I have followed the [guidelines in McSampleRequestProcedure](https://twiki.cern.ch/twiki/bin/view/AtlasProtected/McSampleRequestProcedure)
- [x] I have provided th...## JIRA ticket
ATLMCPROD-8008
## Checklist for requesters
- [x] I have followed the [guidelines in McSampleRequestProcedure](https://twiki.cern.ch/twiki/bin/view/AtlasProtected/McSampleRequestProcedure)
- [x] I have provided the ATLMCPROD JIRA ticket where the request has been discussed
- [x] The request has been approved by the physics group
- [x] I have checked the "Squash commits when merge request is accepted" option below
- [x] I have checked the "Delete source branch when merge request is accepted." option below
## Checklist for approvers
- [x] All the items in the "Checklist for requesters" have been completed successfully
- [x] The pipeline has run and the status is green
- [x] CI jobs have not been skipped (if jobs have been skipped, confirmation from the PMG conveners is necessary before merging)
- [x] Look at the output of the `run_athena` pipeline and make sure that it has run **for all DSIDS added to the commit**. If it's not the case, confirmation from the PMG conveners is necessary before merging
- [x] Check that no `log.generate.short` files are included in the commit to be merged to master. If such files are present, it indicates that something went wrong in the pipeline. Check the job logs and contact the package maintainers if needed.https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/149Pythia8 JZ1 dijet slice with unweighting [skip modfiles],[skip ci]2019-12-17T16:37:30+01:00Simone AmorosoPythia8 JZ1 dijet slice with unweighting [skip modfiles],[skip ci]Pythia8 JO for JZ1W dijet slice with unweightingPythia8 JO for JZ1W dijet slice with unweightinghttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/150Add check for mcgensvc readability of eos directories [skip modfiles]2019-12-17T18:43:14+01:00Spyros ArgyropoulosAdd check for mcgensvc readability of eos directories [skip modfiles]Closes #56
Also remove some log.generate.short files
![Screenshot_2019-12-17_at_17.54.47](/uploads/65eb6f8230c31d4266b65fdcef34f9e2/Screenshot_2019-12-17_at_17.54.47.png)Closes #56
Also remove some log.generate.short files
![Screenshot_2019-12-17_at_17.54.47](/uploads/65eb6f8230c31d4266b65fdcef34f9e2/Screenshot_2019-12-17_at_17.54.47.png)BetaFrank SiegertFrank Siegerthttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/151Adding new DSID 5000212019-12-18T20:20:01+01:00Ewelina Maria LobodzinskaAdding new DSID 500021# Merge Request for the registration of new jobOption files that use existing base fragments
## JIRA ticket
Please provide here the link to the ATLMCPROD JIRA ticket where this request has been discussed (just replace XXX with your re...# Merge Request for the registration of new jobOption files that use existing base fragments
## JIRA ticket
Please provide here the link to the ATLMCPROD JIRA ticket where this request has been discussed (just replace XXX with your request number):
ATLMCPROD-XXX
## Checklist for requesters
- [ x] I have followed the [guidelines in McSampleRequestProcedure](https://twiki.cern.ch/twiki/bin/view/AtlasProtected/McSampleRequestProcedure)
- [x ] I have provided the ATLMCPROD JIRA ticket where the request has been discussed
- [ x] The request has been approved by the physics group
- [x ] I have checked the "Squash commits when merge request is accepted" option below
- [ x] I have checked the "Delete source branch when merge request is accepted." option below
## Checklist for approvers
- [x ] All the items in the "Checklist for requesters" have been completed successfully
- [ x] The pipeline has run and the status is green
- [ x] CI jobs have not been skipped (if jobs have been skipped, confirmation from the PMG conveners is necessary before merging)
- [x ] Look at the output of the `run_athena` pipeline and make sure that it has run **for all DSIDS added to the commit**. If it's not the case, confirmation from the PMG conveners is necessary before merging
- [x ] Check that no `log.generate.short` files are included in the commit to be merged to master. If such files are present, it indicates that something went wrong in the pipeline. Check the job logs and contact the package maintainers if needed.https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/147[skip ci] Update BlackList_caches.txt2019-12-19T12:42:50+01:00Ewelina Maria Lobodzinska[skip ci] Update BlackList_caches.txtupdate blacklist of generators/releasesupdate blacklist of generators/releaseshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/153Adding test JO for MadGraphControl2019-12-19T17:01:10+01:00Zach MarshallAdding test JO for MadGraphControlFollowing discussion at !29087 , adding test job options here. I've skipped a ton of the checklist, since this is for test job options; let me know if there's a different procedure we should follow!
Thanks,
Zach
## Checklist for reques...Following discussion at !29087 , adding test job options here. I've skipped a ton of the checklist, since this is for test job options; let me know if there's a different procedure we should follow!
Thanks,
Zach
## Checklist for requesters
- [?] I have followed the [guidelines in McSampleRequestProcedure](https://twiki.cern.ch/twiki/bin/view/AtlasProtected/McSampleRequestProcedure)
- [N/A] I have provided the ATLMCPROD JIRA ticket where the request has been discussed
- [X] The request has been approved by the physics group
- [X] I have checked the "Squash commits when merge request is accepted" option below
- [X] I have checked the "Delete source branch when merge request is accepted." option below
## Checklist for approvers
- [ ] All the items in the "Checklist for requesters" have been completed successfully
- [ ] The pipeline has run and the status is green
- [ ] CI jobs have not been skipped (if jobs have been skipped, confirmation from the PMG conveners is necessary before merging)
- [ ] Look at the output of the `run_athena` pipeline and make sure that it has run **for all DSIDS added to the commit**. If it's not the case, confirmation from the PMG conveners is necessary before merging
- [ ] Check that no `log.generate.short` files are included in the commit to be merged to master. If such files are present, it indicates that something went wrong in the pipeline. Check the job logs and contact the package maintainers if needed.https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/155Adding new DSID 5000252019-12-19T17:35:29+01:00Ewelina Maria LobodzinskaAdding new DSID 500025# Merge Request for the registration of new jobOption files that use existing base fragments
## JIRA ticket
Please provide here the link to the ATLMCPROD JIRA ticket where this request has been discussed (just replace XXX with your re...# Merge Request for the registration of new jobOption files that use existing base fragments
## JIRA ticket
Please provide here the link to the ATLMCPROD JIRA ticket where this request has been discussed (just replace XXX with your request number):
ATLMCPROD-8141
## Checklist for requesters
- [ ] I have followed the [guidelines in McSampleRequestProcedure](https://twiki.cern.ch/twiki/bin/view/AtlasProtected/McSampleRequestProcedure)
- [ ] I have provided the ATLMCPROD JIRA ticket where the request has been discussed
- [ ] The request has been approved by the physics group
- [ ] I have checked the "Squash commits when merge request is accepted" option below
- [ ] I have checked the "Delete source branch when merge request is accepted." option below
## Checklist for approvers
- [ ] All the items in the "Checklist for requesters" have been completed successfully
- [ ] The pipeline has run and the status is green
- [ ] CI jobs have not been skipped (if jobs have been skipped, confirmation from the PMG conveners is necessary before merging)
- [ ] Look at the output of the `run_athena` pipeline and make sure that it has run **for all DSIDS added to the commit**. If it's not the case, confirmation from the PMG conveners is necessary before merging
- [ ] Check that no `log.generate.short` files are included in the commit to be merged to master. If such files are present, it indicates that something went wrong in the pipeline. Check the job logs and contact the package maintainers if needed.https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/152Dsid ewelina 8300002019-12-19T18:00:07+01:00Ewelina Maria LobodzinskaDsid ewelina 830000# Merge Request for the registration of new jobOption files that use existing base fragments
## JIRA ticket
Please provide here the link to the ATLMCPROD JIRA ticket where this request has been discussed (just replace XXX with you...# Merge Request for the registration of new jobOption files that use existing base fragments
## JIRA ticket
Please provide here the link to the ATLMCPROD JIRA ticket where this request has been discussed (just replace XXX with your request number):
ATLMCPROD-8154
## Checklist for requesters
- [ ] I have followed the [guidelines in McSampleRequestProcedure](https://twiki.cern.ch/twiki/bin/view/AtlasProtected/McSampleRequestProcedure)
- [ ] I have provided the ATLMCPROD JIRA ticket where the request has been discussed
- [ ] The request has been approved by the physics group
- [ ] I have checked the "Squash commits when merge request is accepted" option below
- [ ] I have checked the "Delete source branch when merge request is accepted." option below
## Checklist for approvers
- [ ] All the items in the "Checklist for requesters" have been completed successfully
- [ ] The pipeline has run and the status is green
- [ ] CI jobs have not been skipped (if jobs have been skipped, confirmation from the PMG conveners is necessary before merging)
- [ ] Look at the output of the `run_athena` pipeline and make sure that it has run **for all DSIDS added to the commit**. If it's not the case, confirmation from the PMG conveners is necessary before merging
- [ ] Check that no `log.generate.short` files are included in the commit to be merged to master. If such files are present, it indicates that something went wrong in the pipeline. Check the job logs and contact the package maintainers if needed.https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/156Adding new DSID 700047-502019-12-20T10:29:41+01:00Christian GutschowAdding new DSID 700047-50# Merge Request for the registration of new jobOption files that use existing base fragments
## JIRA ticket
Please provide here the link to the ATLMCPROD JIRA ticket where this request has been discussed (just replace XXX with your re...# Merge Request for the registration of new jobOption files that use existing base fragments
## JIRA ticket
Please provide here the link to the ATLMCPROD JIRA ticket where this request has been discussed (just replace XXX with your request number):
ATLMCPROD-8197
## Checklist for requesters
- [x] I have followed the [guidelines in McSampleRequestProcedure](https://twiki.cern.ch/twiki/bin/view/AtlasProtected/McSampleRequestProcedure)
- [x] I have provided the ATLMCPROD JIRA ticket where the request has been discussed
- [x] The request has been approved by the physics group
- [x] I have checked the "Squash commits when merge request is accepted" option below
- [x] I have checked the "Delete source branch when merge request is accepted." option below
## Checklist for approvers
- [x] All the items in the "Checklist for requesters" have been completed successfully
- [x] The pipeline has run and the status is green
- [x] CI jobs have not been skipped (if jobs have been skipped, confirmation from the PMG conveners is necessary before merging)
- [x] Look at the output of the `run_athena` pipeline and make sure that it has run **for all DSIDS added to the commit**. If it's not the case, confirmation from the PMG conveners is necessary before merging
- [x] Check that no `log.generate.short` files are included in the commit to be merged to master. If such files are present, it indicates that something went wrong in the pipeline. Check the job logs and contact the package maintainers if needed.https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/159Updates to get more tests running2019-12-20T14:46:45+01:00Zach MarshallUpdates to get more tests runningThese are job options updates to get some more tests running. I'm trying the "no template" option as well...
[skip ci]These are job options updates to get some more tests running. I'm trying the "no template" option as well...
[skip ci]https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/160Unify whitelist in commit script and in CI [skip modfiles]2019-12-21T15:02:14+01:00Spyros ArgyropoulosUnify whitelist in commit script and in CI [skip modfiles]`check_added_files.sh` contained 2 `checkWhiteList` functions, one from `checkWhitelist.sh` and a local implementation. I remove the local implementation so that we only have 1 definition in the whole package.`check_added_files.sh` contained 2 `checkWhiteList` functions, one from `checkWhitelist.sh` and a local implementation. I remove the local implementation so that we only have 1 definition in the whole package.BetaFrank SiegertChristian GutschowFrank Siegerthttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/161Compare to origin/master in CI jobs [skip modfiles]2019-12-21T15:10:24+01:00Spyros ArgyropoulosCompare to origin/master in CI jobs [skip modfiles]* When CI job is launched always compare the `HEAD` of that branch against `origin/master`
* Fix bug in `check_unique_physicsShort` - we had forgotten to change `mc??` to `mc` when we switched from `mc16` to `mc` for the jO naming so t...* When CI job is launched always compare the `HEAD` of that branch against `origin/master`
* Fix bug in `check_unique_physicsShort` - we had forgotten to change `mc??` to `mc` when we switched from `mc16` to `mc` for the jO naming so this test has been inactive so far
!160 needs to be merged first hence the WIP
Closes #54BetaFrank SiegertChristian GutschowFrank Siegerthttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/163Fixing whitelist [skip modfiles]2019-12-21T18:50:16+01:00Spyros ArgyropoulosFixing whitelist [skip modfiles]Fixing broken CI in !162
From my test:
(base) MacBook-Pro-5:mcjoboptions_3 sargyrop$ ./scripts/check_added_files.sh
Find files that have been added in last commit...
OK: file: 700xxx/700051/Sherpa_i/NNPDF30NNLO_nf_4.py in white list
...Fixing broken CI in !162
From my test:
(base) MacBook-Pro-5:mcjoboptions_3 sargyrop$ ./scripts/check_added_files.sh
Find files that have been added in last commit...
OK: file: 700xxx/700051/Sherpa_i/NNPDF30NNLO_nf_4.py in white list
OK: file: 700xxx/700051/mc.Sh_228_ttbb_AllHadronic.py in white list
OK: file: 700xxx/700051/mc_13TeV.Sh_228_ttbb.GRID.tar.gz in white list
OK: file: 700xxx/700052/Sherpa_i/NNPDF30NNLO_nf_4.py in white list
OK: file: 700xxx/700052/mc.Sh_228_ttbb_SingleLeptonP.py in white list
OK: file: 700xxx/700052/mc_13TeV.Sh_228_ttbb.GRID.tar.gz in white list
OK: file: 700xxx/700053/Sherpa_i/NNPDF30NNLO_nf_4.py in white list
OK: file: 700xxx/700053/mc.Sh_228_ttbb_SingleLeptonM.py in white list
OK: file: 700xxx/700053/mc_13TeV.Sh_228_ttbb.GRID.tar.gz in white list
OK: file: 700xxx/700054/Sherpa_i/NNPDF30NNLO_nf_4.py in white list
OK: file: 700xxx/700054/mc.Sh_228_ttbb_dilepton.py in white list
OK: file: 700xxx/700054/mc_13TeV.Sh_228_ttbb.GRID.tar.gz in white listFrank SiegertFrank Siegerthttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/162Adding new DSID 7000512019-12-21T22:49:02+01:00Frank SiegertAdding new DSID 700051## JIRA ticket
ATLMCPROD-8199
## Checklist for requesters
- [x] I have followed the [guidelines in McSampleRequestProcedure](https://twiki.cern.ch/twiki/bin/view/AtlasProtected/McSampleRequestProcedure)
- [x] I have provided th...## JIRA ticket
ATLMCPROD-8199
## Checklist for requesters
- [x] I have followed the [guidelines in McSampleRequestProcedure](https://twiki.cern.ch/twiki/bin/view/AtlasProtected/McSampleRequestProcedure)
- [x] I have provided the ATLMCPROD JIRA ticket where the request has been discussed
- [x] The request has been approved by the physics group
- [x] I have checked the "Squash commits when merge request is accepted" option below
- [x] I have checked the "Delete source branch when merge request is accepted." option below
## Checklist for approvers
- [x] All the items in the "Checklist for requesters" have been completed successfully
- [x] The pipeline has run and the status is green
- [x] CI jobs have not been skipped (if jobs have been skipped, confirmation from the PMG conveners is necessary before merging)
- [x] Look at the output of the `run_athena` pipeline and make sure that it has run **for all DSIDS added to the commit**. If it's not the case, confirmation from the PMG conveners is necessary before merging
- [x] Check that no `log.generate.short` files are included in the commit to be merged to master. If such files are present, it indicates that something went wrong in the pipeline. Check the job logs and contact the package maintainers if needed.https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/165Support DSID ranges in commit script [skip modfiles]2019-12-23T20:34:06+01:00Spyros ArgyropoulosSupport DSID ranges in commit script [skip modfiles]Closes #58Closes #58Frank SiegertChristian GutschowFrank Siegerthttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/168Fix calls to generator specific checks [skip modfiles]2019-12-25T12:20:26+01:00Spyros ArgyropoulosFix calls to generator specific checks [skip modfiles]Copy of https://gitlab.cern.ch/atlas-physics/pmg/tools/logParser/merge_requests/12
The calls to the generator checks were implemented with `if` `else` statements, which meant that for e.g. MG+Pythia8 only the Pythia8 checks would be ...Copy of https://gitlab.cern.ch/atlas-physics/pmg/tools/logParser/merge_requests/12
The calls to the generator checks were implemented with `if` `else` statements, which meant that for e.g. MG+Pythia8 only the Pythia8 checks would be performed.
We have not run into this since there is so far no checks for MG but we would run into it in the near future. #43 Frank SiegertChristian GutschowFrank Siegerthttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/166Introduce timing information in logParser [skip modfiles]2019-12-25T13:00:59+01:00Spyros ArgyropoulosIntroduce timing information in logParser [skip modfiles]This MR introduces timing information according to what was discussed in #45 #51 and will allow to reactivate the athena CI job.
## Handling of `CountHepMC == nEventsPerJob`
The **check that `CountHepMC == nEventsPerJob` has...This MR introduces timing information according to what was discussed in #45 #51 and will allow to reactivate the athena CI job.
## Handling of `CountHepMC == nEventsPerJob`
The **check that `CountHepMC == nEventsPerJob` has not been removed**, because I think it might still be a good handle to catch unwanted bugs. Instead I added a flag in the commit script that can bypass this check. Here is what happens when trying to add `421305`, run with `--maxEvents=10`:
```
mcjoboptions_3 sargyrop$ ./scripts/commit_new_dsid.sh -d=421305 -n
INFO: will use following remote for pushing: origin
New DSID directory: 421xxx/421305 ...
OK: log.generate file found.
...
- CountHepMC Events passing all checks and written = 10 <-- ERROR: This is not equal to nEventsPerJob=1000
...
---------------------
Summary:
---------------------
Errors : 1 , Warnings : 5 -> Errors encountered! Not ready for production!
ERROR: log.generate contains errors
Fix them before committing anything!
```
In this case users should **make sure that disabling the `CountHepMC` check is what they really want** and run `commit_new_dsid.sh` with `-t`:
```
./scripts/commit_new_dsid.sh -d=421305 -t -n
INFO: will use following remote for pushing: origin
New DSID directory: 421xxx/421305 ...
OK: log.generate file found.
OK: log.generate file contains no errors
OK: CI job expected to last less than 1h - time estimate: 0.17 hours
Will now add files to git commit
File: log.generate cannot be added to the commit. Skipping.
Skipping: log.generate.short since the logParser CI job will not be run
Added: mc.Sh_228_Wmunu_EnhLogPtV.py
Added: mc_13TeV.Sh_228_Wmunu_EnhLogPtV.GRID.tar.gz
```
## How timing works in logParser
There are 3 timing informations provided:
1. **if `CountHepMC == nEventsPerJob`**: it just prints the timing information from log.generate (same as what we had before)
2. **if `CountHepMC != nEventsPerJob`**: it will extrapolate the timing information using `extrapCPU=float(generateDict['nEventsPerJob'])*cpuPerJob/float(CountHepMC)`
3. **timing for CI job**: this is something extra to help debugging and have more meaningful printouts, it will just extrapolate the timing information to `max(10,0.01*nEventsPerJob)`, i.e. the number of events we will run in the CI
## Timing limits
The actual (case 1) or extrapolated CPU time (case 2 above) lead to the following behaviour:
* **CPU < 1 h OR CPU > 18 h** => ERROR
* **CPU between 8-12 h** => GOOD
* **CPU=1-8 OR 12-18 h** => WARNING
i.e. currently the limits are quite loose and we would only cut jobs which last less than 1h or more than 18h.
For the CI (case 3) above we cut everything that is expected to last more than 1h.
Here is an example from `logParser` run on the above 421305 DSID with 10 events (nEventsPerJob=1000 in this case):
```
---------------------
Performance metrics:
---------------------
- actual CPU (10 events) = 0.17 hrs
- CPU extrapolated to 1000 events = 16.6 hrs <-- WARNING: CPU time not optimal - should be between 8-12h. Adjust nEventsPerJob!
- estimated CPU for CI job = 0.17 hrs
- Virtual memory = 2957.352 Mb
```
Notice that the first line is the actual time it took to run the job (with `--maxEvents=10`), the 2nd is the extrapolation to `nEventsPerJob=1000` and the 3rd is the extrapolation for the `max(10,0.01*1000)=10` events that would run in the CI.
Note that `logParser` in CI is always run with `-t` so there should be no extra change needed in order to re-activate the athena CI jobs.
Closes #51
I think this also closes #15 BetaFrank SiegertChristian GutschowFrank Siegerthttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/167Updating minimum number of events in run_athena and testing only 1 DSID [skip...2019-12-25T14:48:10+01:00Spyros ArgyropoulosUpdating minimum number of events in run_athena and testing only 1 DSID [skip modfiles]Two changes introduced in `run_athena`:
* Only run a single DSID for the moment (until we are confident that enough jobs succeed). Then we can increase this number to be equal to the number of committed DSIDs or a maximum number of DSI...Two changes introduced in `run_athena`:
* Only run a single DSID for the moment (until we are confident that enough jobs succeed). Then we can increase this number to be equal to the number of committed DSIDs or a maximum number of DSIDs allowed by timing information
* Increase the minimum number of events to run in `run_athena` to 10. This was necessary since with 1 event several Sherpa jobs were failing the `logParser` checks due to negative cross-sections
Closes #45BetaFrank SiegertChristian GutschowFrank Siegerthttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/164Change acceptable number of output events in logParser2020-01-02T12:20:51+01:00Spyros ArgyropoulosChange acceptable number of output events in logParserAccording to the discussion we had with @fsiegert @cgutscho I change the acceptable number of events (written in the EVNT file to): `1, 2, 5, 10, 20, 50, 100, 200, 500, 1000, 2000, 5000, 10000`.
Also the timing checks are changed to t...According to the discussion we had with @fsiegert @cgutscho I change the acceptable number of events (written in the EVNT file to): `1, 2, 5, 10, 20, 50, 100, 200, 500, 1000, 2000, 5000, 10000`.
Also the timing checks are changed to that if we use 10k events (written in the EVNT) and the job is very fast, there is no error thrown.
This will be supplemented in a following MR with timing checks as discussed in #51 #45BetaFrank SiegertChristian GutschowFrank Siegerthttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/172Updates for MadGraph tests [skip modfiles]2020-01-06T13:41:29+01:00Zach MarshallUpdates for MadGraph tests [skip modfiles]This updates the final JO for madgraph tests for the time beingThis updates the final JO for madgraph tests for the time being