MC Job Options merge requestshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests2019-11-04T11:44:18+01:00https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/99Patch 72019-11-04T11:44:18+01:00Ewelina Maria LobodzinskaPatch 7correct order of generators in GeneratorsList to avoid crashes for Pythia8B,
correct Sherpa ART testscorrect order of generators in GeneratorsList to avoid crashes for Pythia8B,
correct Sherpa ART testshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/114Store log.generate also for failed CI jobs so that we can debug [skip modfiles]2019-11-23T14:50:58+01:00Spyros ArgyropoulosStore log.generate also for failed CI jobs so that we can debug [skip modfiles]* Increase storage time to 1 week
* Store not only successful `run_athena` output (`log.generate_ci`) but also unsuccessful one (`log.generate`) so that we can debug* Increase storage time to 1 week
* Store not only successful `run_athena` output (`log.generate_ci`) but also unsuccessful one (`log.generate`) so that we can debugSimone AmorosoFrank SiegertChristian GutschowSimone Amorosohttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/126switched off QED ISR from Pythia82019-11-30T19:31:59+01:00Simone Amorososwitched off QED ISR from Pythia8Switched off QED ISR in Pythia as need a proper veto to avoid double counting with the Powheg emissions.Switched off QED ISR in Pythia as need a proper veto to avoid double counting with the Powheg emissions.https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/106WIP: Test with docker image2019-12-03T10:14:36+01:00Mukesh KumarWIP: Test with docker imageWIP: testing CI with docker image
* [ ] Remove anything not associated to the changes in the CI (it seems like only `.gitlab-ci.yml` should stay)
* [x] Probably the correct execution relies on cvmfs being mounted in the gitlab runn...WIP: testing CI with docker image
* [ ] Remove anything not associated to the changes in the CI (it seems like only `.gitlab-ci.yml` should stay)
* [x] Probably the correct execution relies on cvmfs being mounted in the gitlab runner. Check whether we need to use runners with cvmfs tag https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/settings/ci_cd (**we do**)
* [ ] Check if we need to use `atlas_external_cvmfs` image or if it's enough to use `tags:cvmfs`
* [ ] Check whether we need to do anything specific for GRID files that live on eos. Do we need the eos image to run the job?
* [ ] Build docker image from `atlas_external_cvmfs` and gitlab-registry.cern.ch/atlas-physics/pmg/mcjoboptions:eos_bash
* [ ] Check whether slc6 or cc7 is used (most probably it's the former but good to check - then #33 can be closed)
* [ ] Use this new image to test with
* [ ] 421001 (simple jO)
* [ ] 421003 (contains GRID file)
**Use only 10 events for testing**Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/133WIP: Test with docker image2019-12-05T09:15:02+01:00Mukesh KumarWIP: Test with docker imageChecks done with image: atlas/slc6-atlasosChecks done with image: atlas/slc6-atlasosSpyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/154Dsid ewelina 5000252019-12-19T17:24:43+01:00Ewelina Maria LobodzinskaDsid ewelina 500025# Merge Request for the registration of new jobOption files that use existing base fragments
## JIRA ticket
Please provide here the link to the ATLMCPROD JIRA ticket where this request has been discussed (just replace XXX with you...# Merge Request for the registration of new jobOption files that use existing base fragments
## JIRA ticket
Please provide here the link to the ATLMCPROD JIRA ticket where this request has been discussed (just replace XXX with your request number):
ATLMCPROD-8141
## Checklist for requesters
- [ x] I have followed the [guidelines in McSampleRequestProcedure](https://twiki.cern.ch/twiki/bin/view/AtlasProtected/McSampleRequestProcedure)
- [x ] I have provided the ATLMCPROD JIRA ticket where the request has been discussed
- [x ] The request has been approved by the physics group
- [ x] I have checked the "Squash commits when merge request is accepted" option below
- [ x] I have checked the "Delete source branch when merge request is accepted." option below
## Checklist for approvers
- [ ] All the items in the "Checklist for requesters" have been completed successfully
- [ ] The pipeline has run and the status is green
- [ ] CI jobs have not been skipped (if jobs have been skipped, confirmation from the PMG conveners is necessary before merging)
- [ ] Look at the output of the `run_athena` pipeline and make sure that it has run **for all DSIDS added to the commit**. If it's not the case, confirmation from the PMG conveners is necessary before merging
- [ ] Check that no `log.generate.short` files are included in the commit to be merged to master. If such files are present, it indicates that something went wrong in the pipeline. Check the job logs and contact the package maintainers if needed.https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/158Updates to get more tests running2019-12-20T13:13:36+01:00Zach MarshallUpdates to get more tests runningBunch more updates to get a handful of tests running. This complements !29087 in Athena.
## Checklist for requesters
- [X] I have followed the [guidelines in McSampleRequestProcedure](https://twiki.cern.ch/twiki/bin/view/AtlasProtected...Bunch more updates to get a handful of tests running. This complements !29087 in Athena.
## Checklist for requesters
- [X] I have followed the [guidelines in McSampleRequestProcedure](https://twiki.cern.ch/twiki/bin/view/AtlasProtected/McSampleRequestProcedure)
- [X] I have provided the ATLMCPROD JIRA ticket where the request has been discussed
- [X] The request has been approved by the physics group
- [X] I have checked the "Squash commits when merge request is accepted" option below
- [X] I have checked the "Delete source branch when merge request is accepted." option below
## Checklist for approvers
- [ ] All the items in the "Checklist for requesters" have been completed successfully
- [ ] The pipeline has run and the status is green
- [ ] CI jobs have not been skipped (if jobs have been skipped, confirmation from the PMG conveners is necessary before merging)
- [ ] Look at the output of the `run_athena` pipeline and make sure that it has run **for all DSIDS added to the commit**. If it's not the case, confirmation from the PMG conveners is necessary before merging
- [ ] Check that no `log.generate.short` files are included in the commit to be merged to master. If such files are present, it indicates that something went wrong in the pipeline. Check the job logs and contact the package maintainers if needed.https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/157WIP: Compare status of branches to origin/master instead of HEAD [skip modfiles]2019-12-20T14:47:47+01:00Spyros ArgyropoulosWIP: Compare status of branches to origin/master instead of HEAD [skip modfiles]This will ensure that we are always comparing against what is in the remote master branch, so that when the MR gets merged the pipeline will always succeed.
Closes #54This will ensure that we are always comparing against what is in the remote master branch, so that when the MR gets merged the pipeline will always succeed.
Closes #54Frank SiegertFrank Siegerthttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/169WIP: Automatic suggestion of DSID ranges from commit script [skip ci]2020-01-02T15:45:54+01:00Spyros ArgyropoulosWIP: Automatic suggestion of DSID ranges from commit script [skip ci]As discussed in #29
**Still WIP since I think this may be far from what we want**
## How the suggestion mechanism works
When doing the dry-run of the commit script users are supposed to use `-s` in order to get a suggestion o...As discussed in #29
**Still WIP since I think this may be far from what we want**
## How the suggestion mechanism works
When doing the dry-run of the commit script users are supposed to use `-s` in order to get a suggestion of the DSID ranges that they should use.
The script will:
1. loop over all DSIDs in the commit
2. identify the jO in each DSID directory and extract the first generator
3. query all the heads in `origin`, find the min and max allowed DSID for the generator determined in (2) and return the minimum free DSID
## Example
I've added (not yet committed `999xxx/999997:mc.Ph_1.py`, `999xxx/999998/mc.Sh_2.py`, `999xxx/999999/mc.Sh_1.py` and run:
```
./scripts/commit_new_dsid.sh -d=999997-999999 -n -s
INFO: will use following remote for pushing: origin
Suggesting DSID ranges ...
jO: 999xxx/999997/mc.Ph_1.py -> generator = Ph
jO: 999xxx/999998/mc.Sh_2.py -> generator = Sh
jO: 999xxx/999999/mc.Sh_1.py -> generator = Sh
Suggested DSID range for Ph ...
999xxx/999997 -> 600xxx/600010
Suggested DSID range for Sh ...
999xxx/999998 -> 700xxx/700055
999xxx/999999 -> 700xxx/700056
```
## Not yet implemented
* The script does not try to "fill holes", e.g. in `700xxx` the first two DSIDs are `700000, 700005` so in principle the two DSIDs above could be moved to `700001, 700002`
* Generators not implemented: HI, Misc, validation. **Someone should suggest an algorithmic way of deciding what is Misc and what is validation**
* Enforcing a given DSID assignment as suggested by the scriptBetahttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/179[skip modfiles] update the name of the JOs mc.P8B_A14NNPDF23LO_pp_JpsimumuZmu...2020-01-08T18:03:40+01:00Ewelina Maria Lobodzinska[skip modfiles] update the name of the JOs mc.P8B_A14NNPDF23LO_pp_JpsimumuZmumu.pyupdate the name of the JOs mc.P8B_A14NNPDF23LO_pp_JpsimumuZmumu.pyupdate the name of the JOs mc.P8B_A14NNPDF23LO_pp_JpsimumuZmumu.pyhttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/182Adding new DSID 9500112020-01-14T15:20:19+01:00Ewelina Maria LobodzinskaAdding new DSID 950011# Merge Request for the registration of new jobOption files that use existing base fragments
## JIRA ticket
Please provide here the link to the ATLMCPROD JIRA ticket where this request has been discussed (just replace XXX with your re...# Merge Request for the registration of new jobOption files that use existing base fragments
## JIRA ticket
Please provide here the link to the ATLMCPROD JIRA ticket where this request has been discussed (just replace XXX with your request number):
ATLMCPROD-XXX
## Checklist for requesters
- [ ] I have followed the [guidelines in McSampleRequestProcedure](https://twiki.cern.ch/twiki/bin/view/AtlasProtected/McSampleRequestProcedure)
- [ ] I have provided the ATLMCPROD JIRA ticket where the request has been discussed
- [ ] The request has been approved by the physics group
- [ ] I have checked the "Squash commits when merge request is accepted" option below
- [ ] I have checked the "Delete source branch when merge request is accepted." option below
## Checklist for approvers
- [ ] All the items in the "Checklist for requesters" have been completed successfully
- [ ] The pipeline has run and the status is green
- [ ] CI jobs have not been skipped (if jobs have been skipped, confirmation from the PMG conveners is necessary before merging)
- [ ] Look at the output of the `run_athena` pipeline and make sure that it has run **for all DSIDS added to the commit**. If it's not the case, confirmation from the PMG conveners is necessary before merging
- [ ] Check that no `log.generate.short` files are included in the commit to be merged to master. If such files are present, it indicates that something went wrong in the pipeline. Check the job logs and contact the package maintainers if needed.https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/184Test MR to see what happens when we skip pipelines[skip ci]2020-01-15T08:05:45+01:00Spyros ArgyropoulosTest MR to see what happens when we skip pipelines[skip ci]apparently we can't merge...apparently we can't merge...https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/185Test with [skip ci] - MR should not be allowed2020-01-15T08:25:30+01:00Spyros ArgyropoulosTest with [skip ci] - MR should not be allowedMR should not be allowed since pipeline has been skippedMR should not be allowed since pipeline has been skippedhttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/186Test with [skip all]2020-01-15T08:39:09+01:00Spyros ArgyropoulosTest with [skip all]`allow_merge` job should run when MR is created and the button should be available`allow_merge` job should run when MR is created and the button should be availablehttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/187Test2020-01-15T08:47:59+01:00Spyros ArgyropoulosTesthttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/188Test [skip all] after fix2020-01-15T09:00:30+01:00Spyros ArgyropoulosTest [skip all] after fix`allow_merge` should run now`allow_merge` should run nowhttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/189Dsid spalazzo 500016 fix ci2020-01-15T09:53:23+01:00Spyros ArgyropoulosDsid spalazzo 500016 fix ciA test for how a real jO MR would look like after we merge the latest fixes to the CI.
Again we expect the `allow_merge` job to get triggered when the MR is created and the merge button to become available after the first approval.A test for how a real jO MR would look like after we merge the latest fixes to the CI.
Again we expect the `allow_merge` job to get triggered when the MR is created and the merge button to become available after the first approval.https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/192[skip ci] Update mc.MGH7_LHE_UE_MMHT2014_LHE.py2020-01-16T11:17:41+01:00Ewelina Maria Lobodzinska[skip ci] Update mc.MGH7_LHE_UE_MMHT2014_LHE.pyadd line stting number of inputs to 1add line stting number of inputs to 1https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/194[skip ci] Update mc.MGH7_LHE_UE_MMHT2014_LHE.py2020-01-16T11:18:09+01:00Ewelina Maria Lobodzinska[skip ci] Update mc.MGH7_LHE_UE_MMHT2014_LHE.pyadd line for setting number of input files/Job to 1add line for setting number of input files/Job to 1https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/195[skip ci] Update mc.MGH7_LHE_UE_MMHT2014_LHE.py2020-01-16T11:18:29+01:00Ewelina Maria Lobodzinska[skip ci] Update mc.MGH7_LHE_UE_MMHT2014_LHE.pyadd line setting number of input files/job to 1add line setting number of input files/job to 1