MC Job Options merge requestshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests2019-04-07T21:37:26+02:00https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/1Test first CI script2019-04-07T21:37:26+02:00Spyros ArgyropoulosTest first CI scripthttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/3Implementing CI checks discussed in ATLMCPROD-71002019-05-18T14:34:38+02:00Spyros ArgyropoulosImplementing CI checks discussed in ATLMCPROD-7100This MR includes the implementation of the first CI with the following checks:
* checking JO naming based on a modified version of `skel.GENtoEVGEN.py` from https://svnweb.cern.ch/trac/atlasoff/browser/Generators/EvgenJobTransforms/tru...This MR includes the implementation of the first CI with the following checks:
* checking JO naming based on a modified version of `skel.GENtoEVGEN.py` from https://svnweb.cern.ch/trac/atlasoff/browser/Generators/EvgenJobTransforms/trunk/share/skeleton.GENtoEVGEN.py These are implemented in the script `scripts/check_jo_consistency.py`
* checking that the physics short is unique in the repository `scripts/check_unique_physicsShort.sh`
* checking that a MR that requests to register new jO (triggered by the `[jo ci]` commit message) does not modify/delete any existing file in the repository. Implemented in `scripts/check_modified_files.sh`
* checking that a single `log.generate` file is present in directory containing new jO and that it passes the logParser checks. Implemented in `scripts/check_logParser.sh`. If successful the job triggers a new commit saying that the pipeline was successful and removes the `log.generate` file
* README.md file
* merge request templates to be used when making a MR with new jO
The last successful pipeline is here: https://gitlab.cern.ch/sargyrop/mcjoboptions/pipelines/841189 (now without the `share` directory).
Let me also tag @ewelina since there I put `BlackList_caches.txt` and `evgenkeywords.txt` in the `common` directory otherwise `logParser` would fail.
I also added logParser as a git submodule (actually a modified version thereof which is pointing to a new branch)
Closes #5 Frank SiegertFrank Siegerthttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/4correct include statements (add missing "/")2019-05-02T16:39:05+02:00Ewelina Maria Lobodzinskacorrect include statements (add missing "/")correct include statements (add missing "/")correct include statements (add missing "/")https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/5Implement new CI job that checks GRID file accessibility and size2019-06-06T17:06:12+02:00Spyros ArgyropoulosImplement new CI job that checks GRID file accessibility and sizeThis MR implements a new CI job that will run `scripts/check_grid_files.sh` and will test:
* whether the directory in which the file lives on eos is readable by `atlcvmfs`. If it's not the job will fail
* the job will also fail if th...This MR implements a new CI job that will run `scripts/check_grid_files.sh` and will test:
* whether the directory in which the file lives on eos is readable by `atlcvmfs`. If it's not the job will fail
* the job will also fail if the file is not stored on `/eos/atlas` or `/eos/user`
* the job will fail if the files uploaded are bigger than 100 MB
The checks can be skipped using `[skip eos]`.
One thing to discuss would be whether we want to split this job into 2:
1. checking the accessibility by `atlcvmfs`
2. checking the size
since we might want to skip the size check but not the `atlcvmfs` readability check.
@fsiegert @amoroso @ewelina what do you think?[sizes.txt](/uploads/adb07075d3dd6ca432d1e3eccec95605/sizes.txt)
The average files of existing non-LHE files seems to be 40 MB, while the median is 20 MB. If we want we could reduce the size even further - depending on how much fine tuning we want.
The output of a successful pipeline based on what is in `421xxx/421003` can be found [here](https://gitlab.cern.ch/sargyrop/mcjoboptions/-/jobs/4409075)
[sizes.txt](/uploads/ad0b810638a29d3071ac6076846af363/sizes.txt)
[sizesh.txt](/uploads/5ed80c1008becd8ed7c88778ff5ff2ae/sizesh.txt)https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/6Fix behaviour of check_logParser when adding directories that contain common ...2019-06-07T15:41:06+02:00Spyros ArgyropoulosFix behaviour of check_logParser when adding directories that contain common includesSince `scripts/check_logParser.py` would look for a `log.generate` file in **all new directories added in a commit** the `421002` directory would fail the pipeline: see [https://gitlab.cern.ch/sargyrop/mcjoboptions/commit/d5a315d339b66be...Since `scripts/check_logParser.py` would look for a `log.generate` file in **all new directories added in a commit** the `421002` directory would fail the pipeline: see [https://gitlab.cern.ch/sargyrop/mcjoboptions/commit/d5a315d339b66be207dc37bf6728a5e29ecbbc31/pipelines?ref=check_common](here) because it would search for `log.generate` not only in `421002/` but also in `421002/Sherpa_i/`
This MR fixes this so that `log.generate` files are searched for only in directories containing jO named like
`mc[0-9][0-9].*.py`
The successful pipeline after the fix is shown [https://gitlab.cern.ch/sargyrop/mcjoboptions/commit/1e0f2f32dd64fb0b8e3b25526102a3eed2d0e8af/pipelines?ref=check_common](here) Frank SiegertFrank Siegerthttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/7New triggers for skipping specific CI jobs and update of README2019-06-12T11:48:12+02:00Spyros ArgyropoulosNew triggers for skipping specific CI jobs and update of READMEThis job addresses two issues, as discussed in #6
1. Switching to new development model using branches instead of forks
Forks will not work since some CI jobs (logParser, athena) rely on hidden variables that are not transferred to f...This job addresses two issues, as discussed in #6
1. Switching to new development model using branches instead of forks
Forks will not work since some CI jobs (logParser, athena) rely on hidden variables that are not transferred to forks. The only supported way of running those jobs would be to use branches.
Since the master branch is protected (noone can push to it and only maintainers can merge) there is no risk for unwanted merges (everything has to be approved and merged by maintainers) so in principle we could give developer role to everyone
2. Skipping specific jobs
I removed the option to skip the following jobs:
* jO naming (`check_jo_consistency`)
* unique physics short
and have changed the trigger in order to be able to skip specific jobs
* modified files can be skipped with `[skip modfiles]`
* logParser check can be skipped with `[skip logparser]`
3. README has been updated accordingly. Also added a description of the pipelineshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/8Running athena in CI2019-07-03T15:14:26+02:00Spyros ArgyropoulosRunning athena in CIThis MR implements the following:
**Major changes**
- Implementation of athena job in CI
- Automatic script for adding new DSIDs (or preparing the necessary input like `log.generate.short` for git submission (Closes #9)
**�Other smal...This MR implements the following:
**Major changes**
- Implementation of athena job in CI
- Automatic script for adding new DSIDs (or preparing the necessary input like `log.generate.short` for git submission (Closes #9)
**�Other small changes**
- Fix for wrong job triggering when creating a branch (Closes #11)
- Make sure that jobs that might be triggered by mistake finish successfully, e.g. a job that is supposed to check the physics short is triggered when no jO files have been added in the commit (Closes #14)
- Added job that checks which files are being committed against a white-list (for contingency in case the automatic script for git submission is not used)
- Renaming of first CI stage
**�Tests that close existing issues**
- Tested CI with 2 commits - successful output [here](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/pipelines/933513) (Closes #13)https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/13add JOs for ART tests2019-07-12T17:12:47+02:00Ewelina Maria Lobodzinskaadd JOs for ART testsadd JOs for ART testsadd JOs for ART testshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/14Updated documentation and fixes for multiple DSIDs2019-09-07T11:55:16+02:00Spyros ArgyropoulosUpdated documentation and fixes for multiple DSIDsMore info will be added - just to keep track of added items for now.
* [x] Added checklist for approvers in `.gitlab/merge_request_templates/new_jO.md` also in `README` (Closes #19)
* [x] Updated instructions for how to upload new jO...More info will be added - just to keep track of added items for now.
* [x] Added checklist for approvers in `.gitlab/merge_request_templates/new_jO.md` also in `README` (Closes #19)
* [x] Updated instructions for how to upload new jO using the automatic script (Closes #18)
* [x] Added `git push -o ci.skip origin` in automatic script to skip the pipelines when the branch is first created
* [x] Pick up Frank's change for `check_added_files`
* [x] Skip athena jobs for directories where there is no `log.generate.short` (Closes #17)
* [x] Use `Gen_tf` instead of `Generate_tf` (Closes #16)
* [x] Check that there are no explicit "MC15JobOptions/..." includes
* [x] Change `check_modified_files` to fix #25 Frank SiegertFrank Siegerthttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/15Testart2019-07-16T12:30:20+02:00Ewelina Maria LobodzinskaTestartcorrect jo for 421100correct jo for 421100https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/16Testart2019-07-17T16:52:33+02:00Ewelina Maria LobodzinskaTestartcorrect typocorrect typohttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/17Testart2019-08-23T13:38:18+02:00Ewelina Maria LobodzinskaTestartadd some common fragments to the working directoriesadd some common fragments to the working directorieshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/18Adding H7 checks to logParser2019-07-29T09:39:41+02:00Spyros ArgyropoulosAdding H7 checks to logParserImplementing checks from https://gitlab.cern.ch/atlas-physics/pmg/tools/logParser/merge_requests/9
**when merging make sure that the commit message reads `"Adding H7 checks to logParser [skip modfiles]"` otherwise the pipeline will fail**Implementing checks from https://gitlab.cern.ch/atlas-physics/pmg/tools/logParser/merge_requests/9
**when merging make sure that the commit message reads `"Adding H7 checks to logParser [skip modfiles]"` otherwise the pipeline will fail**Frank SiegertFrank Siegerthttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/22Resolve "check_added_files needs to be less strict"2019-08-26T09:46:51+02:00Frank SiegertResolve "check_added_files needs to be less strict"Closes #26Closes #26Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/23add new tests [skip athena][skip ci]2019-08-26T11:38:56+02:00Frank Siegertadd new tests [skip athena][skip ci]This replaces (and hopefully fixes) !21This replaces (and hopefully fixes) !21Ewelina Maria LobodzinskaEwelina Maria Lobodzinskahttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/25Changing naming conventions of jO files, removing year [skip ci]2019-09-05T18:29:04+02:00Spyros ArgyropoulosChanging naming conventions of jO files, removing year [skip ci]As discussed in the last meeting I remove the year from the jO files and also the scripts.
**NB**: the GRID files are still called `mc16_13TeV.*.GRID.tar.gz`
Closes #27 As discussed in the last meeting I remove the year from the jO files and also the scripts.
**NB**: the GRID files are still called `mc16_13TeV.*.GRID.tar.gz`
Closes #27 Frank SiegertFrank Siegerthttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/28[skip ci] Update mc.Py8EG_A2MSTW2008LO_minbias_inelastic_high.py to correct t...2019-09-03T12:03:42+02:00Ewelina Maria Lobodzinska[skip ci] Update mc.Py8EG_A2MSTW2008LO_minbias_inelastic_high.py to correct the Findjets syntaxhttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/29Add Generator List file and update checkconsistency function2019-09-05T14:25:21+02:00Chen Pengchen.peng@cern.chAdd Generator List file and update checkconsistency functionSpyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/30Patch 62019-09-06T15:46:32+02:00Ewelina Maria LobodzinskaPatch 6add JOs for multiinput runningadd JOs for multiinput runninghttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/32Patch 92019-09-06T15:56:51+02:00Ewelina Maria LobodzinskaPatch 9add JOs for ART test with MG producing only LHE (without EVNT)add JOs for ART test with MG producing only LHE (without EVNT)