MC Job Options merge requestshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests2019-10-23T13:44:07+02:00https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/80logParser switch from minEvents to nEventsPerJob [skip modfiles]2019-10-23T13:44:07+02:00Frank SiegertlogParser switch from minEvents to nEventsPerJob [skip modfiles]Closes #39Closes #39AlphaSpyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/66Fix athena [skip modfiles]2019-10-14T14:10:33+02:00Spyros ArgyropoulosFix athena [skip modfiles]* Make `run_athena` run `Gen_tf.py` locally, i.e. not taking jobConfig from cvmfs (#Closes #38)
* Introduce option to skip minEvents checks in logParser
* Improve some printouts in logParser
* Switch from python2 to python
* Onl...* Make `run_athena` run `Gen_tf.py` locally, i.e. not taking jobConfig from cvmfs (#Closes #38)
* Introduce option to skip minEvents checks in logParser
* Improve some printouts in logParser
* Switch from python2 to python
* Only max(1, 0.01*minEvents) events are run in the `run_athena` jobAlphaFrank SiegertFrank Siegerthttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/62Fixes for commit script [skip modfiles]2019-10-04T10:34:51+02:00Spyros ArgyropoulosFixes for commit script [skip modfiles]1. This fixes the bug in !61 where `[skip athena]` was specified as a commit message when running the commit script but `log.generate.short` was added to the commit.
`log.generate.short` is removed automatically by the check_logPar...1. This fixes the bug in !61 where `[skip athena]` was specified as a commit message when running the commit script but `log.generate.short` was added to the commit.
`log.generate.short` is removed automatically by the check_logParser job, so if this doesn't run the `log.generate.short` would be left in the repo. I know added a protection against this, so that if `[skip athena]` or `[skip logparser]` are specified in the commit message `log.generate.short` will not be added to the commit.
2. Fixing bug in #40 (Closes #40)
3. Moving logParser to python3 (Closes #31)
4. Executing scripts from gitlab-ci instead of sourcing them (Closes #37)
Successful test shown here
![Screenshot_2019-09-29_at_15.35.03](/uploads/e50f1ac62b8548df1bddff1d55486f94/Screenshot_2019-09-29_at_15.35.03.png)AlphaFrank SiegertFrank Siegerthttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/67Remove only keywords from CI jobs so that all jobs run on all commits [skip m...2019-10-14T21:16:27+02:00Spyros ArgyropoulosRemove only keywords from CI jobs so that all jobs run on all commits [skip modfiles]As the title says now the jobs will run on all commits.
This fixes the problems that are described in #36 (also see #42)
Closes #36As the title says now the jobs will run on all commits.
This fixes the problems that are described in #36 (also see #42)
Closes #36AlphaFrank SiegertFrank Siegerthttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/59Fix python version problem for using check_naming in gen_tf2019-10-03T10:43:04+02:00Chen Pengchen.peng@cern.chFix python version problem for using check_naming in gen_tfFor using check_naming function in Gen_tf. Python version conflicts because of the print function are fixed. Also, change the location of Generator_list file to a fixed path in cvmfsFor using check_naming function in Gen_tf. Python version conflicts because of the print function are fixed. Also, change the location of Generator_list file to a fixed path in cvmfsAlphaSpyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/190New CI job to allow MRs after the change in gitlab policy [skip modfiles]2020-01-15T13:14:34+01:00Spyros ArgyropoulosNew CI job to allow MRs after the change in gitlab policy [skip modfiles]As discussed in #60 due to a change in the gitlab policy, the new gitlab policy considers non-triggered pipelines as having the same status as failed pipelines, i.e. if there is no pipeline in a MR, the MR cannot be merged. This in parti...As discussed in #60 due to a change in the gitlab policy, the new gitlab policy considers non-triggered pipelines as having the same status as failed pipelines, i.e. if there is no pipeline in a MR, the MR cannot be merged. This in particular means that anything committed with `[skip ci]` etc is no longer mergeable.
The changes implemented here circumvent this by having a "dummy" job (as per the gitlab guidelines) which will get triggered only
* iff a MR is created
* iff the commit message contains `[skip all]`
## Impact on MRs with new jO
For new jO MC contacts first create a new branch, where all the CI jobs will run, as shown [here](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/pipelines/1343813)
![Screenshot_2020-01-15_at_10.01.36](/uploads/e948c67f5c485ac47312c68fa5ff01d2/Screenshot_2020-01-15_at_10.01.36.png)
Then if everything runs successfully, the `check_logParser` job will make a push with `[skip all]` and when the MR is created there will be a single CI job running (and always succeeding) as shown [here](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/pipelines/1343837)
![Screenshot_2020-01-15_at_10.04.19](/uploads/cfec4c35fc99ea8305c1acd841391d90/Screenshot_2020-01-15_at_10.04.19.png)
## Impact on MRs for code development
For anything else related to code development (modifications of jO, scripts etc) we have to use the usual commit messages (e.g. `[skip modfiles]`) so that at least some pipelines run.
There is also a fallback solution to skip all pipelines, i.e. instead of `[skip ci]` one can commit with `[skip all]`, however this is obviously reserved for exceptional cases.
I modified the README and MR template to reflect the above. If anything is not clear let me know.
Closes #60BetaSimone AmorosoFrank SiegertChristian GutschowSimone Amorosohttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/166Introduce timing information in logParser [skip modfiles]2019-12-25T13:00:59+01:00Spyros ArgyropoulosIntroduce timing information in logParser [skip modfiles]This MR introduces timing information according to what was discussed in #45 #51 and will allow to reactivate the athena CI job.
## Handling of `CountHepMC == nEventsPerJob`
The **check that `CountHepMC == nEventsPerJob` has...This MR introduces timing information according to what was discussed in #45 #51 and will allow to reactivate the athena CI job.
## Handling of `CountHepMC == nEventsPerJob`
The **check that `CountHepMC == nEventsPerJob` has not been removed**, because I think it might still be a good handle to catch unwanted bugs. Instead I added a flag in the commit script that can bypass this check. Here is what happens when trying to add `421305`, run with `--maxEvents=10`:
```
mcjoboptions_3 sargyrop$ ./scripts/commit_new_dsid.sh -d=421305 -n
INFO: will use following remote for pushing: origin
New DSID directory: 421xxx/421305 ...
OK: log.generate file found.
...
- CountHepMC Events passing all checks and written = 10 <-- ERROR: This is not equal to nEventsPerJob=1000
...
---------------------
Summary:
---------------------
Errors : 1 , Warnings : 5 -> Errors encountered! Not ready for production!
ERROR: log.generate contains errors
Fix them before committing anything!
```
In this case users should **make sure that disabling the `CountHepMC` check is what they really want** and run `commit_new_dsid.sh` with `-t`:
```
./scripts/commit_new_dsid.sh -d=421305 -t -n
INFO: will use following remote for pushing: origin
New DSID directory: 421xxx/421305 ...
OK: log.generate file found.
OK: log.generate file contains no errors
OK: CI job expected to last less than 1h - time estimate: 0.17 hours
Will now add files to git commit
File: log.generate cannot be added to the commit. Skipping.
Skipping: log.generate.short since the logParser CI job will not be run
Added: mc.Sh_228_Wmunu_EnhLogPtV.py
Added: mc_13TeV.Sh_228_Wmunu_EnhLogPtV.GRID.tar.gz
```
## How timing works in logParser
There are 3 timing informations provided:
1. **if `CountHepMC == nEventsPerJob`**: it just prints the timing information from log.generate (same as what we had before)
2. **if `CountHepMC != nEventsPerJob`**: it will extrapolate the timing information using `extrapCPU=float(generateDict['nEventsPerJob'])*cpuPerJob/float(CountHepMC)`
3. **timing for CI job**: this is something extra to help debugging and have more meaningful printouts, it will just extrapolate the timing information to `max(10,0.01*nEventsPerJob)`, i.e. the number of events we will run in the CI
## Timing limits
The actual (case 1) or extrapolated CPU time (case 2 above) lead to the following behaviour:
* **CPU < 1 h OR CPU > 18 h** => ERROR
* **CPU between 8-12 h** => GOOD
* **CPU=1-8 OR 12-18 h** => WARNING
i.e. currently the limits are quite loose and we would only cut jobs which last less than 1h or more than 18h.
For the CI (case 3) above we cut everything that is expected to last more than 1h.
Here is an example from `logParser` run on the above 421305 DSID with 10 events (nEventsPerJob=1000 in this case):
```
---------------------
Performance metrics:
---------------------
- actual CPU (10 events) = 0.17 hrs
- CPU extrapolated to 1000 events = 16.6 hrs <-- WARNING: CPU time not optimal - should be between 8-12h. Adjust nEventsPerJob!
- estimated CPU for CI job = 0.17 hrs
- Virtual memory = 2957.352 Mb
```
Notice that the first line is the actual time it took to run the job (with `--maxEvents=10`), the 2nd is the extrapolation to `nEventsPerJob=1000` and the 3rd is the extrapolation for the `max(10,0.01*1000)=10` events that would run in the CI.
Note that `logParser` in CI is always run with `-t` so there should be no extra change needed in order to re-activate the athena CI jobs.
Closes #51
I think this also closes #15 BetaFrank SiegertChristian GutschowFrank Siegerthttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/97Enable logParser to handle jO made with sherpaTarCreator [skip modfiles]2019-10-31T20:49:26+01:00Spyros ArgyropoulosEnable logParser to handle jO made with sherpaTarCreator [skip modfiles]This enables logParser to see if the jO is produced with sherpaTarCreator and if it is, it will disable the checks associated to nEventsPerJob.
Technically this has been implemented as a Sherpa-specific test and only searches for the ...This enables logParser to see if the jO is produced with sherpaTarCreator and if it is, it will disable the checks associated to nEventsPerJob.
Technically this has been implemented as a Sherpa-specific test and only searches for the **presence of the work `sherpaTarCreator` anywhere in the log.generate**. This is perhaps too loose and can be tailored according to our needs so @fsiegert and @cgutscho can comment - the point is that you now have all the freedom to define what you print out as an indication of sherpaTarCreator being used ;)
Tested with 950006 that Chris provided after tweaking the file to produce an ERROR and before the change I get this:
```
- CountHepMC Events passing all checks and written = 10 <-- ERROR: This is not equal to nEventsPerJob=100
...
---------------------
Summary:
---------------------
Errors : 2 , Warnings : 2 -> Errors encountered! Not ready for production!
```
and after the change I get this
```
-------------------------
Generator specific tests: Sherpa(v.2.2.8p3)
-------------------------
- retried events Beam_Remnants = 0.9 %
- retried events Jet_Evolution:CSS = 1 %
- jO created with sherpaTarCreator - will disable nEventsPerJob checks
...
- CountHepMC Events passing all checks and written = 10 <-- WARNING: This is not equal to nEventsPerJob=100
...
---------------------
Summary:
---------------------
Errors : 1 , Warnings : 3 -> Errors encountered! Not ready for production!
```
In this case the error is coming from a high memory consumption so there is still 1 ERROR remaining but the point is that the ERROR from nEventsPerJob is reduced to a WARNING (we might even want to reduce it even further to an INFO)
Closes #35BetaFrank SiegertChristian GutschowFrank Siegerthttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/161Compare to origin/master in CI jobs [skip modfiles]2019-12-21T15:10:24+01:00Spyros ArgyropoulosCompare to origin/master in CI jobs [skip modfiles]* When CI job is launched always compare the `HEAD` of that branch against `origin/master`
* Fix bug in `check_unique_physicsShort` - we had forgotten to change `mc??` to `mc` when we switched from `mc16` to `mc` for the jO naming so t...* When CI job is launched always compare the `HEAD` of that branch against `origin/master`
* Fix bug in `check_unique_physicsShort` - we had forgotten to change `mc??` to `mc` when we switched from `mc16` to `mc` for the jO naming so this test has been inactive so far
!160 needs to be merged first hence the WIP
Closes #54BetaFrank SiegertChristian GutschowFrank Siegerthttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/171Reactivate athena in CI [skip modfiles]2020-01-06T15:29:04+01:00Spyros ArgyropoulosReactivate athena in CI [skip modfiles]Reactivating athena in CI
Successful test here: https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/pipelines/1323415
![Screenshot_2020-01-04_at_17.59.20](/uploads/1acad1022ddd903dd3017e68f7b4c1fa/Screenshot_2020-01-04_at_17.59....Reactivating athena in CI
Successful test here: https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/pipelines/1323415
![Screenshot_2020-01-04_at_17.59.20](/uploads/1acad1022ddd903dd3017e68f7b4c1fa/Screenshot_2020-01-04_at_17.59.20.png)
![Screenshot_2020-01-04_at_17.59.59](/uploads/17046eef671c7fdd62a709901eb04b96/Screenshot_2020-01-04_at_17.59.59.png)
![Screenshot_2020-01-04_at_17.59.44](/uploads/7a35d8dcf697e34e860d675d2327bdc9/Screenshot_2020-01-04_at_17.59.44.png)BetaFrank SiegertChristian GutschowFrank Siegerthttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/170DSID checking in commit script and CI2020-01-14T15:57:42+01:00Spyros ArgyropoulosDSID checking in commit script and CIThis MR introduces two changes:
* checking of DSID range in `commit_new_dsid.sh` and in CI
* suggestion of DSID range to use
## Checking of DSIDs
When `commit_new_dsid.sh` is run it will automatically loop over all jO that are ...This MR introduces two changes:
* checking of DSID range in `commit_new_dsid.sh` and in CI
* suggestion of DSID range to use
## Checking of DSIDs
When `commit_new_dsid.sh` is run it will automatically loop over all jO that are added (specified with the `-d` flag) and make sure that they are in the correct DSID range, as discussed in #29.
Example (correct DSID range):
![Screenshot_2020-01-02_at_17.07.32](/uploads/edbe0561c53b8f47eb21220e84197ae8/Screenshot_2020-01-02_at_17.07.32.png)
Example (wrong DSID range):
![Screenshot_2020-01-02_at_17.08.11](/uploads/2cf4e1a8802774df10a38aeb76153b83/Screenshot_2020-01-02_at_17.08.11.png)
The DSID range checking is done via the `check_jo_consistency.py` script, so one also performs all the naming checks at the same time.
Typically one would first run with `-n` so one is supposed to rename the directories according to what the script suggest and then rerun the scripts with the correct DSIDs and eventually commit.
## CI
The CI check is incorporated in the `check_jo_consistency` job, i.e. after this MR is merged one will not only check the jO for their naming but also make sure that they are in a directory in the correct DSID range.
Since this job is not allowed to fail **we won't be accepting jOs if they don't live in the correct DSID range**.
Example: ![Screenshot_2020-01-04_at_14.26.13](/uploads/975350b8ff254cdbe25fe4db4de74906/Screenshot_2020-01-04_at_14.26.13.png)
## How the suggestion mechanism works
When doing the dry-run of the commit script the script will:
1. loop over all DSIDs in the commit
2. identify the jO in each DSID directory and extract the first generator
3. query all the heads in `origin`, find the min and max allowed DSID for the generator determined in (2) and return the minimum free DSID
**Important**
* since the minimum free DSID is returned **if there are "holes" in a DSID range, the suggested range will try to cover them**, e.g. if `700000,700002` are occupied and we ask to commit 3 new Sherpa jO the script will suggest `700001,700003,700004`. As time goes by, this should happen less often if people stick to the suggestion of the script
* if a DSID to be committed is already in the correct DSID range, there is no ERROR thrown. In the above example, if `700000,700002` are occupied and we put a new Sherpa DSID in `799999` the script will not complain.
Closes #29BetaFrank SiegertChristian GutschowFrank Siegerthttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/169WIP: Automatic suggestion of DSID ranges from commit script [skip ci]2020-01-02T15:45:54+01:00Spyros ArgyropoulosWIP: Automatic suggestion of DSID ranges from commit script [skip ci]As discussed in #29
**Still WIP since I think this may be far from what we want**
## How the suggestion mechanism works
When doing the dry-run of the commit script users are supposed to use `-s` in order to get a suggestion o...As discussed in #29
**Still WIP since I think this may be far from what we want**
## How the suggestion mechanism works
When doing the dry-run of the commit script users are supposed to use `-s` in order to get a suggestion of the DSID ranges that they should use.
The script will:
1. loop over all DSIDs in the commit
2. identify the jO in each DSID directory and extract the first generator
3. query all the heads in `origin`, find the min and max allowed DSID for the generator determined in (2) and return the minimum free DSID
## Example
I've added (not yet committed `999xxx/999997:mc.Ph_1.py`, `999xxx/999998/mc.Sh_2.py`, `999xxx/999999/mc.Sh_1.py` and run:
```
./scripts/commit_new_dsid.sh -d=999997-999999 -n -s
INFO: will use following remote for pushing: origin
Suggesting DSID ranges ...
jO: 999xxx/999997/mc.Ph_1.py -> generator = Ph
jO: 999xxx/999998/mc.Sh_2.py -> generator = Sh
jO: 999xxx/999999/mc.Sh_1.py -> generator = Sh
Suggested DSID range for Ph ...
999xxx/999997 -> 600xxx/600010
Suggested DSID range for Sh ...
999xxx/999998 -> 700xxx/700055
999xxx/999999 -> 700xxx/700056
```
## Not yet implemented
* The script does not try to "fill holes", e.g. in `700xxx` the first two DSIDs are `700000, 700005` so in principle the two DSIDs above could be moved to `700001, 700002`
* Generators not implemented: HI, Misc, validation. **Someone should suggest an algorithmic way of deciding what is Misc and what is validation**
* Enforcing a given DSID assignment as suggested by the scriptBetahttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/167Updating minimum number of events in run_athena and testing only 1 DSID [skip...2019-12-25T14:48:10+01:00Spyros ArgyropoulosUpdating minimum number of events in run_athena and testing only 1 DSID [skip modfiles]Two changes introduced in `run_athena`:
* Only run a single DSID for the moment (until we are confident that enough jobs succeed). Then we can increase this number to be equal to the number of committed DSIDs or a maximum number of DSI...Two changes introduced in `run_athena`:
* Only run a single DSID for the moment (until we are confident that enough jobs succeed). Then we can increase this number to be equal to the number of committed DSIDs or a maximum number of DSIDs allowed by timing information
* Increase the minimum number of events to run in `run_athena` to 10. This was necessary since with 1 event several Sherpa jobs were failing the `logParser` checks due to negative cross-sections
Closes #45BetaFrank SiegertChristian GutschowFrank Siegerthttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/164Change acceptable number of output events in logParser2020-01-02T12:20:51+01:00Spyros ArgyropoulosChange acceptable number of output events in logParserAccording to the discussion we had with @fsiegert @cgutscho I change the acceptable number of events (written in the EVNT file to): `1, 2, 5, 10, 20, 50, 100, 200, 500, 1000, 2000, 5000, 10000`.
Also the timing checks are changed to t...According to the discussion we had with @fsiegert @cgutscho I change the acceptable number of events (written in the EVNT file to): `1, 2, 5, 10, 20, 50, 100, 200, 500, 1000, 2000, 5000, 10000`.
Also the timing checks are changed to that if we use 10k events (written in the EVNT) and the job is very fast, there is no error thrown.
This will be supplemented in a following MR with timing checks as discussed in #51 #45BetaFrank SiegertChristian GutschowFrank Siegerthttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/160Unify whitelist in commit script and in CI [skip modfiles]2019-12-21T15:02:14+01:00Spyros ArgyropoulosUnify whitelist in commit script and in CI [skip modfiles]`check_added_files.sh` contained 2 `checkWhiteList` functions, one from `checkWhitelist.sh` and a local implementation. I remove the local implementation so that we only have 1 definition in the whole package.`check_added_files.sh` contained 2 `checkWhiteList` functions, one from `checkWhitelist.sh` and a local implementation. I remove the local implementation so that we only have 1 definition in the whole package.BetaFrank SiegertChristian GutschowFrank Siegerthttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/150Add check for mcgensvc readability of eos directories [skip modfiles]2019-12-17T18:43:14+01:00Spyros ArgyropoulosAdd check for mcgensvc readability of eos directories [skip modfiles]Closes #56
Also remove some log.generate.short files
![Screenshot_2019-12-17_at_17.54.47](/uploads/65eb6f8230c31d4266b65fdcef34f9e2/Screenshot_2019-12-17_at_17.54.47.png)Closes #56
Also remove some log.generate.short files
![Screenshot_2019-12-17_at_17.54.47](/uploads/65eb6f8230c31d4266b65fdcef34f9e2/Screenshot_2019-12-17_at_17.54.47.png)BetaFrank SiegertFrank Siegerthttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/105Change how check_modified_files.sh works #20 [skip modfiles]2019-11-11T20:31:33+01:00Spyros ArgyropoulosChange how check_modified_files.sh works #20 [skip modfiles]Checking merged master version instead of HEAD in check_modified_files script.
This means that when modifying files in a MR which has not yet been merged we won't need to specify `[skip modfiles]`
Closes #20
Checks:
* [x] ...Checking merged master version instead of HEAD in check_modified_files script.
This means that when modifying files in a MR which has not yet been merged we won't need to specify `[skip modfiles]`
Closes #20
Checks:
* [x] Modify existing file -> commit without `[skip modfiles]` -> CI should fail
https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/6104486
![Screenshot_2019-11-10_at_18.02.50](/uploads/8b41edc1e8b80a0732e4e143359ccb2d/Screenshot_2019-11-10_at_18.02.50.png)
* [x] Add file in existing directory -> commit without `[skip modfiles]` -> CI should fail
https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/6104493
![Screenshot_2019-11-10_at_18.05.49](/uploads/1ca22513f6d5df9c59a6721509a43c41/Screenshot_2019-11-10_at_18.05.49.png)
* [x] Modify file in unmerged branch -> commit without `[skip modfiles]` -> CI should succeed
https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/6104498
![Screenshot_2019-11-10_at_18.10.05](/uploads/30abe192d881deec1731293f4db5210f/Screenshot_2019-11-10_at_18.10.05.png)
Note that before merging this it will not be possible for the CI to succeed since `scripts/check_modified_files.sh` has been modified from master, however what I tested here was that I added a new DSID 950007 and then modified the jO inside there. You can see that the modification of the jO is not picked up by the script since 950007 does not yet exist in the master.BetaFrank SiegertChristian GutschowFrank Siegerthttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/516Fix bug when copying links pointing to eos2020-06-19T13:19:55+02:00Spyros ArgyropoulosFix bug when copying links pointing to eos## Description of bug
When assigning DSIDs the commit script would do first a `cp -a` which would copy files and links.
For links a second pass was made in order to fix relative links and make them point to the correct (new) DSID whic...## Description of bug
When assigning DSIDs the commit script would do first a `cp -a` which would copy files and links.
For links a second pass was made in order to fix relative links and make them point to the correct (new) DSID which was being booked.
The bug is that this would also try to fix links pointing to eos and since the DSID directory of an eos file could not be extracted it would lead to an error:
```
ERROR: 500xxx/500336/mc_13TeV.MGPy8EG_A14NNPDF23LO_Wmunu3jets_FxFx.GRID.tar.gz is pointing to a file in the same DSID directory. Avoid the link and consider renaming the original file.
```
reported by @fgiuli
## Changes introduced
Skip the second pass (link reassignement) when a link is pointing to eos
## Tests
Running on directory from Francesco
```
./scripts/commit_new_dsid.sh -d=100002 -m='FxFX V+3jets inclusive samples' --nogit
INFO: will use following remote for pushing: origin
Will use branch: dsid_sargyrop_100002...
Will create new branch: dsid_sargyrop_100002
Checking jO consistency and DSID ranges ...
New DSID directory: 500xxx/500335 ...
OK: log.generate file found.
OK: log.generate file contains no errors
OK: CI job expected to last less than 1h - time estimate: 0.12 hours
Will now add files to git commit
File: 500xxx/500335/log.generate cannot be added to the commit. Skipping.
Will add: 500xxx/500335/log.generate.short
Will add: 500xxx/500335/mc_13TeV.MGPy8EG_A14NNPDF23LO_Wenu3jets_FxFx.GRID.tar.gz
Will add: 500xxx/500335/mc.MGPy8EG_A14NNPDF23LO_Wenu3jets_FxFx.py
```
## Issues resolved
Closes #S1.2020Christian GutschowChristian Gutschowhttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/449Fix inputFilesPerJob check2020-05-19T10:36:36+02:00Spyros ArgyropoulosFix inputFilesPerJob check## Description of bug
* The check for `inputFilesPerJob` was not formatted properly and was not catching cases with more than 100 files
## Changes introduced
* Fix check for `inputFilesPerJob`
* Reformat printouts for `Gen_t...## Description of bug
* The check for `inputFilesPerJob` was not formatted properly and was not catching cases with more than 100 files
## Changes introduced
* Fix check for `inputFilesPerJob`
* Reformat printouts for `Gen_tf` checks
## Tests
Using the log provided in #112 also illustrating some of the reformatting changes:
![Screenshot_2020-05-19_at_08.52.55](/uploads/6c35ba8e39680ede9b03afe1f230cec7/Screenshot_2020-05-19_at_08.52.55.png)
![Screenshot_2020-05-19_at_08.55.41](/uploads/d2d65ada8ac1fc265e9359f7c24eccec/Screenshot_2020-05-19_at_08.55.41.png)
## Issues resolved
Closes #112S1.2020Christian GutschowChristian Gutschowhttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/445Another fix for nevents check for MG2020-05-18T11:58:36+02:00Spyros ArgyropoulosAnother fix for nevents check for MG## Description of bug
* nevents check in MG was comparing an int with a float resulting in a wrong behaviour
* the error message printed in this case was confusing since I forgot to change the prinout when I fixed this in !443
...## Description of bug
* nevents check in MG was comparing an int with a float resulting in a wrong behaviour
* the error message printed in this case was confusing since I forgot to change the prinout when I fixed this in !443
## Changes introduced
* convert everything to int when checking nevents for MG
* fix mistake in printout
## Tests
Using the output from this pipeline: https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/8411228
#### Before fix
```
ERROR: Increase nevents to be generated in MG from 110 to 11000
```
The bug here is that
* it should have succeeded since we need 110 events
* it prints out 11000 (1.1*nEventsPerJob) instead of 110 (1.1*nEventsRequested) (where the latter is actually used in the check that triggers the error)
#### After fix
The above test succeeds.
Also if I change
```
20:18:36 Py:MadGraphUtils INFO "nevents" = 110.0
```
to
```
20:18:36 Py:MadGraphUtils INFO "nevents" = 100.0
```
in the log it prints out
```
ERROR: Increase nevents to be generated in MG from 100 to 110
```
so it seems to be correct now.
## Issues resolved
Closes #S1.2020Christian GutschowChristian Gutschow