MC Job Options issueshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues2021-01-04T15:27:52+01:00https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/104Harmonise whitelist with Gen_tf2021-01-04T15:27:52+01:00Spyros ArgyropoulosHarmonise whitelist with Gen_tfCurrently the transform allows setups which are explicitly excluded in the whitelist, e.g. `DSID/dat/*.dat` which is excluded here: https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/blob/master/scripts/whitelist.sh#L9 as discussed ...Currently the transform allows setups which are explicitly excluded in the whitelist, e.g. `DSID/dat/*.dat` which is excluded here: https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/blob/master/scripts/whitelist.sh#L9 as discussed in !298
I no longer remember why we excluded some cases but we should definitely harmonise what is done in the transform and what is done in the CI.
@ewelina could you go through the whitelist and let me know what is treated differently there and in `Gen_tf` so that we harmonise?
Tag @cgutscho @fsiegertS1.2021https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/92logParser: catch cases where LHE events are not enough (low efficiency)2021-06-17T13:30:43+02:00Spyros ArgyropouloslogParser: catch cases where LHE events are not enough (low efficiency)* [x] **OTF generation**: check that `N(LHE events) >= 1.1* nEventsPerJob/(filter efficiency)`
* [x] **Showering with external LHE events**: same as above
* [x] This might require different code for each generator? To be checked
* [x]...* [x] **OTF generation**: check that `N(LHE events) >= 1.1* nEventsPerJob/(filter efficiency)`
* [x] **Showering with external LHE events**: same as above
* [x] This might require different code for each generator? To be checked
* [x] Might need to take into account how many input LHE files are used per jobS1.2021Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/64Allow CI to run with external LHE inputs2021-04-27T12:14:25+02:00Spyros ArgyropoulosAllow CI to run with external LHE inputsThe CI can currently not yet run jobs with input event files. Largely the same tasks appear for both LHE->EVNT and EVNT->EVNT files:
* [x] Upload mcgensvc grid certificate in CI and make sure `voms-proxy-init -voms atlas` works in the CI...The CI can currently not yet run jobs with input event files. Largely the same tasks appear for both LHE->EVNT and EVNT->EVNT files:
* [x] Upload mcgensvc grid certificate in CI and make sure `voms-proxy-init -voms atlas` works in the CI
* [x] Write (rucio) file that was used in local testing into `log.generate.short`, e.g. `inputGeneratorFile=TXT.<dsid>.tar.gz` for LHE or `inputEVNTFile=1231231.EVNT.pool.root` for EVNT input.
* [x] If input file is specified in `log.generate.short`, the CI `run_athena.sh` job should `rucio get` that file and add the corresponding arguments to the `Gen_tf.py` command line as described in [Twiki](https://twiki.cern.ch/twiki/bin/view/AtlasProtected/SpecialConfigurations#Using_event_input_LHE_EVNT_or_EV).
* [x] Special treatment for EVNT->EVNT jobs: Apparently such jobs do not produce a `log.generate` file but a `log.afterburn`. Disable logParser for these, or special treatment? (move to #139)
* [ ] Ideally, the CI should also check the `inputFilesPerJob` in the JO for reasonable values at this stage: it should not exceed `10GB/sizeof(downloadedTestFile)` to make sure they fit into a grid node.
* [x] Similar case that came up in #89 is running athena with `--inputGeneratorFile`
Example !203
LHE-only log: `~sargyrop/public/log.generate_LHE`S1.2021Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/141check_unique_controlFile.sh fails when it shouldn't?2021-05-13T09:00:06+02:00Jeff Shahiniancheck_unique_controlFile.sh fails when it shouldn't?[check_unique_controlFile.sh](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/blob/master/scripts/check_unique_controlFile.sh) is apparently a new part of the CI. I noticed that it fails even when given symlinks. For example, whe...[check_unique_controlFile.sh](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/blob/master/scripts/check_unique_controlFile.sh) is apparently a new part of the CI. I noticed that it fails even when given symlinks. For example, when uploading JOs (with symlinks to one control file) that look like this:
```
$ ls -a *
100001:
myJO_1.py
myControlFile.py
100002:
myJO_2.py
myControlFile.py -> ../100001/myControlFile.py
```
The CI job fails and recommends that you use symlinks (even if you already are):
```
ERROR: Duplicate file(s) found:
./100xxx/100001/myControlFile.py
If the files have exactly the same content, please only keep one physical file replacing the rest with symbolic links.
If the files have differences consider renaming the files that you added.
You can check for differences with diff -w file1 file2
```
Perhaps we need to add ```-type f``` to [this line](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/blob/master/scripts/check_unique_controlFile.sh#L23) as well?
Here's an example of a failing CI job:
https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/13818890
Tagging @sargyrop
Best,
JeffS1.2021Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/143Follow-up from ATLMCPROD-93322021-06-07T14:15:29+02:00Spyros ArgyropoulosFollow-up from ATLMCPROD-9332The following discussion from !1198 should be addressed:
- [ ] @cgutscho started a [discussion](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/1198#note_4542318): (+5 comments)
> Hi @sargyrop - this might b...The following discussion from !1198 should be addressed:
- [ ] @cgutscho started a [discussion](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/merge_requests/1198#note_4542318): (+5 comments)
> Hi @sargyrop - this might be a long shot, but do you think this is something we can catch in the CI? e.g. if there's a variable in MadGraph JOs that has `gridpack` or `grid_pack` in the name and it's still set to `True`, we put out a warning ... ?
>
> Cheers,
> Chris
Make logParser fail if the info in Chris's message below appearsS1.2021https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/142Follow-up from "LO EFT samples for 4top"2021-05-18T11:32:21+02:00Spyros ArgyropoulosFollow-up from "LO EFT samples for 4top"The following discussions from !1161 should be addressed:
> This shouldn't be part of a JobOption. The first part was fixed properly in 21.6.60 and the second part is obviously gonna cause problems. `ATHENA_PROC_NUMBER` is set to 8 ...The following discussions from !1161 should be addressed:
> This shouldn't be part of a JobOption. The first part was fixed properly in 21.6.60 and the second part is obviously gonna cause problems. `ATHENA_PROC_NUMBER` is set to 8 because the machine has 8 cores, it shouldn't be set to 80 in the JOs.
Should we add the following checks/changes:
- if ATHENA_PROC_NUMBER > 1 and release < 21.2.60 => ERROR
- if ATHENA_PROC_NUMBER > 1 => run only 1 event in CI
- change the way we check whether the jO changes ATHENA_PROC_NUMBER - this would only be safe to catch in the transform btw, but until it is implemented there we could change the check to not use anywhere ATHENA_PROC_NUMBER (not even printing it), so e.g. look in the jO and if there is an uncommented line with "ATHENA_PROC_NUMBER" in it then give error
@cgutschoS1.2021Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/140Strange behaviour of commit script when athena is skipped and run time is > 1h2021-05-06T11:03:19+02:00Spyros ArgyropoulosStrange behaviour of commit script when athena is skipped and run time is > 1hFor example in a case with 1 event, CPU=3.09h, the output is the following:
![Screenshot_2021-05-06_at_09.46.11](/uploads/63cb47b36de5f75bb8e9e6a275602971/Screenshot_2021-05-06_at_09.46.11.png)
which is correct, but when skipping athen...For example in a case with 1 event, CPU=3.09h, the output is the following:
![Screenshot_2021-05-06_at_09.46.11](/uploads/63cb47b36de5f75bb8e9e6a275602971/Screenshot_2021-05-06_at_09.46.11.png)
which is correct, but when skipping athena:
![Screenshot_2021-05-06_at_09.45.42](/uploads/c01022a7990d31535fb7cca7aa2e6a4c/Screenshot_2021-05-06_at_09.45.42.png)
the
```
printGood -f "\tOK: CI job time estimate: $cpu hours, but athena will not run in the CI"
```
message is not printed because the script never reaches that point.S1.2021Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/138Check for multiple instances of TestHepMC (and TestLHE?)2021-04-01T14:20:41+02:00Christian GutschowCheck for multiple instances of TestHepMC (and TestLHE?)In general, the transform will create an instance of TestHepMC (and in the future also TestLHE) and run some checks as part of the job. For some setups the default thresholds used in these packages may be too strict and occasionally we g...In general, the transform will create an instance of TestHepMC (and in the future also TestLHE) and run some checks as part of the job. For some setups the default thresholds used in these packages may be too strict and occasionally we get JOs that try to loosen them a bit, which is usually fine.
We recently had a case (!1066) where a fresh instance of TestHepMC was created, and the threshold were tweaked on the new instance but not the one that the transform had already created, which was then causing issues down the line.
Could we catch this sort of thing in the CI? I imagine it would just be a case of checking for a line like
```
genSeq += TestHepMC()
```
and throwing an error?S1.2021Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/135Sanity check for EVNT-to-EVNT transforms2021-06-17T11:07:17+02:00Christian GutschowSanity check for EVNT-to-EVNT transformsHi,
here's an [example JO](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/blob/master/950xxx/950096/mc.Sh_2210_Zee_E2Etransform_valid.py) for an EVNT-to-EVNT transform.
This basically clones an input EVNT, but only copies the ...Hi,
here's an [example JO](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/blob/master/950xxx/950096/mc.Sh_2210_Zee_E2Etransform_valid.py) for an EVNT-to-EVNT transform.
This basically clones an input EVNT, but only copies the event if it passes some Athena filter, hence most of the logic being protected by the `if runArgs.trfSubstepName == 'afterburn':` statement.
Now, because it copies the original EVNT, the new EVNT would have the MC channel number (or run number in the HepMC GenEvent) set to the original DSID and not the new DSID (of the E2E transform JO).
This can now be patched using the `postSeq.CountHepMC.CorrectRunNumber = True` flag seen at the bottom. Could we use the CI to catch cases where such a JO is being added, but that tag is missing from the JO?
(In principle, there is a printout in the `log.afterburn` produced by an E2E transform which one could grep for, but the CI doesn't handle jobs without input EVNT files yet.)
Thoughts/ideas?S1.2021Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/128evgen keywords not always being checked2021-01-14T18:12:46+01:00Christian Gutschowevgen keywords not always being checkedJust stumbled across this by accident:
The key words listed in `evgenConfig.keywords` should match the ones in the [official list](https://gitlab.cern.ch/atlas/athena/-/blob/21.6/Generators/EvgenJobTransforms/share/file/evgenkeywords.tx...Just stumbled across this by accident:
The key words listed in `evgenConfig.keywords` should match the ones in the [official list](https://gitlab.cern.ch/atlas/athena/-/blob/21.6/Generators/EvgenJobTransforms/share/file/evgenkeywords.txt). It turns out that when the transform doesn't find the official list in the JobOptions search path for some reason, it will be unable to check for potential mismatches and hence also not be able to print an error message.
If there's an undefined key word, the transform _should_ print:
```
msg = "evgenConfig.keywords contains non-standard keywords: %s. " % ", ".join(evil_keywords)
msg += "Please check the allowed keywords list and fix."
```
but if it cannot find the standard list it just says
```
08:29:01 Py:Gen_tf WARNING Could not find evgenkeywords.txt file EvgenJobTransforms/evgenkeywords.txt in $JOBOPTSEARCHPATH
```
in the log and the CI continues happily, see example log here:
```
/eos/atlas/atlascerngroupdisk/phys-gener/WeakBoson/SingleBoson/log/log.generate
```
Could we get the logParser to perform the check as well?S1.2021Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/124New Pythia 8 checks for changing parameters2023-10-26T16:13:33+02:00Spyros ArgyropoulosNew Pythia 8 checks for changing parametersImplement code to use new developments by Giancarlo mentioned in AGENE-1915.
- [ ] To be seen which of these should result in an error and which should be a warning.
- [ ] Also check if this catches the bug reported in ATLMCPROD-7723Implement code to use new developments by Giancarlo mentioned in AGENE-1915.
- [ ] To be seen which of these should result in an error and which should be a warning.
- [ ] Also check if this catches the bug reported in ATLMCPROD-7723S1.2021Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/57logParser gives missing weights for LHE-only geneation2021-05-08T12:30:45+02:00Spyros ArgyropouloslogParser gives missing weights for LHE-only geneationReported by @xiaohu
> Just to be 100% sure, we generate LHE files externally using Powheg-Box-V2. Weight variations due to PDF and scales are prepared. With checkMetaSG.py, we do see all the weights in evgen files (showered by Herwig7 ...Reported by @xiaohu
> Just to be 100% sure, we generate LHE files externally using Powheg-Box-V2. Weight variations due to PDF and scales are prepared. With checkMetaSG.py, we do see all the weights in evgen files (showered by Herwig7 with 21.6.12,AthGeneration) [1] and looked at the weight variations using truth derivation. But using logParser.py to check log.generate, it says “weights missing”.
Probably what happens is that the weights in logParser are read from lines that look like this:
09:33:22 MetaData: weights = MUR=0.5 MUF=0.5 | PDF=260000 MemberID=1
and for LHE-only generation this does not get written out.
Need to confirm and perhaps fix for LHE only generationS1.2021https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/12Dynamical job creation for multiple DSID commits2021-01-10T17:33:29+01:00Spyros ArgyropoulosDynamical job creation for multiple DSID commitsWhen we run athena the job will check if the expected time is > 1h and will abort if it is.
If several DSIDs are added simultaneously (e.g. 10-30 DSIDs for SUSY, EXO signal grids) the `run_athena` job will run for a very long time.
We ...When we run athena the job will check if the expected time is > 1h and will abort if it is.
If several DSIDs are added simultaneously (e.g. 10-30 DSIDs for SUSY, EXO signal grids) the `run_athena` job will run for a very long time.
We should find a workaround for that. Could be handled with the `commit_new_dsid.sh` script making several branches.
Some interesting material here: https://gitlab.com/gitlab-org/gitlab-ce/issues/45828
In particular there's a feature request for dynamic CI jobs: https://gitlab.com/gitlab-org/gitlab-ce/issues/44199 probably to be implemented in early 2020.
**Update**: this seems to be exactly what we need: https://gitlab.com/gitlab-org/gitlab/issues/35632S1.2021Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/126incorrect printout for multi-block DSIDs2020-12-22T12:49:06+01:00Christian Gutschowincorrect printout for multi-block DSIDsA minor issue, but I just came across a case in !765 where I submitted two JOs, one for a physics block and one for the validation block and the commit script ended up saying:
```
The following DSIDs have been assigned:
100xxx/100000 -...A minor issue, but I just came across a case in !765 where I submitted two JOs, one for a physics block and one for the validation block and the commit script ended up saying:
```
The following DSIDs have been assigned:
100xxx/100000 -> 950xxx/950098
100xxx/100001 -> 500xxx/500332
Run: ./scripts/commit_new_dsid.sh -d=950098-500332 -m="aMC@NLOPy8 triphoton setups for PMG pub note" to push them to git
```
Note the range being suggested for the `-d` flag :sweat_smile:Futurehttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/132Harmonisation of printouts in tranform and generator interfaces2021-06-17T13:03:33+02:00Spyros ArgyropoulosHarmonisation of printouts in tranform and generator interfacesMost of the printouts from the transform have the following format:
```
IDENTIFIER KEYWORD = VALUE
```
e.g.
```
08:29:01 Py:Gen_tf INFO .transform = Gen_tf
```
however this **very often not the case**. A f...Most of the printouts from the transform have the following format:
```
IDENTIFIER KEYWORD = VALUE
```
e.g.
```
08:29:01 Py:Gen_tf INFO .transform = Gen_tf
```
however this **very often not the case**. A few examples:
```
08:29:01 Py:Gen_tf INFO nEventsPerJob set to 2000
08:29:01 Py:Gen_tf INFO Requested output events 100
08:29:01 Py:Gen_tf WARNING Could not find evgenkeywords.txt file EvgenJobTransforms/evgenkeywords.txt in $JOBOPTSEARCHPATH
05:14:02 Nb of events : 20000
```
This means that new checks that would otherwise be trivial to implement require changes in several places (e.g. !863) and the introduction of logic which is "hacky".
We should make sure that new printouts always conform to the correct format `IDENTIFIER KEYWORD = VALUE` both in the **transform** but also in the **generator interfaces** and the above line should be **printed only once in log.generate**
I am not sure what is the best approach here. Perhaps put this in place as a "coding rule" and make everyone aware of this. (Strict checks would probably be more time-consuming to implement than just putting in place coding practices)
@ewelina I just opened this so that we somehow bring it up with the generator experts to make things easier in the future. You probably know best how to address this and maybe can discuss this in a GIT meeting.Futurehttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/101Long compilation time when running MadGraph in atlas/slc6-atlasos causing CI...2021-04-22T16:46:58+02:00Jason Robert VeatchLong compilation time when running MadGraph in atlas/slc6-atlasos causing CI timeouts## How to reproduce the problem
```
# Mount cvmfs
sudo mkdir -p /cvmfs/atlas.cern.ch
sudo mkdir -p /cvmfs/atlas-condb.cern.ch
sudo mkdir -p /cvmfs/grid.cern.ch
sudo mkdir -p /cvmfs/sft.cern.ch
sudo mount -t cvmfs atlas.cern.ch /cvmfs/at...## How to reproduce the problem
```
# Mount cvmfs
sudo mkdir -p /cvmfs/atlas.cern.ch
sudo mkdir -p /cvmfs/atlas-condb.cern.ch
sudo mkdir -p /cvmfs/grid.cern.ch
sudo mkdir -p /cvmfs/sft.cern.ch
sudo mount -t cvmfs atlas.cern.ch /cvmfs/atlas.cern.ch
sudo mount -t cvmfs atlas-condb.cern.ch /cvmfs/atlas-condb.cern.ch
sudo mount -t cvmfs grid.cern.ch /cvmfs/grid.cern.ch
sudo mount -t cvmfs sft.cern.ch /cvmfs/sft.cern.ch
# Get the docker image
docker pull atlas/slc6-atlasos
# Run image in a container and mount cvmfs
docker run -it -v /cvmfs:/cvmfs b4cfa1203c45
# Inside the docker container get the mcjoboptions repo (or alternatively you can copy it from your local area with docker cp)
kinit USER@CERN.CH
git clone https://:@gitlab.cern.ch:8443/atlas-physics/pmg/mcjoboptions.git
cd mcjoboptions
git checkout dsid_jveatch_500538
export ATLAS_LOCAL_ROOT_BASE=/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase
./scripts/run_athena.sh
```
## Debugging
#### Bottleneck: compilation time
Comparing the running times at several execution points on lxplus and in the container it seems that the problem lies on the compilation times:
```
Docker (running on a dual-core laptop with cvmfs mounted via fuse):
generate 19:24:54 INFO: Using LHAPDF v6.2.3 interface for PDFs
generate 19:26:19 INFO: Compiling source…
generate 19:31:53 INFO: ...done, continuing with P* directories => 334 sec
generate 19:31:53 INFO: Compiling StdHEP (can take a couple of minutes) ...
generate 19:45:23 INFO: …done. => 810 sec
generate 19:45:24 INFO: Compiling on 1 cores
generate 19:45:24 INFO: Compiling P0_gg_ttx...
generate 19:54:37 INFO: P0_gg_ttx done. => 553 sec
vs lxplus (interactive run)
10:15:08 INFO: Using LHAPDF v6.2.3 interface for PDFs
10:15:14 INFO: Compiling source...
10:15:26 INFO: ...done, continuing with P* directories => 12 sec
10:15:26 INFO: Compiling StdHEP (can take a couple of minutes) ...
10:16:04 INFO: …done. => 38 sec
10:16:05 INFO: Compiling on 1 cores
10:16:05 INFO: Compiling P0_gg_ttx...
10:16:45 INFO: P0_gg_ttx done. => 40 sec
```
#### Size/memory
The container available space is 53GB and where the compilation becomes slow the size of the container is ~230 MB so much smaller => **disk size does not seem to be causing the slowdown**
The available memory was changed from 1GB to 8GB without any effect on the compilation time in the container.
#### Reading from cvmfs
I run a script that 1) reads all the lines from a file that lives on cvmfs and 2) copies this script to a local directory and remove it.
The local run on my laptop (with cvmfs mounted with fuse gives this):
```
Reading 500 times
real 0m21.504s
user 0m12.937s
sys 0m8.429s
Copying 500 times
real 0m4.993s
user 0m0.620s
sys 0m2.440s
```
Running the script from the container, where the locally available cvmfs directory (see above) is mounted to the container as a volume, gives this:
```
Reading 500 times
real 1m44.217s
user 0m18.329s
sys 0m20.376s
Copying 500 times
real 0m3.716s
user 0m0.570s
sys 0m0.981s
```
**So reading a file seems to be 5x slower when running from the docker container**
#### Next steps
* [ ] To debug further we would need to know exactly how cvmfs is mounted in the gitlab runner
* [ ] Also need to check whether there is any correlation between slow reading times on cvmfs and MG - does MG call the compilers from cvmfs/reads any other info from cvmfs? Probably
---
Original report from Jason - similar issues observed with other processes which are apparently very different than this one (an NLO one and a LO one with a long decay chain)
Job [#7937441](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/7937441) failed for 9a6a4445a5bcf7ae08ac81888cccd79ef4cc4af3:
Dear experts,
The run_athena job for my branch times out. I have been trying to debug this from my side, but I am at a loss about how to proceed. The estimated execution time from each log.generate.short is ~0.1 hours, so I wouldn't expect this to be an issue. Could you please advise?
Thanks in advance,
JasonFutureSpyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/96Reviewing changes in control files etc2020-04-03T14:45:59+02:00Spyros ArgyropoulosReviewing changes in control files etcI am wondering whether it makes sense to introduce [CODEOWNERS](https://docs.gitlab.com/ee/user/project/code_owners.html) so that changes such as !317 can be verified by the appropriate people (e.g. in Exotics normally theory contacts wo...I am wondering whether it makes sense to introduce [CODEOWNERS](https://docs.gitlab.com/ee/user/project/code_owners.html) so that changes such as !317 can be verified by the appropriate people (e.g. in Exotics normally theory contacts would be responsible for a control file).
Tagging @amoroso @cgutscho @fsiegert @gstarkFuturehttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/83Use python's logging module instead of print2021-04-22T16:47:30+02:00Spyros ArgyropoulosUse python's logging module instead of printWhen running the commit script on hundreds of files buffering of the stderr/out seems to mangle the output.
```
./scripts/commit_new_dsid.sh -n -d=421332-421475
INFO: will use following remote for pushing: origin
Checking jO consistency...When running the commit script on hundreds of files buffering of the stderr/out seems to mangle the output.
```
./scripts/commit_new_dsid.sh -n -d=421332-421475
INFO: will use following remote for pushing: origin
Checking jO consistency and DSID ranges ...
ERROR in jO consistency checks
ERROR: file 421xxx/421332/mc.MGPy8EG_A14N23LO_VBFWBCpCp_150_145_MET75_MS.py
```
We should switch to using python's logging module or handle this in a smarter way so that flushing is handled in a better way.
Reported by @gstarkFuturehttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/69Checks for madgraph mode and integration time2022-02-18T15:51:06+01:00Spyros ArgyropoulosChecks for madgraph mode and integration time* if multicore mode then throw ERROR and print that they should use single core
* if integration time very long then ERROR and print that they should create a gridpack
https://prodtask-dev.cern.ch/prodtask/inputlist_with_request/26924...* if multicore mode then throw ERROR and print that they should use single core
* if integration time very long then ERROR and print that they should create a gridpack
https://prodtask-dev.cern.ch/prodtask/inputlist_with_request/26924/
https://its.cern.ch/jira/browse/ATLMCPROD-7731FutureHannes MildnerHannes Mildnerhttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/65Running pipelines in custom swarm runner2021-06-17T13:04:50+02:00Spyros ArgyropoulosRunning pipelines in custom swarm runnerInstructions from Lukas here: https://clouddocs.web.cern.ch/containers/tutorials/swarmgitlab.html
Idea would be that if we set this up we could run with the full number of events exactly as in production.
The instructions below work. ...Instructions from Lukas here: https://clouddocs.web.cern.ch/containers/tutorials/swarmgitlab.html
Idea would be that if we set this up we could run with the full number of events exactly as in production.
The instructions below work. Wo have to understand whether this is what we want:
* [ ] does it have access to cvmfs? If not how would we set it up so that it has?
* [ ] does it buy us anything from using the shared runners?
* [ ] how tough would the maintenance be?
* [ ] is it better to just set up a dedicated machine? Maybe we should ask someone from the CERN IT to do it?Future