MC Job Options issueshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues2020-12-09T10:09:13+01:00https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/107logParser crash due to double print of nevents in MG2020-12-09T10:09:13+01:00Spyros ArgyropouloslogParser crash due to double print of nevents in MGlogParser failing again because of double print-out of nevents:
```
22:32:58 Py:MadGraphUtils INFO Setting nevents = 11000.
22:33:05 Py:MadGraphUtils INFO "nevents" = 11000
```
The first printout seemed to be the old implementa...logParser failing again because of double print-out of nevents:
```
22:32:58 Py:MadGraphUtils INFO Setting nevents = 11000.
22:33:05 Py:MadGraphUtils INFO "nevents" = 11000
```
The first printout seemed to be the old implementation before the restructuring in rel. 21.6.23, however I don't understand why both printouts are printed now. Is this expected @zmarshal @hmildner @mcfayden ?
The jO is attached - provided by @ewelina - this was run in 21.6.27.
[mc.MGPy8EG_A14NNPDF23_tWgamma.py](/uploads/66b17b0604410f93d826969cc504c7ef/mc.MGPy8EG_A14NNPDF23_tWgamma.py)
Just to say if this is expected we can easily change the behaviour to parse lines containing `"nevents"` (with quotes) currently it tries to find lines containing `nevents` (without quotes) and since the printout is different (trailing dot) the first print-out is not parsed correctly. S1.2020Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/106Job Failed #81849032020-04-29T21:17:04+02:00Xiaohu SunJob Failed #8184903Job [#8184903](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/8184903) failed for 127510e74ddbca868a29efecbd1b8c6144bf63b8:Job [#8184903](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/8184903) failed for 127510e74ddbca868a29efecbd1b8c6144bf63b8:https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/105int conversion of a string "nevents" that contains a float2020-04-29T21:17:33+02:00Xiaohu Sunint conversion of a string "nevents" that contains a floatQuite often people define nevents by multiplying a bunch of numbers (safe margin, truth efficiency etc.), then nevents is a float. The log file would contain
20:49:39 Py:MadGraphUtils INFO Setting nevents = 11000.0.
where "1100.0" ...Quite often people define nevents by multiplying a bunch of numbers (safe margin, truth efficiency etc.), then nevents is a float. The log file would contain
20:49:39 Py:MadGraphUtils INFO Setting nevents = 11000.0.
where "1100.0" is picked by logParser as a string.
Then in the check script
https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/blob/master/scripts/logParser.py#L271
neventsMG=int(generatorDict['nevents'][0])
will crash, as int("11000.0") would not work.
ValueError: invalid literal for int() with base 10
Would this be fixed? Thanks!
Best,
Xiaohu2020-04-30https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/104Harmonise whitelist with Gen_tf2021-01-04T15:27:52+01:00Spyros ArgyropoulosHarmonise whitelist with Gen_tfCurrently the transform allows setups which are explicitly excluded in the whitelist, e.g. `DSID/dat/*.dat` which is excluded here: https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/blob/master/scripts/whitelist.sh#L9 as discussed ...Currently the transform allows setups which are explicitly excluded in the whitelist, e.g. `DSID/dat/*.dat` which is excluded here: https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/blob/master/scripts/whitelist.sh#L9 as discussed in !298
I no longer remember why we excluded some cases but we should definitely harmonise what is done in the transform and what is done in the CI.
@ewelina could you go through the whitelist and let me know what is treated differently there and in `Gen_tf` so that we harmonise?
Tag @cgutscho @fsiegertS1.2021https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/102Add checks for input files2020-07-30T07:37:56+02:00Spyros ArgyropoulosAdd checks for input filesAdd checks:
* [ ] no `evgenConfig.inputfilecheck`
* [ ] no `evgenConfig.inputconfcheck` allowed
both are always in the top JO
Also
* [ ] Restructure checks so that everything related to reading the jO is done in one place and everyt...Add checks:
* [ ] no `evgenConfig.inputfilecheck`
* [ ] no `evgenConfig.inputconfcheck` allowed
both are always in the top JO
Also
* [ ] Restructure checks so that everything related to reading the jO is done in one place and everything related to reading the log is done in `logParser`S2.2020https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/101Long compilation time when running MadGraph in atlas/slc6-atlasos causing CI...2021-04-22T16:46:58+02:00Jason Robert VeatchLong compilation time when running MadGraph in atlas/slc6-atlasos causing CI timeouts## How to reproduce the problem
```
# Mount cvmfs
sudo mkdir -p /cvmfs/atlas.cern.ch
sudo mkdir -p /cvmfs/atlas-condb.cern.ch
sudo mkdir -p /cvmfs/grid.cern.ch
sudo mkdir -p /cvmfs/sft.cern.ch
sudo mount -t cvmfs atlas.cern.ch /cvmfs/at...## How to reproduce the problem
```
# Mount cvmfs
sudo mkdir -p /cvmfs/atlas.cern.ch
sudo mkdir -p /cvmfs/atlas-condb.cern.ch
sudo mkdir -p /cvmfs/grid.cern.ch
sudo mkdir -p /cvmfs/sft.cern.ch
sudo mount -t cvmfs atlas.cern.ch /cvmfs/atlas.cern.ch
sudo mount -t cvmfs atlas-condb.cern.ch /cvmfs/atlas-condb.cern.ch
sudo mount -t cvmfs grid.cern.ch /cvmfs/grid.cern.ch
sudo mount -t cvmfs sft.cern.ch /cvmfs/sft.cern.ch
# Get the docker image
docker pull atlas/slc6-atlasos
# Run image in a container and mount cvmfs
docker run -it -v /cvmfs:/cvmfs b4cfa1203c45
# Inside the docker container get the mcjoboptions repo (or alternatively you can copy it from your local area with docker cp)
kinit USER@CERN.CH
git clone https://:@gitlab.cern.ch:8443/atlas-physics/pmg/mcjoboptions.git
cd mcjoboptions
git checkout dsid_jveatch_500538
export ATLAS_LOCAL_ROOT_BASE=/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase
./scripts/run_athena.sh
```
## Debugging
#### Bottleneck: compilation time
Comparing the running times at several execution points on lxplus and in the container it seems that the problem lies on the compilation times:
```
Docker (running on a dual-core laptop with cvmfs mounted via fuse):
generate 19:24:54 INFO: Using LHAPDF v6.2.3 interface for PDFs
generate 19:26:19 INFO: Compiling source…
generate 19:31:53 INFO: ...done, continuing with P* directories => 334 sec
generate 19:31:53 INFO: Compiling StdHEP (can take a couple of minutes) ...
generate 19:45:23 INFO: …done. => 810 sec
generate 19:45:24 INFO: Compiling on 1 cores
generate 19:45:24 INFO: Compiling P0_gg_ttx...
generate 19:54:37 INFO: P0_gg_ttx done. => 553 sec
vs lxplus (interactive run)
10:15:08 INFO: Using LHAPDF v6.2.3 interface for PDFs
10:15:14 INFO: Compiling source...
10:15:26 INFO: ...done, continuing with P* directories => 12 sec
10:15:26 INFO: Compiling StdHEP (can take a couple of minutes) ...
10:16:04 INFO: …done. => 38 sec
10:16:05 INFO: Compiling on 1 cores
10:16:05 INFO: Compiling P0_gg_ttx...
10:16:45 INFO: P0_gg_ttx done. => 40 sec
```
#### Size/memory
The container available space is 53GB and where the compilation becomes slow the size of the container is ~230 MB so much smaller => **disk size does not seem to be causing the slowdown**
The available memory was changed from 1GB to 8GB without any effect on the compilation time in the container.
#### Reading from cvmfs
I run a script that 1) reads all the lines from a file that lives on cvmfs and 2) copies this script to a local directory and remove it.
The local run on my laptop (with cvmfs mounted with fuse gives this):
```
Reading 500 times
real 0m21.504s
user 0m12.937s
sys 0m8.429s
Copying 500 times
real 0m4.993s
user 0m0.620s
sys 0m2.440s
```
Running the script from the container, where the locally available cvmfs directory (see above) is mounted to the container as a volume, gives this:
```
Reading 500 times
real 1m44.217s
user 0m18.329s
sys 0m20.376s
Copying 500 times
real 0m3.716s
user 0m0.570s
sys 0m0.981s
```
**So reading a file seems to be 5x slower when running from the docker container**
#### Next steps
* [ ] To debug further we would need to know exactly how cvmfs is mounted in the gitlab runner
* [ ] Also need to check whether there is any correlation between slow reading times on cvmfs and MG - does MG call the compilers from cvmfs/reads any other info from cvmfs? Probably
---
Original report from Jason - similar issues observed with other processes which are apparently very different than this one (an NLO one and a LO one with a long decay chain)
Job [#7937441](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/7937441) failed for 9a6a4445a5bcf7ae08ac81888cccd79ef4cc4af3:
Dear experts,
The run_athena job for my branch times out. I have been trying to debug this from my side, but I am at a loss about how to proceed. The estimated execution time from each log.generate.short is ~0.1 hours, so I wouldn't expect this to be an issue. Could you please advise?
Thanks in advance,
JasonFutureSpyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/100Make -m obligatory in commit script2020-04-28T18:32:35+02:00Spyros ArgyropoulosMake -m obligatory in commit script* [x] Remove current parsing logic
* [x] Check that skipping athena,logParser works as before* [x] Remove current parsing logic
* [x] Check that skipping athena,logParser works as beforeS1.2020Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/99Throw error in CI if no `evgenConfig.nEventsPerJob` is used in the file2020-04-08T10:51:29+02:00Spyros ArgyropoulosThrow error in CI if no `evgenConfig.nEventsPerJob` is used in the filePerhaps better to incorporate into #98Perhaps better to incorporate into #98S1.2020Spyros ArgyropoulosSpyros Argyropoulos2020-04-05https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/98Extraction of nEventsPerJob from file2020-07-27T13:10:20+02:00Spyros ArgyropoulosExtraction of nEventsPerJob from file### Bug
```
nEventsPerJob=5000 -> this is what the logParser reads from the file
# I can change it here
if (efficiencyLow) nEventsPerJob*=2
evgenConfig.nEventsPerJob = nEventsPerJob -> this is what the transform will use
```
### Solut...### Bug
```
nEventsPerJob=5000 -> this is what the logParser reads from the file
# I can change it here
if (efficiencyLow) nEventsPerJob*=2
evgenConfig.nEventsPerJob = nEventsPerJob -> this is what the transform will use
```
### Solution
Credits to Frank Sauerburger
Put the following in `scriptB.py`
```
import argparse
def readParamFromJO(jOpath, param):
locals = {"evgenConfig": argparse.Namespace()}
with open(jOpath) as jOFile:
for line in jOFile.readlines():
if "os.system" in line: continue # for security
try:
exec(line, {}, locals)
except:
# print(f"fail to parse {line}") # uncomment for debugging
pass
return getattr(locals["evgenConfig"], param) if hasattr(locals["evgenConfig"], param) else None
jOFile="./source/mc.scriptA.py"
nEventsPerJob=readParamFromJO(jOFile, 'nEventsPerJob')
# Check nEventsPerJob
if nEventsPerJob is None:
print(f"WARNING: evgenConfig.nEventsPerJob is not defined in the jO. Will set to default=10000")
nEventsPerJob=10000
else:
print(f"nEventsPerJob from jO={nEventsPerJob}")
# Check minEvents
if readParamFromJO(jOFile, 'minEvents') is not None:
print(f"ERROR: {jOFile} is using deprecated parameter evgenConfig.minEvents. Please switch to evgenConfig.nEventsPerJob")
```
### Testing:
Put the following in `./source/mc.scriptA.py`
```
import Sherpa_i.Sherpa_iConf
import os
import GeneratorFilters.GeneratorFiltersConf
include("./scriptA.py") # this doesn't work because python doesn't know what include is
evgenConfig.XVAR=5
filtSeq.YVAR=10
evgenConfig.nEventsPerJob=1
evgenConfig.nEventsPerJob=2
evgenConfig.nEventsPerJob*=3
evgenConfig.nEventsPerJob=os.system("rm test")
#evgenConfig.nEventsPerJob=10
#print(f"{evgenConfig.nEventsPerJob}")
```
Running `python3 scriptB.py` gives
```
fail to parse import Sherpa_i.Sherpa_iConf
fail to parse import GeneratorFilters.GeneratorFiltersConf
fail to parse include("./scriptA.py") # this doesn't work because python doesn't know what include is
fail to parse filtSeq.YVAR=10
Final Answer: nEventsPerJob=6
```
The added bonus is that if there is no `evgenConfig.nEventsPerJob` defined this would automatically throw an error.
## What is done in ProdSys
The first occurence of `evgenConfig.nEventsPerJob` is usedS2.2020Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/97Handling of jO files processed in non-Unix filesystems2020-04-17T13:57:23+02:00Spyros ArgyropoulosHandling of jO files processed in non-Unix filesystemsFrom @avroy
> calculating [nEvents](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/blob/master/scripts/run_athena.sh#L70) failed with the following errors:
```
(standard_in) 1: illegal character: ^M
(standard_in) 1: illegal c...From @avroy
> calculating [nEvents](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/blob/master/scripts/run_athena.sh#L70) failed with the following errors:
```
(standard_in) 1: illegal character: ^M
(standard_in) 1: illegal character: ^M
```
> Naively, this is due to carriage return and not uniformly processed across operating systems
## Todo
See how to handle this:
* during commit script? : probably not ideal since not everyone uses it
* doing a `dos2unix` in the CI? : might require special image - need to see if `dos2unix` is available in the images we use
* doing a `sed 's/^M//g'` as described [here](https://stackoverflow.com/questions/2658931/why-error-illegal-character-m?answertab=votes#tab-top) in all CI jobs? : @avroy can you test whether this works for you?S1.2020Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/96Reviewing changes in control files etc2020-04-03T14:45:59+02:00Spyros ArgyropoulosReviewing changes in control files etcI am wondering whether it makes sense to introduce [CODEOWNERS](https://docs.gitlab.com/ee/user/project/code_owners.html) so that changes such as !317 can be verified by the appropriate people (e.g. in Exotics normally theory contacts wo...I am wondering whether it makes sense to introduce [CODEOWNERS](https://docs.gitlab.com/ee/user/project/code_owners.html) so that changes such as !317 can be verified by the appropriate people (e.g. in Exotics normally theory contacts would be responsible for a control file).
Tagging @amoroso @cgutscho @fsiegert @gstarkFuturehttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/95Add check to see if gridpack was used and if the grid pack is provided2020-08-04T13:43:36+02:00Spyros ArgyropoulosAdd check to see if gridpack was used and if the grid pack is provided
I was wondering how to catch such cases and avoid having pipeline jobs running for 1h and failing without apparent reason. We would need an indicator in log.generate that a gridpack was used.
I don't see PowhegConfig.gridpack printed i...
I was wondering how to catch such cases and avoid having pipeline jobs running for 1h and failing without apparent reason. We would need an indicator in log.generate that a gridpack was used.
I don't see PowhegConfig.gridpack printed in the log that Olga provided. I see
```
16:47:17 Py:PowhegControl INFO | powheginput keyword use-old-grid set to 1.0000000000000000
Does this tell us whether a gridpack was used?
```
Comment by @fsiegert
> Hi @sargyrop,
I think there are things which we'll never be able to catch if requesters modify the DSID directory before submitting but after having run the evgen test. This is not only relevant for gridpacks, but also potentially removing include files etc. So I wouldn't put too much effort into catching these cases if it's not easy.
We just need to educate users that they:
run the evgen test in a clean working directory
should not modify the DSID directory before submission
Best,
Frank
I think this is a pretty straightforward check: if ((gridpack used) && ! (gridpack present)) then ERROR So I am only asking how to specify (gridpack used)
Comment by @amoroso :
> Hi @fsiegert, @sargyrop,
I wonder if we couldn't catch case 2 within the CI. We could add a checksum to the DSID directory to the Gen_tf output, and have a pipeline check that the checksum in the attached logfile and the one recomputed by the CI are the same.
cheers, Simone
## Solution for Madgraph
GRID presence can be identified by lines like:
```
06:17:07 Py:MadGraphUtils INFO Generating events from gridpack
```S2.2020Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/94Automatic assignment of dsids2020-04-17T19:56:01+02:00Spyros ArgyropoulosAutomatic assignment of dsids* [x] implement logic to only assign blocks of DSIDs
* [x] test 1: 10 DSIDs that don't fill in a gap
* [x] test 2: continuous set of DSIDs that fills in a gap
* [x] test 3: 1 DSID should be assigned to the lowest possible DSID
**Onc...* [x] implement logic to only assign blocks of DSIDs
* [x] test 1: 10 DSIDs that don't fill in a gap
* [x] test 2: continuous set of DSIDs that fills in a gap
* [x] test 3: 1 DSID should be assigned to the lowest possible DSID
**Once the above works**
* [x] test `git fetch origin master && git checkout -b new origin/master` - if it works replace in master
* [x] implement flag for specifying branch name (`--branch`)
* [x] replace `-n` with `--dry`
* [x] implement flag for moving DSIDs but not pushing to git (`--nogit`)
* [x] keep track of the DSIDs that have changed and suggest the command to run after dry run
* [x] update list of DSIDs in commit script (provided with `-d` argument)
* [x] Update READMES1.2020Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/93CI addition to JO name check: no minus signs allowed2020-03-26T11:10:45+01:00Christian GutschowCI addition to JO name check: no minus signs allowedIt looks like the production system doesn't allow "-" in the JO name, can we get the CI to check this?It looks like the production system doesn't allow "-" in the JO name, can we get the CI to check this?S1.2020Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/92logParser: catch cases where LHE events are not enough (low efficiency)2021-06-17T13:30:43+02:00Spyros ArgyropouloslogParser: catch cases where LHE events are not enough (low efficiency)* [x] **OTF generation**: check that `N(LHE events) >= 1.1* nEventsPerJob/(filter efficiency)`
* [x] **Showering with external LHE events**: same as above
* [x] This might require different code for each generator? To be checked
* [x]...* [x] **OTF generation**: check that `N(LHE events) >= 1.1* nEventsPerJob/(filter efficiency)`
* [x] **Showering with external LHE events**: same as above
* [x] This might require different code for each generator? To be checked
* [x] Might need to take into account how many input LHE files are used per jobS1.2021Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/91Commit script: block usage of DSIDs that are already used in any remote branch2020-03-16T13:55:42+01:00Spyros ArgyropoulosCommit script: block usage of DSIDs that are already used in any remote branchCurrently the commit script only checks that a DSID is not used in remote branches **only if the DSID is outside the allowed range**.
We should extend this to cover all DSIDs, so that users who try to assign DSIDs themselves do not cr...Currently the commit script only checks that a DSID is not used in remote branches **only if the DSID is outside the allowed range**.
We should extend this to cover all DSIDs, so that users who try to assign DSIDs themselves do not create problems with other MRs.S1.2020Spyros ArgyropoulosSpyros Argyropoulos2020-03-15https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/90Job Failed #7575152 'RunArguments' object has no attribute 'inputGeneratorFile'2020-04-16T13:32:39+02:00Xiaohu SunJob Failed #7575152 'RunArguments' object has no attribute 'inputGeneratorFile'Job [#7575152](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/7575152) failed for c3035481c6ec0ec00926a14dc33427bb6b590fb1:
Dear experts,
To my understanding this relates to external LHE files. Please correct me if I am ...Job [#7575152](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/7575152) failed for c3035481c6ec0ec00926a14dc33427bb6b590fb1:
Dear experts,
To my understanding this relates to external LHE files. Please correct me if I am wrong.
The only connection between the JO / DSID and the external LHE files seems only in the spreadsheet.
The spreadsheet is not known by the CI system while uploading JOs.
Do we need some special setups to point to some afs or eos location of the external LHE files?
Thanks!
Best,
Xiaohuhttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/89logParser fail due to inputGeneratorFile argument2020-03-12T17:57:53+01:00Serena Palazzoserena.palazzo@cern.chlogParser fail due to inputGeneratorFile argumentDear all,
I just added a branch (spalazzo_600017) with 4 new JOs to register. These JOs depend on existing LHE and I tested them locally by givin the argument --inputGeneratorFile=../mc15_13TeV/410659/TXT.15180944._016918.tar.gz.1. These...Dear all,
I just added a branch (spalazzo_600017) with 4 new JOs to register. These JOs depend on existing LHE and I tested them locally by givin the argument --inputGeneratorFile=../mc15_13TeV/410659/TXT.15180944._016918.tar.gz.1. These files are therefore outside the directories with the JOs. Locally, the log parser run currectly but now that I pushed the branch I got this error:
Logfile error in log.generate: "AttributeError: 'RunArguments' object has no attribute 'inputGeneratorFile'"
PyJobTransforms.transform.execute 2020-03-12 16:49:47,279 WARNING Transform now exiting early with exit code 65 (Non-zero return code from generate (8); Logfile error in log.generate: "AttributeError: 'RunArguments' object has no attribute 'inputGeneratorFile'")
here the full pipeline: https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/7564892
I guess this error is due to the fact that inside the JO directory I should put the soft link with the LHE file (as done for the gridpacks). If this is the case, I was wondering if it would be possible/useful to add a warning in the parser check to advise whether the inputGeneratorFile is linked correctly or not.
Thanks in advance!
Cheers,
SerenaS1.2020Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/88Issue with logParser due to the nEventsPerJob2020-03-12T09:19:07+01:00Serena Palazzoserena.palazzo@cern.chIssue with logParser due to the nEventsPerJobDear all,
I am preparing an MC request and I have an issue with the parser step. In my JO I have nEventsPerJob=10000. But, if I run the following command line:
./scripts/commit_new_dsid.sh -d=600017 -n
I have the following error:
-----...Dear all,
I am preparing an MC request and I have an issue with the parser step. In my JO I have nEventsPerJob=10000. But, if I run the following command line:
./scripts/commit_new_dsid.sh -d=600017 -n
I have the following error:
---------------------
Performance metrics:
---------------------
- actual CPU (500 events) = 0.02 hrs
- CPU extrapolated to 10000 events = 0.3 hrs
- CPU = 0.33 hrs <-- ERROR: Too low CPU time - should be between 6-12h. Adjust nEventsPerJob!
- estimated CPU for CI job = 0.00 hrs
- Virtual memory = 1497.668 Mb
---------------------
Others:
---------------------
- Effective lumi (fb-1): 0.0133816141973574 <-- WARNING: low effective luminosity
- Total no. of events: 500 <-- WARNING: This total is low enough that the mu profile may be problematic - INFORM MC PROD
---------------------
Summary:
---------------------
Errors : 1 , Warnings : 2 -> Errors encountered! Not ready for production!
ERROR: log.generate contains errors
Fix them before committing anything!
So it seems that 10k is not enough. But on the other hand, I cannot increase the number because otherwise there will be issues in production (according to @dhirsch). How can I solve this issue?
Attached you can find the log.generate and the JO.
Thanks in advance!
Cheers,
Serena[log.generate](/uploads/c0da6de6b6c075e818791e4216e1b511/log.generate)[mc.PhH7EG_H7UE_716_tchan_lept_antitop.py](/uploads/32012a54748990b2cfc5f9fb44b46561/mc.PhH7EG_H7UE_716_tchan_lept_antitop.py)S1.2020Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/87Allo jO files to be links in whitelist2020-03-09T15:22:42+01:00Spyros ArgyropoulosAllo jO files to be links in whitelistAs needed in !265
`mc.*.py` should be allowed as a link tooAs needed in !265
`mc.*.py` should be allowed as a link tooS1.2020Spyros ArgyropoulosSpyros Argyropoulos