Skip to content

Changes to handling of nEventsPerJob

Spyros Argyropoulos requested to merge logParser3 into master

Changes

  • Removal of -t option
  • Modification of check CountHepMC == nEventsPerJob to CountHepMC == Requested output events (which should always be satisfied, hence the removal of the -t option)
  • nEventsPerJobFromJO is introduced and always read from the jO file
  • The extrapolation of the CPU time now happens automatically if nEventsPerJobFromJO != nEventsPerJobFrom (where the latter is read from log.generate)

Tests

Using 700078 prepared by @fsiegert:

  1. Using -maxEvents=100 and NEventsPerJob=1000, committing with the commit script
./scripts/commit_new_dsid.sh -d=700078 -n
Generate params:
---------------------
- nEventsPerJob =   1000 
- Requested output events =   100 
- transform =   Gen_tf 
- inputFilesPerJob =   0 
...
---------------------
Performance metrics:
---------------------
- actual CPU (100 events) = 9.71 hrs
- CPU extrapolated to 1000 events =  97.1 hrs 
- CPU =   97.10 hrs  <-- ERROR: Too high CPU time - should be between 6-12h. Adjust nEventsPerJob! 
- estimated CPU for CI job = 0.97 hrs
  1. Standalone logParser run without the jO present in the directory
    logParser internally assigns nEventsPerJobFromJO=nEventsPerJob (where nEventsPerJob is read from the evgenConfig.nEventsPerJob printed in log.generate).
python scripts/logParser.py -i 700xxx/700078/log.generate
---------------------
Generate params:
---------------------
- nEventsPerJob =   1000 
- Requested output events =   100 
- transform =   Gen_tf 
- inputFilesPerJob =   0 
...
---------------------
Performance metrics:
---------------------
- actual CPU (100 events) = 9.71 hrs
- CPU extrapolated to 1000 events =  97.1 hrs 
- CPU =   97.10 hrs  <-- ERROR: Too high CPU time - should be between 6-12h. Adjust nEventsPerJob! 
- estimated CPU for CI job = 0.97 hrs
  1. Standalone logParser run pointing to a jO file in a different directory
    Here I put the jO file in 999999 and used evgenConfig.nEventsPerJob = 100. The jO in 700078 still has evgenConfig.nEventsPerJob = 1000:
python scripts/logParser.py -i 700xxx/700078/log.generate -j 999xxx/999999/mc.Sh_228_yyy_01NLO.py
---------------------
Generate params:
---------------------
- nEventsPerJob =   100 
- Requested output events =   100 
- transform =   Gen_tf 
- inputFilesPerJob =   0 
...
---------------------
Performance metrics:
---------------------
- CPU =   9.71 hrs 
- estimated CPU for CI job = 0.10 hrs
- Virtual memory =   3061.180 Mb 

Todo

  • Update README

Closes #73 (closed), #81 (closed)

Edited by Spyros Argyropoulos

Merge request reports