Skip to content

WIP: Take nEventsPerJob from jO file if -t is used [skip modfiles]

Spyros Argyropoulos requested to merge logParser into master

Implementation

When running test jobs to determine the optimal number of events logParser would throw an error from the CPU information if for instance the number of events would be too low. This would prevent someone from committing unless they would tinker with log.generate (discouraged) or they would rerun the whole job (not what we want).

This MR introduces a new feature, where nEventsPerJob is read directly from the jO file instead of log.generate. It exploits the already existing -t option which was used to skip the check that nEventsPerJob in log.generate is equal to the number of events written in the EVNT file.

So now when someone runs with python scripts/logParser.py -i ... **-t** they would

  • disable the nEventsPerJob==CountHEPMC check
  • overwrite nEventsPerJob in log.generate with evgenConfig.nEventsPerJob as read from the jO file

I also removed opts.TOTAL_EVENTS which was taken from "Requested output events" and was a relic of the past that could be confusing. When using -t everything (e.g. also the effective lumi) is scaled from nEventsPerJob in the jO file.

Testing

@fsiegert can you check if this is what you had in mind?

When I run over your jO (the one you mentioned in #73 (closed)) I get:

python scripts/logParser.py -i 700xxx/700008/log.generate -t

---------------------
jobOptions and release:
---------------------
- jobOptions =   
  mc.Sh_228_yyy_0NLO12LO_EW.py 
- using release =   AthGeneration-21.6.18 

---------------------
Generate params:
---------------------
- nEventsPerJob (from jO file) =  1000 <-- WARNING: This total is low enough that the mu profile may be problematic - INFORM MC PROD 
- nEventsPerJob (from log.generate) =  100 
- transform =   Gen_tf 
- inputFilesPerJob =   0 

---------------------
Metadata:
---------------------
- physicsComment =   Sherpa yyy + 0j@NLO + 1,2j@LO 
- generatorName =   Sherpa(v.2.2.8p3) 
- generatorTune =   NNPDF3.0 NNLO 
- keywords =   sm, diphoton 
- specialConfig =   NONE 
- contactPhysicist =   atlas-generators-sherpa@cern.ch, frank.siegert@cern.ch 
- cross-section (nb) =   0.00035264 
- generator =   Sherpa2.2.8 
- PDF =   LHA[NNPDF30_nnlo_as_0118] 
- GenFiltEff =   1.000000e+00 
- sumOfNegWeights =   0.000000e+00 
- sumOfPosWeights =   1.022592e+02 
---------------------
MISSING Metadata:
INFO:  genFilterNames is missing 
INFO:  weights is missing 

-------------------------
Generator specific tests: Sherpa(v.2.2.8p3)
-------------------------
- retried events Jet_Evolution:CSS =   0.2 % 

---------------------
Event tests:
---------------------
- TestHepMC Events passed =   100, Events Failed  0 
- TestHepMC Efficiency =   100% 
- EvgenFilterSeq Weighted Filter Efficiency =   1.000000 [102.259187 / 102.259187] 
- EvgenFilterSeq Filter Efficiency =   1.000000 [100 / 100] 
- CountHepMC Events passing all checks and written =   100 
- SimTimeEstimate RUN INFORMATION =   Processed 100 events and found 3.05463e+08 of energy to be processed. 

---------------------
Performance metrics:
---------------------
- actual CPU (100 events) = 0.88 hrs
- CPU extrapolated to 1000 events =  8.8 hrs 
- estimated CPU for CI job = 0.01 hrs
- Virtual memory =   2221.098 Mb 

---------------------
 Others:
---------------------
- Effective lumi (fb-1) (1000 events):  2.8357531760435575 <-- WARNING: low effective luminosity 

---------------------
 Summary:
---------------------
Errors : 0 , Warnings : 2   -> Some warnings encountered, check that these are ok before submitting production! 

Notice that nEventsPerJob is taken from the jO file and CPU is extrapolated from what is in log.generate (100) to what is in the jO file (1000).

Closes #73 (closed)

Edited by Frank Siegert

Merge request reports