Skip to content

Fitting toys on the batch system

Hello everyone,

A week ago I tried to use the the submit_toys.py in analysis-tools/analysis/toys to speed up the fitting of the toys. The .yaml file I used has the following structure:


name: batch-fitted-toys-RealStat-RUN-Q2BIN-TRIGGER-BREM-0.95-VRM-VCM-SI-EFF-Q2SEL-MODELID-B_DTF_PV_M_0
fit:
    nfits: 10
    minos: False
    extended: True
    nfits-per-job: 10
    models: model
    strategies: simple

model:
    categories: category
    pdf:
        B02Kst0Jpsi2ee:

The jobs failed and the log files show the following error:

------------------------------------------------------------------------
Job started on mer 27 set 2017, 09.21.44, CEST
------------------------------------------------------------------------
/home/hep/matzeni/gangadir/Analysis/Systematics/PartSys/RealStatistics
/disk/gangadir/matzeni/Analysis/Systematics/PartSys/RealStatistics

^[[1mRooFit v3.60 -- Developed by Wouter Verkerke and David Kirkby^[[0m
                Copyright (C) 2000-2013 NIKHEF, University of California & Stanford University
                All rights reserved, please read http://roofit.sourceforge.net/license.txt

2017-09-27 09:21:52,220 - analysis.toys.fit | ^[[31mERROR   ^[[39;49;0m | ^[[31mMissing model configuration -> 'm'^[[39;49;0m
2017-09-27 09:21:52,220 - analysis.toys.fit | ^[[31mERROR   ^[[39;49;0m | ^[[31mBad configuration given^[[39;49;0m
Traceback (most recent call last):
  File "/disk/gangadir/matzeni/Analysis/analysis-tools/analysis/toys/fit_toys.py", line 363, in main
    run(args.config, args.link_from, args.verbose)
  File "/disk/gangadir/matzeni/Analysis/analysis-tools/analysis/toys/fit_toys.py", line 142, in run
    raise KeyError("Missing model configuration")
KeyError: 'Missing model configuration'^[[39;49;0m
------------------------------------------------------------------------
Job ended on mer 27 set 2017, 09.21.52, CEST
------------------------------------------------------------------------

The error seems to arise from the fact that around l.142 in fit_toys.py we have:

    try:
        models = {model_name: config[model_name]
                  for model_name
    -------->     in config['fit'].get('models', ['model'])}
    except KeyError as error:
        logger.error("Missing model configuration -> %s", str(error))

where config['fit'].get('models', ['model']) should be a list and not a string.

However if I modify the configuration file in the following way (notice that this is the usual syntax):

name: batch-fitted-toys-RealStat-RUN-Q2BIN-TRIGGER-BREM-0.95-VRM-VCM-SI-EFF-Q2SEL-MODELID-B_DTF_PV_M_0-testforlist
fit:
    nfits: 10
    minos: False
    extended: True
    nfits-per-job: 10
    models:
        - model
    strategies:
        - simple

model:
    categories:
        - category
    pdf:

the submission fails with the following error:

2017-10-04 10:41:31,092 - analysis.toys.submit | ERROR    | Uncaught exception -> TypeError("unhashable type: 'list'",)
Traceback (most recent call last):
  File "/home/hep/matzeni/gangadir/Analysis/analysis-tools/analysis/toys/submit_toys.py", line 273, in main
    args.overwrite).run(script_to_run)
  File "/disk/gangadir/matzeni/Analysis/analysis-tools/analysis/toys/submitter.py", line 123, in run
    if _config.compare_configs(config, config_dest).difference(set(self.ALLOWED_CONFIG_DIFFS)):
  File "/disk/gangadir/matzeni/Analysis/analysis-tools/analysis/utils/config.py", line 146, in compare_configs
    return {key for key, _
TypeError: unhashable type: 'list'

due to the fact that we are trying to convert into sets a quantity that contains list.

I remember I observed this problem with @jeschle in the past but I cannot remember if we managed to fix this!

How should we fix this?

Cheers,

Michele