Skip to content

Crash in HLepRare_skim_dev Preprocess step for Tau_2022E

Issue Description

As reported by @fbilandz there seems to be a problem with the Preprocessing of Tau_2022E using the HLepRare_skim_dev branch. The crash is the "classic" RDataFrame::Run: event loop was interrupted with little additional information:

RDataFrame::Run: event loop was interrupted
RDataFrame::Run: event loop was interrupted
ERROR: [pid 2087939] Worker Worker(salt=307914599, workers=25, host=hercules.mib.infn.it, username=brivio, pid=2087914) failed    PreprocessRDF(effective_workflow=local, branch=0, version=test_Tau2022E, request_cpus=1, config_name=run3_2022_postEE, category_name=base, dataset_name=Tau_2022E, tree_name=Events, custom_output_tag=_, request_memory=-1, logs_to_eos=False, shared_preprocessed=False, max_memory=, allow_redefinition=False, systematic=, systematic_direction=, modules_file=run3_preprocess_modulesrdf, keep_and_drop_file=run3_keep_and_drop_file, compute_filter_efficiency=False)
Traceback (most recent call last):
  File "../CCLUbbtautauAnalysis/nanoaod_base_analysis/cmt/base_tasks/base.py", line 731, in Snapshot
    self.rdf.Snapshot(*args)
TypeError: Template method resolution failed:
  none of the 3 overloaded methods succeeded. Full details:
  ROOT::RDF::RResultPtr<ROOT::RDF::RInterface<ROOT::Detail::RDF::RLoopManager,void> > ROOT::RDF::RInterface<ROOT::Detail::RDF::RJittedFilter,void>::Snapshot(basic_string_view<char,char_traits<char> > treename, basic_string_view<char,char_traits<char> > filename, initializer_list<string> columnList, const ROOT::RDF::RSnapshotOptions& options = ROOT::RDF::RSnapshotOptions()) =>
    TypeError: could not convert argument 3
  ROOT::RDF::RResultPtr<ROOT::RDF::RInterface<ROOT::Detail::RDF::RLoopManager,void> > ROOT::RDF::RInterface<ROOT::Detail::RDF::RJittedFilter,void>::Snapshot(basic_string_view<char,char_traits<char> > treename, basic_string_view<char,char_traits<char> > filename, const vector<string>& columnList, const ROOT::RDF::RSnapshotOptions& options = ROOT::RDF::RSnapshotOptions()) =>
    runtime_error: Index below bounds in Binning for input argument 5 value: -1.000000
  ROOT::RDF::RResultPtr<ROOT::RDF::RInterface<ROOT::Detail::RDF::RLoopManager,void> > ROOT::RDF::RInterface<ROOT::Detail::RDF::RJittedFilter,void>::Snapshot(basic_string_view<char,char_traits<char> > treename, basic_string_view<char,char_traits<char> > filename, basic_string_view<char,char_traits<char> > columnNameRegexp = "", const ROOT::RDF::RSnapshotOptions& options = ROOT::RDF::RSnapshotOptions()) =>
    TypeError: could not convert argument 3
  Failed to instantiate "Snapshot(std::string,std::string,std::initializer_list<std::string*>)"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "../CCLUbbtautauAnalysis/nanoaod_base_analysis/data/software/lib/python3.9/site-packages/luigi/worker.py", line 199, in run
    new_deps = self._run_get_new_deps()
  File "../CCLUbbtautauAnalysis/nanoaod_base_analysis/data/software/lib/python3.9/site-packages/luigi/worker.py", line 141, in _run_get_new_deps
    task_gen = self.task.run()
  File "../CCLUbbtautauAnalysis/nanoaod_base_analysis/data/software/lib/python3.9/site-packages/law/decorator.py", line 231, in wrapper
    result = call(state)
  File "../CCLUbbtautauAnalysis/nanoaod_base_analysis/data/software/lib/python3.9/site-packages/law/decorator.py", line 388, in call
    return fn(task, *args, **kwargs)
  File "../CCLUbbtautauAnalysis/nanoaod_base_analysis/data/software/lib/python3.9/site-packages/law/decorator.py", line 244, in wrapper
    return decorator(fn, _opts, *args, **kwargs)
  File "../CCLUbbtautauAnalysis/nanoaod_base_analysis/data/software/lib/python3.9/site-packages/law/decorator.py", line 540, in localize
    return fn(task, *args, **kwargs)
  File "../CCLUbbtautauAnalysis/nanoaod_base_analysis/cmt/base_tasks/preprocessing.py", line 445, in run
    snapshot_ensuring_output_tree(filtered_df, self.tree_name, create_file_dir(outp.path), branches)
  File "../CCLUbbtautauAnalysis/nanoaod_base_analysis/cmt/base_tasks/base.py", line 154, in snapshot_ensuring_output_tree
    df.Snapshot(tree_name, output_path, branches)
  File "../CCLUbbtautauAnalysis/nanoaod_base_analysis/cmt/base_tasks/base.py", line 737, in Snapshot
    self.rdf.Snapshot(*args)
TypeError: Template method resolution failed:
  none of the 3 overloaded methods succeeded. Full details:
  ROOT::RDF::RResultPtr<ROOT::RDF::RInterface<ROOT::Detail::RDF::RLoopManager,void> > ROOT::RDF::RInterface<ROOT::Detail::RDF::RJittedFilter,void>::Snapshot(basic_string_view<char,char_traits<char> > treename, basic_string_view<char,char_traits<char> > filename, initializer_list<string> columnList, const ROOT::RDF::RSnapshotOptions& options = ROOT::RDF::RSnapshotOptions()) =>
    TypeError: could not convert argument 3
  ROOT::RDF::RResultPtr<ROOT::RDF::RInterface<ROOT::Detail::RDF::RLoopManager,void> > ROOT::RDF::RInterface<ROOT::Detail::RDF::RJittedFilter,void>::Snapshot(basic_string_view<char,char_traits<char> > treename, basic_string_view<char,char_traits<char> > filename, const vector<string>& columnList, const ROOT::RDF::RSnapshotOptions& options = ROOT::RDF::RSnapshotOptions()) =>
    runtime_error: Index below bounds in Binning for input argument 5 value: -1.000000
  ROOT::RDF::RResultPtr<ROOT::RDF::RInterface<ROOT::Detail::RDF::RLoopManager,void> > ROOT::RDF::RInterface<ROOT::Detail::RDF::RJittedFilter,void>::Snapshot(basic_string_view<char,char_traits<char> > treename, basic_string_view<char,char_traits<char> > filename, basic_string_view<char,char_traits<char> > columnNameRegexp = "", const ROOT::RDF::RSnapshotOptions& options = ROOT::RDF::RSnapshotOptions()) =>
    TypeError: could not convert argument 3
  ROOT::RDF::RResultPtr<ROOT::RDF::RInterface<ROOT::Detail::RDF::RLoopManager,void> > ROOT::RDF::RInterface<ROOT::Detail::RDF::RJittedFilter,void>::Snapshot(basic_string_view<char,char_traits<char> > treename, basic_string_view<char,char_traits<char> > filename, initializer_list<string> columnList, const ROOT::RDF::RSnapshotOptions& options = ROOT::RDF::RSnapshotOptions()) =>
    TypeError: could not convert argument 3
INFO: Informed scheduler that task   PreprocessRDF_False_0__2ac0a73f12   has status   FAILED
INFO: Worker Worker(salt=307914599, workers=25, host=hercules.mib.infn.it, username=brivio, pid=2087914) was stopped. Shutting down Keep-Alive thread
INFO: 
===== Luigi Execution Summary =====

Scheduled 2 tasks of which:
* 1 complete ones were encountered:
    - 1 InputData(...)
* 1 failed:
    - 1 PreprocessRDF(...)

This progress looks :( because there were failed tasks

===== Luigi Execution Summary =====

Apparently with other datasets this crash does not happen.

The crash can be reproduced with:

law run PreprocessRDF --version test_Tau2022E \
                      --config-name run3_2022_postEE  \
                      --category-name base \
                      --dataset-name Tau_2022E \
                      --PreprocessRDF-modules-file run3_preprocess_modulesrdf \
                      --PreprocessRDF-keep-and-drop-file run3_keep_and_drop_file \
                      --workers 25 \
                      --branch 0 \
                      --print-status -1