Segmentation violation when running HLT1 on MC simulation

added Crash label

Just to add, I was able to trace the same error with gdb till here:

#34 0x00007f5a8e062171 in DataSvc::retrieveObject (this=0x9b14ca0, pRegistry=<optimized out>, path=..., pObject=@0x7f5a763f16d8: 0x0) at ../GaudiKernel/src/Lib/DataSvc.cpp:630
630      StatusCode status = i_retrieveEntry( parent, path, result );
(gdb) print(path)
$8 = {_M_len = 16, _M_str = 0x5a189820 "/Event/MC/Header"}

Full log from my side weird_error.log, I can see it with both MiniBrunel_2018_MinBias_FTv4_DIGI and Upgrade_BsPhiPhi_MD_FTv4_DIGI. Options files options

Thanks for reporting @jedavies, I will try to take a look in the next days.

Just following up on this. Is this still being looked into?

Yes, sorry for being slow on this. With ongoing FEST I have not found time yet. But it's high in my list.

The problem always shows up when writing a ROOT file as output when calling run_allen (regardless of the input).

I think that it is linked to the way output writing is now linked to streams here. But I'm not quite sure what is going wrong, since the setup looks similar as for the moore_control_flow. The only differences I see are

in moore_control_flow the stream writers post algs are combined with prefilters in a CompositeNode with AND and then all streams are combined in a CompositeNode with OR (here and lines before), while in allen_control_flow the stream writer post algorithms are "only" added to the list of algorithms in the main CompositeNode with AND here. Since we only have the default stream in the Allen HLT1 case, I don't quite see how this could cause a difference.
No extra_outputs are set when calling the stream_writer(...) function in allen_control_flow, while they are set for Hlt2 and sprucing case in moore_control_flow. But since the Moore HLT1 works w/o problems with DST output, I don't think this is the cause.

Maybe I overlooked something and @nskidmor or @rmatev have advice?

Once the problem is understood I will add a test similar to dst_write for Moore HLT1, that should catch it in the future.

I can take a look today

Thank you @nskidmor !

The problem lies in stream_writer. Im debugging now :)

This is the issue. @jedavies can you comment-out such that Gaudi__Hive__FetchLeavesFromFile() is not run and confirm whether or not this fixes the problem.

I wonder if when running hlt1 we need to "# Always copy the locations from the input" using Gaudi__Hive__FetchLeavesFromFile(). @dovombru what do you think?

Sure. If Gaudi__Hive__FetchLeavesFromFile() is removed what should input_leaves be instead?

Can you try just commenting out this line

I commented out lines 414 and 417 and it complained that it didn't know what input_leaves was so i commented out also line 422 but that gave

  File "/afs/cern.ch/work/j/jedavies/public/stack/Gaudi/InstallArea/x86_64_v2-centos7-gcc11-opt/bin/gaudirun.py", line 584, in <module>
    exec(o, g, l)
  File "<string>", line 1, in <module>
  File "/afs/cern.ch/work/j/jedavies/public/stack/Gaudi/InstallArea/x86_64_v2-centos7-gcc11-opt/bin/gaudirun.py", line 543, in __call__
    importOptions(arg)
  File "/afs/cern.ch/work/j/jedavies/public/stack/Gaudi/InstallArea/x86_64_v2-centos7-gcc11-opt/python/GaudiKernel/ProcessJobOptions.py", line 552, in importOptions
    _import_function_mapping[ext](optsfile)
  File "/afs/cern.ch/work/j/jedavies/public/stack/Gaudi/InstallArea/x86_64_v2-centos7-gcc11-opt/python/GaudiKernel/ProcessJobOptions.py", line 486, in _import_python
    exec(code, {})
  File "/afs/cern.ch/work/j/jedavies/public/stack/Moore/Hlt/Hlt1Conf/options/allen_hlt1_pp_default.py", line 15, in <module>
    run_allen(options)
  File "/afs/cern.ch/work/j/jedavies/public/stack/Moore/Hlt/Moore/python/Moore/config.py", line 716, in run_allen
    top_cf_node = allen_control_flow(options)
  File "/afs/cern.ch/work/j/jedavies/public/stack/Moore/Hlt/Moore/python/Moore/config.py", line 694, in allen_control_flow
    pre_algs, post_algs = stream_writer(
  File "/afs/cern.ch/work/j/jedavies/public/stack/Moore/Hlt/Moore/python/Moore/config.py", line 420, in stream_writer
    root_copy_input_writer(
  File "/afs/cern.ch/work/j/jedavies/public/stack/LHCb/InstallArea/x86_64_v2-centos7-gcc11-opt/python/PyConf/application.py", line 155, in root_copy_input_writer
    writer = CopyInputStream(
  File "/afs/cern.ch/work/j/jedavies/public/stack/LHCb/InstallArea/x86_64_v2-centos7-gcc11-opt/python/PyConf/tonic.py", line 644, in _configurable_wrapper
    return wrapped(**kwargs)
  File "/afs/cern.ch/work/j/jedavies/public/stack/LHCb/InstallArea/x86_64_v2-centos7-gcc11-opt/python/PyConf/importers.py", line 33, in wrapped
    return wrapper(component_type, **kwargs)
  File "/afs/cern.ch/work/j/jedavies/public/stack/LHCb/InstallArea/x86_64_v2-centos7-gcc11-opt/python/PyConf/components.py", line 528, in __new__
    _check_input_integrity(alg_type, _inputs, kwargs, input_transform)
  File "/afs/cern.ch/work/j/jedavies/public/stack/LHCb/InstallArea/x86_64_v2-centos7-gcc11-opt/python/PyConf/components.py", line 175, in _check_input_integrity
    raise TypeError(
TypeError: Expected DataHandle (or Algorithm) properties for <class 'GaudiCommonSvc.GaudiCommonSvcConf.CopyInputStream'> but got {'InputFileLeavesLocation': ['/Event/DAQ/RawEvent']}

But leaving 414 uncommented seems to make it work.

ok so now we need to try and understand this

Thanks a lot for tracing this down @nskidmor!

What does Gaudi__Hive__FetchLeavesFromFile() do exaxtly?

I think its required to persist the MC objects mainly. In which case I guess we do require it. Im still very confused. Im using the following to try and debug this alg

Click for snippet

from PyConf.Algorithms import Gaudi__Hive__FetchLeavesFromFile
from PyConf.application import configure_input, configure, root_copy_input_writer
from PyConf.control_flow import CompositeNode
from Moore import options

options.input_files = [
    '/eos/lhcb/grid/prod/lhcb/MC/Upgrade/XDIGI/00125247/0000/00125247_00000002_1.xdigi'
    #'root://eoslhcb.cern.ch//eos/lhcb/grid/prod/lhcb/MC/Upgrade/LDST/00075219/0000/00075219_00000001_1.ldst'
]
options.input_type = 'ROOT'
options.input_raw_format = 0.3
options.simulation = True
options.data_type = 'Upgrade'
options.conddb_tag = "sim-20201218-vc-md100"
options.dddb_tag = "dddb-20201211"
options.evt_max = 5


def test():
    config = configure_input(options)
    input_leaves = Gaudi__Hive__FetchLeavesFromFile()
    writer = root_copy_input_writer("test.dst", input_leaves)

    moore_children = [input_leaves] + [writer]
    moore = CompositeNode('moore', children=moore_children, force_order=True)
    config.update(configure(options, moore))

    return config


test()

Im none the wiser currently as the above snippet works and I can read the output

EventSelector     SUCCESS Reading Event record 1. Record number within stream 1: 1
/Event
/Event/Gen
/Event/PrevPrev
/Event/Prev
/Event/Next
/Event/NextNext
/Event/MC
/Event/Link
/Event/pSim
/Event/DAQ
>>>

I think I need some support on this - @rmatev, @clemenci. Do you have insights?

I had to check because I could not remember what Gaudi::Hive::FetchLeavesFromFile is.

If I'm not mistaken it is an algorithm that detects which entries in a transient store come from an input file, so that later we can use this list to know which entries have been produced in the current job and which were read from a file. To recursively collect all entries from the file it also has to read every object from the file.

From the stack trace, I would say that the problem come from the application not being able to read an entry from the file, so the problem is not really in FetchLeavesFromFile but on the actual reading. Disabling FetchLeavesFromFile (apart from the issue with the copy algorithm) would just hide the until something tries to read the problematic entry.

The warning

Warning in <TClass::GetStreamer>: For SmartRefBase, the TClassStreamer passed does not properly implement the Generate method (N9GaudiRoot9IOHandlerI12SmartRefBaseEE vs 14TClassStreamer)

looks pretty much related to the actual segfault.

Do I understand correctly that the problem is reading an entry that existed in the input file? I.e. not something that was produced in the job running?

Then I wonder what could be the difference between the Moore and Allen HLT1 calls. Because they both read data from the file. Would it show up if different data is used in the two applications?

yes, the segfault is in the part of the call stack where ROOT tries to decode data from disk.

The warning I highlighted seems to say that the instruction we are giving ROOT to decode SmartRefBase from file are not correct. Something like that might happen if the version of SmartRefBase used to write to disk is different from what we use to read (although I would expect a different error).

Given that @valukash sees this with other files some versioning problem seems reasonable. @jedavies could you try running HLT2 with one of the xdigi files? And let us know if that works? The output type needs to be ROOT

FYI: I also tried with the default input for running Allen, which fails, and running Moore, which works.

So I'm not sure it is linked to the input file.

Im afraid Im still none the wiser on this. I have tried everything I can think of to debug. The file itself I can open and read just fine

Just for transparency, I did try running HLT2 on an xdigi and that seemed to work.

I just tried doing

./run gaudirun.py Hlt/Moore/tests/options/default_input_hlt1.py Hlt/Hlt1Conf/tests/options/allen_hlt1_mdf_output.py  Hlt/Hlt1Conf/options/allen_hlt1_pp_default.py

i.e. the initial line that I was running but with the output type as mdf rather than dst and that seemed to finish successfully ...

mentioned in merge request !1103 (merged)

Hi, I just want to report that this affects also MooreAnalysis, for example when running

MooreAnalysis/run MooreAnalysis/HltEfficiencyChecker/scripts/hlt_eff_checker.py debug_only_hlt1.yaml --ignore-broken-inputs

where the content of debug_only_hlt1.yaml is:

annotated_decay_descriptor:
    "(${B}[B- => ${D}(D0 => ${P1}K- ${P2}pi+) ${mu}mu- Nu ]CC) || (${B}[B- => (Xc ==> ${D}(D0 => ${P1}K- ${P2}pi+) (pi0|gamma) { pi0 }) ${mu}mu- Nu ]CC) || (${B}[B- => (Xc ==> ${D}(D0 => ${P1}K- ${P2}pi+) pi+ pi-) ${mu}mu- Nu ]CC) || ([${B}B- => ${D}(D0 => ${P1}K- ${P2}pi+) (tau- ==> ${mu}mu- Nu Nu) Nu ]CC)"
ntuple_path: &NTUPLE debug-hlt1_ntuple.root
job:
    trigger_level: 1
    evt_max: 10000
    ft_decoding_version: 6
    input_raw_format: 0.3
    input_type: 'ROOT'
    conddb_tag: sim-20210617-vc-mu100
    dddb_tag: dddb-20210617
    allen_sequence: hlt1_pp_default
    run_reconstruction: true
    input_files: [# 'root://x509up_u75343@lhcbxrootd-kit.gridka.de//pnfs/gridka.de/lhcb/LHCb-Disk/lhcb/MC/Upgrade/XDIGI/00150598/0000/00150598_00000001_1.xdigi',
'root://x509up_u75343@eoslhcb.cern.ch//eos/lhcb/grid/prod/lhcb/MC/Upgrade/XDIGI/00150598/0000/00150598_00000008_1.xdigi',
# 'root://xrootd.echo.stfc.ac.uk/lhcb:prod/lhcb/MC/Upgrade/XDIGI/00150598/0000/00150598_00000009_1.xdigi',
]
    options:
        - $HLTEFFICIENCYCHECKERROOT/options/options_template.py.jinja  # first rendered with jinja2
analysis:
    script: $HLTEFFICIENCYCHECKERROOT/scripts/hlt_line_efficiencies.py
    args:
        input: *NTUPLE
        reconstructible_children: P1,P2,mu
        legend_header: "B^{#minus} #to D^{0}(K#pi) #mu X"
        make_plots: true
        vars: "PT"
        lines: Hlt1TwoTrackMVADecision,Hlt1TrackMVADecision
        true_signal_to_match_to: "B"

Hi, also for the HLT1+HLT2 chaining in MooreAnalysis. Attached is a log with stack trace and the yaml that I used. 11166071.yaml 11166071_100000_fail.log

which makes it very hard to study the effect of current HLT1 implementation on HLT2 lines in terms of efficiencies and rates. The alternative is using hlt1_filtered sample which is very old and seems to only contain the TrackMVA lines

I agree this is a quite critical problem, not least because we need to move towards being able to run HLT1 in MC productions. The coordination team is aware of the situation and we will rediscuss as a matter of priority at our next meeting on Monday.

mentioned in merge request Allen!772 (merged)

Short summary of my investigations on this. (Spoiler: I have not found the reason for the crash yet.)

@nskidmor and I were wondering if the difference between the allen and moore hlt1 configuration could be ordering of algorithms. Therefore, I dumped the control flow and data flow charts for the two cases and then modified the allen configuration slightly to make it as similar as possible to the moore one.

I ran the following to produce the charts, on branch dovombru_fix_root_segmentation_fault within the Moore directory of a stack setup:

./run gaudirun.py Hlt/Moore/tests/options/default_input_and_conds_hlt1.py Hlt/Hlt1Conf/tests/options/allen_hlt1_dst_output.py Hlt/Hlt1Conf/options/allen_hlt1_pp_default.py -> crash
./run gaudirun.py Hlt/Moore/tests/options/default_input_and_conds_hlt1.py Hlt/Hlt1Conf/tests/options/hlt1_dst_output.py Hlt/Hlt1Conf/options/hlt1_pp_default.py -> no crash

The control flow charts should be the same now, but the moore hlt1 node is much more complex of course, as all the allen hlt1 configuration happens within the RunAllen algorithm.

The data flow charts also look similar to me, one difference being the configuration of CopyInputStream, which takes its input from Gaudi__Hive__FetchLeavesFromFile. In the Moore case, TESVetoList = ['/Event/HltLumiWriter/RawEventLocation', '/Event/[...]tRawEvent', '/Event/HltSelReportsWriter/RawEvent'], while in the Allen case TESVetoList = ['/Event/AllenReportsToRawEvent/OutputRawReports']. The AllenReportsToRawEvent/OutputRawReports contains the SelReports and DecReports.

If anyone has ideas / a detective's eye, I'm happy for any input. @clemenci maybe you have more ideas on the ROOT file reading given the configuration in the flow charts?

moore_control_flow.pdf

moore_data_flow.pdf

03_allen_control_flow.pdf

03_allen_data_flow.pdf

mentioned in merge request !1224 (merged)

mentioned in merge request Allen!775 (merged)

I was on a wrong path previously, finally tracked it down to Allen's ROOTService and changes introduced with Allen!729 (merged). A call to ROOT::EnableImplicitMT(); was added with that MR, which apparently screwed up the DST writing in Moore.

Allen!775 (merged) should be the fix.

closed

unassigned @jedavies

Segmentation violation when running HLT1 on MC simulation

Report of a crash/segfault with Moore

Brief summary of encountered problem

Dataset you were running on

Moore version used (or nightly, as appropriate)

Option file used to execute Moore

Additional information

Designs

Child items ...

Activity