Segmentation violation when trying to remove GEC from SingleHighPTMuon HLT1 line (non-deterministic)

Report of a crash/segfault with Allen

Brief summary of encountered problem

Tried to add a new HLT1 line with no GEC and I see seg faults/failures/sometimes it works. Feels indeterministic/transient.

Dataset you were running on

Used a single input file (Z2mumu signal MC - roughly 5000 events).

Allen version used (or nightly, as appropriate)

Allen branch: "rjhunter-remove-gec-from-highptmuon". See !664 (closed)

Please include the platform Allen was compiled for (CPU or GPU as well as the type of build)

CPU default with lb-stack setup ("x86_64_v2-centos7-gcc11-opt")

Option file used to execute Allen

moore_options_file.py terminal command: MooreAnalysis/run gaudirun.py moore_options_file.py

Additional information

I compiled Allen standalone to regenerate the sequence and then copied the sequence to replace hlt1_pp_default.json in the lb-stack. I then recompiled afterwards.

Steps after following instructions to compile Allen standalone:

git submodule update --init --recursive
git fetch origin rjhunter-remove-gec-from-highptmuon
git checkout -b rjhunter-remove-gec-from-highptmuon FETCH_HEAD
mkdir build
cd build
cmake -DSTANDALONE=ON -DSEQUENCES=hlt1_pp_default ..
make

2. Run /Allen --sequence hlt1_pp_default

3. Copy that Sequence.json into stack replacing its pregenerated sequence e.g. cp Sequence.json /path/to/stack/Allen/configuration/pregenerated/hlt1_pp_default.json

make Allen/purge
make MooreAnalysis 

5. Ran the job using the attached script.

Log files:

moore_options_file_result__run_did_not_end_+seg_fault.txt

moore_options_file_result__run_ended_+no_seg_faults.txt

moore_options_file_result__run_ended_+seg_fault.txt

Moreover, Saw similar problems using upgrade_DC19_01_Z2mumuMU from TestFileDB. Options file attached below:

test_file_db_moore_options_file.py

test_file_db_moore_options_file_result__1_.txt

test_file_db_moore_options_file_result__2_.txt

FYI @rjhunter @mvesteri @wbarter