Add an option file to run Sprucing throughput test

Add an option file spruce_all_lines.py to run Sprucing throughput test.

The test is configured by a analytic run and categorize all Sprucing lines into one single stream.

A local test was run, and the flame graphs are attached below. The throughput for Sprucing on my local machine is found to be 492 Hz.

link to DPA WP1 task lhcb-dpa/project#25 and lhcb-dpa/project#90.

Needed by lhcb-datapkg/PRConfig!236 (merged).

cc @erodrigu @nskidmor @abertoli

FlameBars.pdf

added DPA-WP1 selection labels

assigned to @shunan

added RTA label

mentioned in merge request lhcb-datapkg/PRConfig!236 (merged)

changed the description

Great to see this extra validation being put in place !

added 26 commits

• 243db056...51f46c71 - 25 commits from branch master
• b1e74b92 - Merge branch 'master' into spruce-throughput-test

Compare with previous version

resolved all threads

added 8 commits

• b1e74b92...27e892b3 - 4 commits from branch master
• 847ad211 - add spruce_all_lines to the MooreCache CMakeLists for the throughput test
• b2cd9df4 - bug fix in the test option
• c024b756 - Merge branch 'master' into spruce-throughput-test
• fd96a29c - do not use make_fastest_reconstruction for Sprucing

Compare with previous version

@rmatev @chasse There's a warning when I used make_fastest_reconstruction:

/data4/lhcb/zhangsn/Upgrade/stack/LHCb/InstallArea/x86_64_v2-centos7-gcc11-opt/python/PyConf/tonic.py:436: UserWarning: Bound function <reconstruction() at hlt2_global_reco.py:197> was not called within a bind. Stack trace:
  File "/data4/lhcb/zhangsn/Upgrade/stack/Moore/Hlt/Hlt2Conf/options/sprucing/spruce_all_lines.py", line 147, in <module>
        config = run_moore(options, make_streams, public_tools, analytics=True)

so I just removed this bind in the test option.

However after I required functor cache for this test by adding force_functor_cache upfront, the throughput test failed. The log file is attached below. I simply put the name of this option into MooreCache/CMakeLists.txt and re-compiled the stack. Did I miss anything?

The log file is too large to attach so I removed O(10000) lines in the middle. I believe most of them are useless, and seems the critical message is:

HLTControlFlowMgr                     FATAL Event failed in Node VoidFilter/Streaming_filter
HLTControlFlowMgr                     FATAL *** Event 25 on slot 25 failed! ***

ThroughputTest.x86_64_v2-centos7-gcc11-opt.128t.1j.100e.0.log.require_cache.crash

added 1 commit

• 3287346f - Fixed formatting

Compare with previous version

added 1 commit

• 8d32f159 - don't compile functor library for spruce_all_lines by default

Compare with previous version

marked this merge request as ready

resolved all threads

@rmatev this MR is also ready, and it needs to be tested and merged with lhcb-datapkg/PRConfig!236 (merged). Could you also take care of this? Thanks.

assigned to @rmatev

unassigned @shunan

@rmatev ping

/ci-test lhcb-datapkg/PRConfig!236 (merged)

added ci-test-triggered label

• [2022-07-18 13:35] Validation started with lhcb-master-mr#5100

assigned to @sponce

resolved all threads

approved this merge request

merged

mentioned in commit a2883a92

Hey @sponce, @shunan, where is the Sprucing throughput test (in particular the flame graph) similar to Moore_hlt1_pp_default above? It will only come in future CI tests?

mentioned in merge request lhcb-core/LHCbNightlyConf!863 (merged)

resolved all threads

mentioned in issue #456 (closed)

mentioned in issue #456 (closed)