Speed up Moore tests
Even with many cores available the Moore tests take a non-negligible amount of time (~20 min) for x86_64_v2-centos7-gcc10-opt.
This is mostly driven by a few slow (chains of) tests.
-
Significantly over 10 minutes:
- RecoConf.mc_cheated_pattern_recognition
- Hlt2Conf.test_hlt2_2or3bodytopo_fromfile + Hlt2Conf.test_spruce_fromfile/Hlt2Conf.test_spruce_all_lines_fromfile
-
About 10 minutes:
- RecoConf.hlt2_lead_lead_light_reco_pr_kf_with_mcchecking
- RecoConf.hlt2_lead_lead_light_reco_pr_kf
- RecoConf.hlt2_reco_baseline_with_mcchecking_no_gec
- Hlt2Conf.test_hlt2_streaming
We need to speed these up, most likely by reducing the number of events we run over (and/or adjusting input sample).
/cc @cmarinbe @decianm @mvesteri @poluekt @nskidmor
Extract from a test run
...
105/127 Test #106: RecoConf.hlt2_reco_calo_efficiency ............................................ Passed 526.18 sec
...
118/127 Test #41: Hlt2Conf.test_hlt2_streaming .................................................. Passed 580.08 sec
Start 43: Hlt2Conf.test_hlt2_streaming_check_B
Start 42: Hlt2Conf.test_hlt2_streaming_check_A
119/127 Test #81: RecoConf.hlt2_lead_lead_light_reco_pr_kf ...................................... Passed 586.33 sec
120/127 Test #42: Hlt2Conf.test_hlt2_streaming_check_A .......................................... Passed 17.00 sec
121/127 Test #43: Hlt2Conf.test_hlt2_streaming_check_B .......................................... Passed 17.06 sec
122/127 Test #100: RecoConf.hlt2_reco_baseline_with_mcchecking_no_gec ............................ Passed 611.95 sec
123/127 Test #82: RecoConf.hlt2_lead_lead_light_reco_pr_kf_with_mcchecking ...................... Passed 703.25 sec
124/127 Test #24: Hlt2Conf.test_hlt2_2or3bodytopo_fromfile ...................................... Passed 728.44 sec
Start 54: Hlt2Conf.test_spruce_fromfile
Start 51: Hlt2Conf.test_spruce_all_lines_fromfile
125/127 Test #54: Hlt2Conf.test_spruce_fromfile ................................................. Passed 172.00 sec
126/127 Test #51: Hlt2Conf.test_spruce_all_lines_fromfile ....................................... Passed 201.12 sec
127/127 Test #113: RecoConf.mc_cheated_pattern_recognition ....................................... Passed 1203.82 sec
100% tests passed, 0 tests failed out of 127
Label Time Summary:
Hlt1Conf = 2387.92 sec*proc (21 tests)
Hlt2Conf = 5906.45 sec*proc (36 tests)
Moore = 22613.84 sec*proc (127 tests)
QMTest = 22587.12 sec*proc (124 tests)
RecoConf = 14148.13 sec*proc (67 tests)
disabled = 9.85 sec*proc (4 tests)
Total Test time (real) = 1203.89 secFull CTestCostData.txt from 4 runs
$ awk '{ t=$1 ; $1=$3; $3=t; print }' build.x86_64_v2-centos7-gcc10-opt/Testing/Temporary/CTestCostData.txt | sort -nr
1131.57 4 RecoConf.mc_cheated_pattern_recognition
760.861 4 Hlt2Conf.test_hlt2_2or3bodytopo_fromfile
671.301 4 RecoConf.hlt2_lead_lead_light_reco_pr_kf_with_mcchecking
576.936 4 RecoConf.hlt2_reco_baseline_with_mcchecking_no_gec
571.111 4 Hlt2Conf.test_hlt2_streaming
566.655 4 RecoConf.hlt2_lead_lead_light_reco_pr_kf
476.743 4 Hlt2Conf.test_hlt2_bremsstrahlung
476.74 4 RecoConf.hlt2_reco_calo_efficiency
467.213 4 RecoConf.hlt2_light_reco_calo_efficiency
431.506 4 RecoConf.hlt2_reco_calo_selectivematching
406.003 4 Hlt2Conf.test_persistreco_realtime
349.478 4 RecoConf.hlt2_reco_baseline_with_cheatedseeding
346.858 4 Hlt1Conf.persistency.dst_write
307.889 4 RecoConf.hlt2_reco_baseline_DC
304.614 4 RecoConf.hlt2_reco_full_geometry
295.438 4 Hlt2Conf.test_hlt2_2or3bodytopo_realtime
283.396 4 RecoConf.hlt2_reco_baseline_with_mcchecking
283.089 4 RecoConf.hlt2_reco_baseline_UTTELL40_with_mcchecking
280.809 4 RecoConf.hlt2_reco_baseline_with_mcchecking_mpp_with_bfield
272.658 4 RecoConf.VP_cluster_monitoring_retina
269.419 4 RecoConf.performance.hlt1_reco_trackresolution
261.523 4 RecoConf.VP_cluster_monitoring_baseline
260.726 4 RecoConf.VP_cluster_monitoring
260.646 4 Hlt2Conf.test_hlt2_all_lines_with_reco_with_streams
260.548 4 RecoConf.hlt2_reco_baseline_with_mcchecking_MagUp
259.646 4 RecoConf.hlt1_reco_baseline_with_mcchecking_FTv6
254.41 4 Hlt2Conf.test_hlt2_jet_lines
252.713 4 RecoConf.hlt1_reco_baseline_with_mcchecking
250.554 4 RecoConf.hlt1_reco_retina_with_mcchecking_FTv6
250.33 4 Hlt2Conf.test_hlt2_all_lines_with_reco
241.144 4 Hlt2Conf.test_hlt2_thor_selections
240.261 4 RecoConf.hlt1_reco_muonmatching_with_mcchecking
240.176 4 Hlt1Conf.persistency.mdf_write
238.705 4 RecoConf.hlt2_reco_brunelesque_with_mcchecking
234.813 4 RecoConf.hlt2_fast_reco_with_mcchecking
229.239 4 Hlt2Conf.test_persistreco_fromfile
222.63 4 RecoConf.performance.hlt1_reco_muonIDeff
221.267 4 RecoConf.mc_matching_example
219.675 4 Hlt1Conf.test_hlt1_example
218.573 4 RecoConf.performance.VP_cluster_monitoring_comparison_baseline_retina_plots
216.526 4 RecoConf.hlt2_light_reco_pr_kf_with_mcchecking
215.025 4 RecoConf.hlt2_reco_retina_with_mcchecking
209.521 4 RecoConf.hlt2_fastest_reco_with_mcchecking
206.368 4 RecoConf.hlt2_light_reco_with_mcchecking
202.788 4 Hlt2Conf.test_hlt2_all_lines
200.517 4 Hlt2Conf.test_hlt2_line_example
199.328 4 Hlt1Conf.persistency.mdf_read
193.382 4 Hlt2Conf.test_spruce_all_lines_fromfile
188.38 4 Hlt2Conf.test_spruce_fromfile
187.26 4 RecoConf.hlt1_reco_baseline_UTTELL40_with_mcchecking
178.845 4 Hlt1Conf.persistency.dst_read
172.124 4 RecoConf.performance.hlt1_reco_IPresolution
170.045 4 Hlt2Conf.test_hlt2_jet_lines_with_reco
166.863 4 RecoConf.hlt2_particles_baseline
165.425 4 Hlt2Conf.test_spruce_all_lines_realtime
165.085 4 Hlt2Conf.test_hlt2_standard_particles
164.785 4 Hlt2Conf.test_spruce_realtime
161.953 4 RecoConf.decoding.hlt1_ut_decoding_v4r2
157.43 4 Hlt1Conf.test_hlt1_smog_example
156.512 4 Hlt2Conf.test_persistreco_mdf_fromfile
155.579 4 Hlt2Conf.test_hlt2_example
132.852 4 RecoConf.hlt2_protoparticles_baseline
130.555 4 RecoConf.decoding.hlt1_ut_decoding
130.511 4 Hlt1Conf.test_hlt1_lumi_minbias
130.44 4 Hlt1Conf.persistency.allen_mdf_write
130.079 4 RecoConf.hlt2_reco_calo_resolution_gamma
128.48 4 Hlt2Conf.thor.loki_comparison
127.54 4 Hlt1Conf.persistency.selreports
123.109 4 RecoConf.pr_kf_velo_only
122.615 4 Hlt1Conf.persistency.hlt2_read_hlt1_filtered_mdf
119.711 4 RecoConf.hlt2_reco_baseline_with_data_monitoring_alignment_rich
114.229 4 RecoConf.hlt2_reco_brunelesque_with_data_monitoring
113.974 4 RecoConf.examples.hlt2_reco_test_ldst_input
113.829 4 RecoConf.performance.VP_cluster_monitoring_retina_plots
113.712 4 RecoConf.hlt2_reco_baseline_with_monitoring
113.242 4 RecoConf.performance.VP_cluster_monitoring_baseline_plots
111.82 4 RecoConf.hlt2_reco_calo_resolution_pi0
109.208 4 RecoConf.hlt2_reco_baseline
107.702 4 RecoConf.decoding.hlt1_velo_decoding
105.662 4 RecoConf.hlt2_reco_baseline_with_parametrised_scatters
101.868 4 Hlt1Conf.performance.pp_default
101.229 4 RecoConf.hlt2_reco_brunelesque
100.963 4 RecoConf.mc_hit_resolution_monitor
100.247 4 RecoConf.hlt2_fast_reco
98.2531 4 Hlt1Conf.performance.pp_comparison
94.6053 4 Hlt1Conf.test_hlt1_dump_tracks
93.5894 4 RecoConf.examples.hlt2_reco_tracking
91.6823 4 RecoConf.hlt1_reco_baseline
90.8996 4 RecoConf.hlt2_reco_calo_from_dst
90.8856 4 RecoConf.hlt2_reco_baseline_multi_threaded
90.3704 4 Hlt1Conf.performance.pp_track_mva
88.9504 4 RecoConf.hlt2_light_reco_tracking
84.6719 4 RecoConf.hlt1_reco_baseline_multi_threaded
84.2432 4 Moore.test_ioextension_input
82.3521 4 RecoConf.examples.hlt1_reco_velo_only
80.9645 4 RecoConf.hlt1_reco_muonmatching
79.6414 4 Moore.test_minimal_options
74.2377 4 RecoConf.mc_run_unpackers
46.7102 4 Hlt2Conf.test_persistreco_output_fromfile
45.0896 4 Hlt2Conf.test_persistreco_output_realtime
43.0002 4 Hlt2Conf.test_persistreco_mdf_output_fromfile
40.1622 4 Hlt2Conf.test_hlt2_check_output
39.9022 4 Hlt2Conf.test_spruce_check
37.1766 4 Hlt2Conf.test_hlt2_check_output_with_reco
27.0368 4 Hlt1Conf.persistency.mdf_read_decs_allen
25.6721 4 Hlt1Conf.persistency.make_allen_tck
25.494 4 Hlt1Conf.persistency.mdf_read_decs
24.3082 4 Hlt1Conf.persistency.assign_allen_tck
21.5194 4 Hlt1Conf.persistency.mdf_test_routing_bits
20.9128 4 Hlt2Conf.profile_config
17.2546 4 Hlt2Conf.test_hlt2_streaming_check_B
16.8403 4 Hlt2Conf.test_hlt2_streaming_check_A
16.0264 4 Hlt1Conf.profile_config
12.0521 4 RecoConf.hlt2_plot_calo_resolution_pi0
11.5768 4 RecoConf.hlt2_plot_calo_resolution_gamma
10.4604 4 Hlt1Conf.python
9.5531 4 RecoConf.performance.hlt1_reco_trackingeff_plots
9.20935 4 RecoConf.performance.hlt1_reco_IPresolution_plots
8.70833 4 RecoConf.performance.hlt1_reco_trackresolution_plots
8.56641 4 Hlt2Conf.python
8.02355 4 RecoConf.performance.hlt1_reco_muonIDeff_plots
7.46202 4 Moore.python
3.63841 4 Hlt2Conf.thor.loki_comparison_check
0 0 Hlt2Conf.test_hlt2_relations_full_mc
0 0 Hlt2Conf.test_hlt2_relations
0 0 Hlt2Conf.test_hlt2_lines_reco_mix_full_mc
0 0 Hlt2Conf.test_hlt2_lines_reco_mixActivity
possibly there is some rationalization to be made but it would take a more careful look to do that. Also, sometimes merging might be not what we want (so that tests are more targeted).
Edited by Rosen Matev
Hlt2Conf.test_spruce_all_lines_fromfileshould not be there anymore https://lhcb-nightlies.web.cern.ch/nightly/lhcb-master-mr/2689/Moore/x86_64_v2-centos7-gcc10-opt/tests.I would like to keep
Hlt2Conf.test_spruce_fromfileif possible but I will invetigate whether we can reduce the number of eventsEdited by Nicole Skidmore-
I have reduced the number of events to 100 for
Hlt2Conf.test_spruce_fromfile!932 (diffs)
@rmatev I guess 100/1000... events may not provide the most targeted testing anyway. I suppose what we really want (aside from dedicated throughput tests) is
- larger scale tests to measure the rates/efficiencies.
- tests with single events where we compare the (post reconstruction/selection) content for that single event.
I replaced the data set for the
RecoConf.mc_cheated_pattern_recognitionwith one that only runs over 100 events, and it reduces the time to maybe 2min.I also reduced the number of events for the PbPb tests to 20 (from 50) and the number of events for
hlt2_reco_baseline_with_mcchecking_no_gecto 100 (from 500).Edited by Michel De Cian
Please register or sign in to reply