libClang parser and updated configuration framework

Daniel Hugo Campora Perezrequested to merge
dcampora_libtooling_parser into master

This MR modifies substantially the way configurations are parsed and generated in Allen.

The configuration generation is now integrated with the make process.

  • Python sequences exist in configuration/sequences/.
  • Each of the Python sequences can be identified by its name, such as <sequence_name>.py.
  • The cmake option SEQUENCE can be specified to be one of these <sequence_name>.
  • cmake will then check that the generation process works (it should work if either /cvmfs/sft.cern.ch is available, or clang >= 9.0.0 is installed).
  • If the generation process works: When make is invoked, the specified SEQUENCE will be generated. The resulting .h will be used for compilation, and the resulting .json will be copied to ${PROJECT_BINARY_DIR}/Sequence.json.
  • If the generation process doesn't work: A warning is issued at cmake configuration time. When make is invoked, the specified SEQUENCE will be copied (both .h and .json) from the pregenerated ones. Not finding said sequence will result in a compilation error.
  • Once Allen is compiled, ./Allen will by default pick the Sequence.json in its same folder.

Examples

Generating the default sequence, HLT1:

$ cmake ..
[...]
-- Testing code generation with LLVM
-- Testing code generation with LLVM - Success
[...]

$ make -j
[...]
Parsing algorithms...
Generating /home/dcampora/projects/allen/build/sequences/definitions/algorithms.py...
File /home/dcampora/projects/allen/build/sequences/definitions/algorithms.py was successfully generated.
Validating sequence...
Warning: OutputParameter "dev_sel_rb_substr_t" appears on algorithms: prepare_raw_banks_t, package_sel_reports_t
Warning: OutputParameter "dev_sel_rep_offsets_t" appears on algorithms: prefix_sum_sel_reps, package_sel_reports_t
Warning: OutputParameter "dev_atomics_muon_t" appears on algorithms: muon_pre_decoding_t, muon_add_coords_crossing_maps_t
Warning: OutputParameter "dev_kf_tracks_t" appears on algorithms: kalman_velo_only_t, kalman_pv_ipchi2_t
Warning: OutputParameter "dev_sel_rb_hits_t" appears on algorithms: prepare_raw_banks_t, package_sel_reports_t
Warning: OutputParameter "dev_ut_windows_layers_t" appears on algorithms: ut_search_windows_t, ut_select_velo_tracks_with_windows_t
Warning: OutputParameter "dev_sel_rb_objtyp_t" appears on algorithms: prepare_raw_banks_t, package_sel_reports_t
Warning: OutputParameter "dev_scifi_lf_tracks_t" appears on algorithms: lf_triplet_keep_best_t, lf_extend_tracks_x_t, lf_extend_tracks_uv_t, lf_quality_filter_length_t
Warning: OutputParameter "dev_sel_results_offsets_t" appears on algorithms: run_hlt1_t, run_postscale_t
Warning: OutputParameter "dev_dec_reports_t" appears on algorithms: prepare_decisions_t, prepare_raw_banks_t
Warning: OutputParameter "dev_scifi_lf_length_filtered_tracks_t" appears on algorithms: lf_quality_filter_length_t, lf_quality_filter_t
Warning: OutputParameter "dev_sel_results_t" appears on algorithms: run_hlt1_t, run_postscale_t
Warning: OutputParameter "dev_scifi_hits_t" appears on algorithms: scifi_pre_decode_v4_t, scifi_raw_bank_decoder_v4_t, scifi_direct_decoder_v4_t
Warning: OutputParameter "dev_muon_hits_t" appears on algorithms: muon_add_coords_crossing_maps_t, muon_sort_by_station_t
Warning: OutputParameter "dev_sel_rb_stdinfo_t" appears on algorithms: prepare_raw_banks_t, package_sel_reports_t
Warning: OutputParameter "dev_storage_tile_id_t" appears on algorithms: muon_pre_decoding_t, muon_sort_station_region_quarter_t, muon_add_coords_crossing_maps_t
Warning: OutputParameter "dev_storage_tdc_value_t" appears on algorithms: muon_pre_decoding_t, muon_sort_station_region_quarter_t, muon_add_coords_crossing_maps_t
Number of sequence warnings: 17
Generating sequence file...
Generated sequence file Sequence.h
Generating JSON configuration file...
Generated JSON configuration file Sequence.json
[ 57%] Built target configured_sequence

Changing the sequence:

cmake -DSEQUENCE=VELO ..
make -j

Example where generation is not available (CVMFS not available, and clang version < 9.0.0):

$ cmake ..
[...]
-- Testing code generation with LLVM
CMake Warning at stream/CMakeLists.txt:123 (message):
  Testing code generation with LLVM - Failed.  CVMFS (sft.cern.ch) or clang
  >= 9.0.0 are required to be able to generate configurations.
[...]

$ make -j
[...]
[  7%] Built target configured_sequence
[...]

Available sequences:

https://gitlab.cern.ch/lhcb/Allen/-/tree/dcampora_libtooling_parser/configuration%2Fsequences

Missing features:

  • (Done) Ensure configurations are generated deterministically.
  • (Done) The configuration is always updated, triggering a compilation, every time the sequence is changed, irrespective of whether the changes would only impact the .json. This should be easily fixable by checking the configured header file is the same.

Known caveats:

  • At present, include files are parsed without the c++ include files (option -nostdinc++). Otherwise, some algorithms are not parsed correctly by clang.cindex. Seemingly this causes no issues in recognizing the parsed classes. A happy side effect is that this makes the parsing process faster.
  • No additional include folders (-I) are added. The obvious candidate -Istream/gear/include unfortunately makes parsing parameters much harder, and I did not find a way to properly traverse individual parameters (here is my last attempt). Instead, for these cases, the tokens are traversed. It is arguable whether this is more unsafe than using macros (ie. parameters and properties are still a comma-separated list of tokens after all). Adding -Istream/gear/include makes parsing the tokens much harder as it converts the parameters block into a single list of tokens.
  • The DeviceAlgorithm and HostAlgorithm classes are not defined, and are therefore parsed as tokens. Including -Istream/gear/include solves this at the cost of making parsing the parameters harder.
Edited by Daniel Hugo Campora Perez

Merge request reports