Commit 76e29b5d authored by Daniel Campora's avatar Daniel Campora
Browse files

Updated readmes.

parent 95bb02f8
Pipeline #1651470 passed with stages
in 14 minutes and 42 seconds
Configuring the sequence of algorithms
===============
======================================
`Allen` centers around the idea of running a __sequence of algorithms__ on input events. This sequence is predefined and will always be executed in the same order.
Some events from the input will be discarded throughout the execution, and only a fraction of them will be kept for further processing.
Allen is configured with the python scripts located in `generator`. Both the sequence of algorithms and the characteristics of individual algorithms are configurable. The properties of individual algorithms include
the input, output and properties.
Explore the algorithms available and view their properties from within the `generator` directory:
Invoke `python3 /parse_algorithms.py`. The generated file `algorithms.py` contains information of the algorithms in the C++ code.
You may `import algorithms` from a python3 shell and check with auto-complete the algorithms available and their options, for example:
```sh=
foo@bar:~$ python3 parse_algorithms.py
Generating algorithms.py...
File algorithms.py was successfully generated.
foo@bar:~$ python3
Python 3.6.5 (default, Jun 15 2019, 23:43:55)
[GCC 8.2.0] on linux
The sequence can be configured with python. Existing configurations can be browsed under `configuration/sequences`. The sequence name is the name of each individual file, without the `.py` extension, in that folder. For instance, some sequence names are `velo`, `ut`, or `hlt1_pp_default`.
The sequence can be chosen _at compile time_ with the cmake option `SEQUENCE`, and passing a concrete sequence name. For instance:
# Configure the VELO sequence
cmake -DSEQUENCE=velo ..
# Configure the ut sequence
cmake -DSEQUENCE=ut ..
# Configure the hlt1_pp_default sequence (by default)
cmake ..
The rest of this readme explains the workflow to generate a new sequence.
Inspecting algorithms
---------------------
In order to generate a new sequence, python 3 and (cvmfs and CentOS 7, or clang 9 or higher) are required.
It is possible to inspect all algorithms defined in Allen interactively by using the _python view_ that the parser automatically generates. From a build directory:
```sh
foo@bar:build$ cmake ..
foo@bar:build$ cd sequences
foo@bar:build/sequences$ python3
Python 3.8.2 (default, Feb 28 2020, 00:00:00)
[GCC 10.0.1 20200216 (Red Hat 10.0.1-0.8)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import algorithms
>>> from definitions import algorithms
>>>
```
Now, you can inspect any existing algorithm. For instance, all algorithms starting with `velo_`:
```sh
>>> algorithms.velo_
algorithms.velo_calculate_number_of_candidates_t( algorithms.velo_estimate_input_size_t( algorithms.velo_pv_ip_t(
algorithms.velo_calculate_phi_and_sort_t( algorithms.velo_fill_candidates_t( algorithms.velo_search_by_triplet_t(
algorithms.velo_consolidate_tracks_t( algorithms.velo_kalman_filter_t( algorithms.velo_three_hit_tracks_filter_t(
algorithms.velo_copy_track_hit_number_t( algorithms.velo_masked_clustering_t(
algorithms.velo_calculate_number_of_candidates_t( algorithms.velo_kalman_filter_t(
algorithms.velo_calculate_phi_and_sort_t( algorithms.velo_masked_clustering_t(
algorithms.velo_consolidate_tracks_t( algorithms.velo_pv_ip_t(
algorithms.velo_copy_track_hit_number_t( algorithms.velo_search_by_triplet_t(
algorithms.velo_estimate_input_size_t( algorithms.velo_three_hit_tracks_filter_t(
```
One can see the input and output parameters and properties of an algorithm by just printing the class representation of an algorithm (ie. _without parentheses_). For instance:
```sh
>>> algorithms.velo_calculate_number_of_candidates_t
class AlgorithmRepr : DeviceAlgorithm
inputs: ('host_number_of_selected_events_t', 'dev_event_list_t', 'dev_velo_raw_input_t', 'dev_velo_raw_input_offsets_t')
outputs: ('dev_number_of_candidates_t',)
properties: ('block_dim_x',)
```
Creating a new sequence
-----------------------
In order to create a new sequence, head to `configuration/sequences` and create a new sequence file with extension `.py`.
You may reuse what exists already in `definitions` and extend that instead. In order to create a new sequence, you should:
* Instantiate (ie. _with parentheses_) algorithms. Algorithm inputs must be assigned other algorithm outputs.
* Create a `Sequence` object with the desired algorithm instances in order of execution.
* The `Sequence` class also accepts line objects.
* `Sequences` can also be composed or extended.
* Generate the configuration with the `generate()` method of the `Sequence` instance.
As an example, let us add the SAXPY algorithm to a custom sequence. Start by including algorithms and the VELO sequence:
```sh
from definitions.algorithms import *
from definitions.VeloSequence import VeloSequence
velo_sequence = VeloSequence()
```
Printing any one algorithm will tell you its parameters and properties:
```sh=
>>> algorithms.velo_calculate_phi_and_sort_t()
velo_calculate_phi_and_sort_t "velo_calculate_phi_and_sort_t" (
host_number_of_selected_events_t = HostInput("host_number_of_selected_events_t", uint32_t),
host_total_number_of_velo_clusters_t = HostInput("host_total_number_of_velo_clusters_t", uint32_t),
dev_offsets_estimated_input_size_t = DeviceInput("dev_offsets_estimated_input_size_t", uint32_t),
dev_module_cluster_num_t = DeviceInput("dev_module_cluster_num_t", uint32_t),
dev_velo_cluster_container_t = DeviceInput("dev_velo_cluster_container_t", char),
dev_sorted_velo_cluster_container_t = DeviceOutput("dev_sorted_velo_cluster_container_t", char),
dev_hit_permutation_t = DeviceOutput("dev_hit_permutation_t", uint32_t),
dev_hit_phi_t = DeviceOutput("dev_hit_phi_t", half_t),
block_dim = PROPERTY(DeviceDimensions, 1, 1}, block dimensions, {64) = "")
`velo_sequence` contains a `Sequence` object with all algorithms defined therein. The individual algorithms are still accessible with `__get_item__` and the algorithm name (like a dictionary).
We should now add the SAXPY algorithm. We can use the interactive session to explore what it requires:
```sh
>>> algorithms.saxpy_t
class AlgorithmRepr : DeviceAlgorithm
inputs: ('host_number_of_selected_events_t', 'dev_offsets_all_velo_tracks_t', 'dev_offsets_velo_track_hit_number_t')
outputs: ('dev_saxpy_output_t',)
properties: ('saxpy_scale_factor', 'block_dim')
```
You can also create an instance of the algorithm and auto-complete the parameters:
```sh=
>>> a = algorithms.ut_search_windows_t()
>>> a.
a.block_dim_y_t( a.dev_ut_selected_velo_tracks_t( a.min_momentum_t( a.properties(
a.dev_offsets_all_velo_tracks_t( a.dev_ut_windows_layers_t( a.min_pt_t( a.requires_lines(
a.dev_offsets_velo_track_hit_number_t( a.dev_velo_states_t( a.name( a.y_tol_slope_t(
a.dev_ut_hit_offsets_t( a.filename( a.namespace( a.y_tol_t(
a.dev_ut_hits_t( a.host_number_of_reconstructed_velo_tracks_t( a.original_name(
a.dev_ut_number_of_selected_velo_tracks_t( a.host_number_of_selected_events_t( a.parameters(
>>> a.block_dim_y_t()
PROPERTY(uint32_t, block dimension Y) = ""
The inputs should be passed into our sequence to be able to instantiate `saxpy_t`. Knowing which inputs to pass is up to the developer. For this one, let's just pass:
```sh
saxpy = algorithms.saxpy_t(
name = "saxpy",
host_number_of_selected_events_t = velo_sequence["initialize_lists"].host_number_of_selected_events_t(),
dev_offsets_all_velo_tracks_t = velo_sequence["velo_copy_track_hit_number"].dev_offsets_all_velo_tracks_t(),
dev_offsets_velo_track_hit_number_t = velo_sequence["prefix_sum_offsets_velo_track_hit_number"].dev_output_buffer_t())
```
Generate a configuration from within the `generator` directory with python3:
* You have some examples of sequences in the `configuration/generator/` folder, such as `VeloSequence.py`, `UTSequence.py` and so on.
* `python3 MakeDefaultConfigurations.py` will generate the configuration and store one header and one json file per sequence in the subdirectory `generated`.
* Taking as example the default HLT1 sequence: `Defaultequence.h` contains the C++ generated code to successfully compile the code. `DefaultSequence.json` can be used to configure the Allen application when invoking it.
* `ConfigurationGuide.json` contains all available options and their default values, for reference.
* Copy the generated `DefaultSequence.h` into `configuration/sequences/`, and the `DefaultSequence.json` into `configuration/constants/`.
* Now you should be able to compile your sequence by issuing `cmake -DSEQUENCE=DefaultSequence .. && make`.
* You can invoke the program with your options with `./Allen --configuration=../configuration/constants/Configuration.json`.
* In the case of calling Allen from Gaudi:
* `export CMAKEFLAGS="-DSEQUENCE=YourOwnSequence.h"` before the call to `make configure`
* set the "JSON" property of the RunAllen algorithm to the json file you just generated.
* Generate your own custom sequence along the same principles!
To understand the configuration in more detail, we let's take a look at how to configure
the same algorithm with different properties. As example we take the UT tracking,
with either loose or restricted cuts. The full code can be found in `generator/UTSequence.py`.
```clike=
ut_search_windows = None
compass_ut = None
if restricted:
ut_search_windows = ut_search_windows_t(
min_momentum="1500.0", min_pt="300.0", 64)
compass_ut = compass_ut_t(
max_considered_before_found="6",
min_momentum_final="2500.0",
min_pt_final="425.0")
else:
ut_search_windows = ut_search_windows_t(
min_momentum="3000.0", min_pt="0.0")
compass_ut = compass_ut_t(
max_considered_before_found="16",
min_momentum_final="0.0",
min_pt_final="0.0")
Finally, let's extend the `velo_sequence` with our newly created algorithm, and generate the sequence:
```sh
extend_sequence(velo_sequence, saxpy).generate()
```
In this case, the search windows for the pattern recognition step are set differently,
and the settings for the UT track reconstruction (compass_ut) also vary,
depending on whether `restricted` is true or false.
Alternatively, one can call the same algorithm twice in the same sequence, with different configurations.
```clike=
ut_search_windows_restricted = ut_search_windows_t(
min_momentum="1500.0", min_pt="300.0",
dev_ut_windows_layers=dev_ut_windows_layers_restricted)
ut_search_windows_loose = ut_search_windows_t(
min_momentum="3000.0", min_pt="0.0",
dev_ut_windows_layers=dev_ut_windows_layers_loose)
...
ut_sequence = Sequence(
ut_calculate_number_of_hits, prefix_sum_ut_hits, ut_pre_decode,
ut_find_permutation, ut_decode_raw_banks_in_order,
ut_select_velo_tracks, ut_search_windows_restricted,
ut_search_windows_loose)
The final configuration file is therefore:
```sh
from definitions.algorithms import *
from definitions.VeloSequence import VeloSequence
velo_sequence = VeloSequence()
saxpy = algorithms.saxpy_t(
name = "saxpy",
host_number_of_selected_events_t = velo_sequence["initialize_lists"].host_number_of_selected_events_t(),
dev_offsets_all_velo_tracks_t = velo_sequence["velo_copy_track_hit_number"].dev_offsets_all_velo_tracks_t(),
dev_offsets_velo_track_hit_number_t = velo_sequence["prefix_sum_offsets_velo_track_hit_number"].dev_output_buffer_t())
extend_sequence(velo_sequence, saxpy).generate()
```
Then, one can add other algorithms afterwards, for example one taking as input `dev_ut_windows_layers_restricted`,
the other taking `dev_ut_windows_layers_loose`.
Now, we can save this configuration as `configuration/sequences/saxpy.py`, and build it and run it:
To find out how to write a trigger line in Allen and how to add it to the sequence, follow the instructions [here](../selections.md).
```sh
mkdir build
cd build
cmake -DSEQUENCE=saxpy ..
make
./Allen
```
To find out how to write a trigger line in Allen and how to add it to the sequence, follow the instructions [here](../selections.md).
......@@ -7,7 +7,8 @@ def HLT1Sequence(initialize_lists, velo_copy_track_hit_number,
velo_consolidate_tracks, prefix_sum_ut_tracks,
prefix_sum_ut_track_hit_number, ut_consolidate_tracks,
prefix_sum_scifi_track_hit_number, scifi_consolidate_tracks,
is_muon):
is_muon,
add_default_lines=True):
velo_pv_ip = velo_pv_ip_t(
host_number_of_reconstructed_velo_tracks_t=velo_copy_track_hit_number.
......@@ -175,39 +176,42 @@ def HLT1Sequence(initialize_lists, velo_copy_track_hit_number,
dev_sel_rb_substr_t=prepare_raw_banks.dev_sel_rb_substr_t(),
dev_sel_rep_offsets_t=prefix_sum_sel_reps.dev_output_buffer_t())
ErrorEvent_line = ErrorEvent_t()
PassThrough_line = PassThrough_t()
NoBeams_line = NoBeams_t()
BeamOne_line = BeamOne_t()
BeamTwo_line = BeamTwo_t()
BothBeams_line = BothBeams_t()
ODINNoBias_line = ODINNoBias_t()
ODINLumi_line = ODINLumi_t()
GECPassthrough_line = GECPassthrough_t()
VeloMicroBias_line = VeloMicroBias_t()
TrackMVA_line = TrackMVA_t()
TrackMuonMVA_line = TrackMuonMVA_t()
SingleHighPtMuon_line = SingleHighPtMuon_t()
LowPtMuon_line = LowPtMuon_t()
TwoTrackMVA_line = TwoTrackMVA_t()
DiMuonHighMass_line = DiMuonHighMass_t()
DiMuonLowMass_line = DiMuonLowMass_t()
LowPtDiMuon_line = LowPtDiMuon_t()
DisplacedDiMuon_line = DisplacedDiMuon_t()
DiMuonSoft_line = DiMuonSoft_t()
D2KPi_line = D2KPi_t()
D2PiPi_line = D2PiPi_t()
D2KK_line = D2KK_t()
hlt1_sequence = Sequence(
velo_pv_ip, kalman_velo_only, filter_tracks,
prefix_sum_secondary_vertices, fit_secondary_vertices, odin_banks,
run_hlt1, prepare_raw_banks, prefix_sum_sel_reps, package_sel_reports,
ErrorEvent_line, PassThrough_line, NoBeams_line, BeamOne_line,
BeamTwo_line, BothBeams_line, ODINNoBias_line, ODINLumi_line,
GECPassthrough_line, VeloMicroBias_line, TrackMVA_line,
TrackMuonMVA_line, SingleHighPtMuon_line, LowPtMuon_line,
TwoTrackMVA_line, DiMuonHighMass_line, DiMuonLowMass_line,
LowPtDiMuon_line, DiMuonSoft_line, D2KPi_line, D2PiPi_line, D2KK_line)
run_hlt1, prepare_raw_banks, prefix_sum_sel_reps, package_sel_reports)
if add_default_lines:
ErrorEvent_line = ErrorEvent_t()
PassThrough_line = PassThrough_t()
NoBeams_line = NoBeams_t()
BeamOne_line = BeamOne_t()
BeamTwo_line = BeamTwo_t()
BothBeams_line = BothBeams_t()
ODINNoBias_line = ODINNoBias_t()
ODINLumi_line = ODINLumi_t()
GECPassthrough_line = GECPassthrough_t()
VeloMicroBias_line = VeloMicroBias_t()
TrackMVA_line = TrackMVA_t()
TrackMuonMVA_line = TrackMuonMVA_t()
SingleHighPtMuon_line = SingleHighPtMuon_t()
LowPtMuon_line = LowPtMuon_t()
TwoTrackMVA_line = TwoTrackMVA_t()
DiMuonHighMass_line = DiMuonHighMass_t()
DiMuonLowMass_line = DiMuonLowMass_t()
LowPtDiMuon_line = LowPtDiMuon_t()
DisplacedDiMuon_line = DisplacedDiMuon_t()
DiMuonSoft_line = DiMuonSoft_t()
D2KPi_line = D2KPi_t()
D2PiPi_line = D2PiPi_t()
D2KK_line = D2KK_t()
return extend_sequence(hlt1_sequence,
ErrorEvent_line, PassThrough_line, NoBeams_line, BeamOne_line,
BeamTwo_line, BothBeams_line, ODINNoBias_line, ODINLumi_line,
GECPassthrough_line, VeloMicroBias_line, TrackMVA_line,
TrackMuonMVA_line, SingleHighPtMuon_line, LowPtMuon_line,
TwoTrackMVA_line, DiMuonHighMass_line, DiMuonLowMass_line,
LowPtDiMuon_line, DiMuonSoft_line, D2KPi_line, D2PiPi_line, D2KK_line)
return hlt1_sequence
......@@ -221,7 +221,7 @@ void saxpy::saxpy_t::set_arguments_size(
const Constants&,
const HostBuffers&) const
{
arguments.set_size<dev_saxpy_output_t>(arguments.first<host_number_of_selected_events_t>());
set_size<dev_saxpy_output_t>(arguments, first<host_number_of_selected_events_t>(arguments));
}
```
......
......@@ -226,9 +226,10 @@ that takes the trigger candidate as an argument and returns a bool.
Selections are added to the Allen sequence similarly to
algorithms. After creating the selection source code, a new sequence
must be generated. From `configuration/generator`, do
`./parse_algorithms.py` to generate the relevant python code. The
selection can then be added to a sequence. The sequence header file
must be generated. Head to `configuration/sequences` and add a new
configuration file.
The selection can then be added to a sequence. The sequence header file
can then be generated in the usual way. The line will automatically be
included in a tuple of selections, which will be accessed using the
`LineTraverser`. The traverser evaluates the selections on candidates
......@@ -242,54 +243,39 @@ checker, DecReports, and SelReports.
algorithms and the example selections we created above. Calling
generate using the returned sequence will produce an Allen sequence
that automatically runs the example selection.
from algorithms import *
from MuonSequence import Muon_sequence
def MinimalHLT1_sequence(validate=False):
kalman_velo_only = kalman_velo_only_t()
kalman_pv_ipchi2 = kalman_pv_ipchi2_t()
filter_tracks = filter_tracks_t()
fit_secondary_vertices = fit_secondary_vertices_t()
prefix_sum_secondary_vertices = host_prefix_sum_t("prefix_sum_secondary_vertices",
host_total_sum_holder_t="host_number_of_svs_t",
dev_input_buffer_t=filter_tracks.dev_sv_atomics_t(),
dev_output_buffer_t="dev_sv_offsets_t")
run_hlt1 = run_hlt1_t()
run_postscale = run_postscale_t()
prepare_decisions = prepare_decisions_t()
prepare_raw_banks = prepare_raw_banks_t()
prefix_sum_sel_reps = host_prefix_sum_t("prefix_sum_sel_reps",
host_total_sum_holder_t="host_number_of_sel_rep_words_t",
dev_input_buffer_t=prepare_raw_banks.dev_sel_rep_sizes_t(),
dev_output_buffer_t="dev_sel_rep_offsets_t")
package_sel_reports = package_sel_reports_t()
ExampleOneTrack_line = ExampleOneTrack_t()
ExampleTwoTrack_line = ExampleTwoTrack_t()
muon_sequence = Muon_sequence()
hlt1_sequence = extend_sequence(muon_sequence,
kalman_velo_only,
kalman_pv_ipchi2,
filter_tracks,
prefix_sum_secondary_vertices,
fit_secondary_vertices,
run_hlt1,
run_postscale,
prepare_decisions,
prepare_raw_banks,
prefix_sum_sel_reps,
package_sel_reports,
ExampleOneTrack_line,
ExampleTwoTrack_line)
if validate:
hlt1_sequence.validate()
return hlt1_sequence
First, copy the contents of `hlt1_pp_default.py` into `custom_sequence.py`.
Modify the argument list to `HLT1Sequence` by adding the keyword argument
`add_default_lines = False`, and add the lines:
hlt1_sequence = HLT1Sequence(
initialize_lists=velo_sequence["initialize_lists"],
velo_copy_track_hit_number=velo_sequence["velo_copy_track_hit_number"],
velo_kalman_filter=pv_sequence["velo_kalman_filter"],
prefix_sum_offsets_velo_track_hit_number=velo_sequence[
"prefix_sum_offsets_velo_track_hit_number"],
pv_beamline_multi_fitter=pv_sequence["pv_beamline_multi_fitter"],
prefix_sum_forward_tracks=forward_sequence["prefix_sum_forward_tracks"],
velo_consolidate_tracks=velo_sequence["velo_consolidate_tracks"],
prefix_sum_ut_tracks=ut_sequence["prefix_sum_ut_tracks"],
prefix_sum_ut_track_hit_number=ut_sequence[
"prefix_sum_ut_track_hit_number"],
ut_consolidate_tracks=ut_sequence["ut_consolidate_tracks"],
prefix_sum_scifi_track_hit_number=forward_sequence[
"prefix_sum_scifi_track_hit_number"],
scifi_consolidate_tracks=forward_sequence["scifi_consolidate_tracks_t"],
is_muon=muon_sequence["is_muon_t"],
# Disable default lines
add_default_lines=False)
# New lines
ExampleOneTrack_line = ExampleOneTrack_t()
ExampleTwoTrack_line = ExampleTwoTrack_t()
# Compose final sequence with lines
compose_sequences(velo_sequence, pv_sequence, ut_sequence, forward_sequence,
muon_sequence, hlt1_sequence, ExampleOneTrack_line,
ExampleTwoTrack_line).generate()
Now, you should be able to build and run the newly generated `custom_sequence`.
\ No newline at end of file
......@@ -166,7 +166,7 @@ auto data(const Args& arguments) {
template<typename Arg, typename Args>
auto first(const Args& arguments) {
return Arg{arguments.template data<Arg>()}[0];
return arguments.template first<Arg>();
}
template<typename Arg, typename Args, typename T>
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment