Allen merge requestshttps://gitlab.cern.ch/lhcb/Allen/-/merge_requests2022-10-31T13:40:20+01:00https://gitlab.cern.ch/lhcb/Allen/-/merge_requests/1044Follow for removal of Det/DetCond in DD4hep builds2022-10-31T13:40:20+01:00Roel AaijFollow for removal of Det/DetCond in DD4hep buildshttps://gitlab.cern.ch/lhcb/Allen/-/merge_requests/866Configuring the VeloOpen ODIN line2022-08-25T11:03:50+02:00Suzanne KlaverConfiguring the VeloOpen ODIN lineThis MR configures the VeloOpen ODIN line such that it can be used for the Velo closing monitoring procedure, as addressed in https://gitlab.cern.ch/lhcb/Allen/-/issues/310. It also configures the other missing ODIN lines (Physics, Beam1...This MR configures the VeloOpen ODIN line such that it can be used for the Velo closing monitoring procedure, as addressed in https://gitlab.cern.ch/lhcb/Allen/-/issues/310. It also configures the other missing ODIN lines (Physics, Beam1Gas and Beam2Gas).
@mmulderhttps://gitlab.cern.ch/lhcb/Allen/-/merge_requests/835Add algorithms to support Moore-based Gaudi Allen configuration2022-06-21T19:18:58+02:00Patrick SpradlinAdd algorithms to support Moore-based Gaudi Allen configurationAdd algorithms to support Moore-based Gaudi Allen configuration.
Most of the algorithms in Rec/Allen/src that rely on the output of RunAllen will need to be updated or replaced by algorithms that interface directly with the output Allen ...Add algorithms to support Moore-based Gaudi Allen configuration.
Most of the algorithms in Rec/Allen/src that rely on the output of RunAllen will need to be updated or replaced by algorithms that interface directly with the output Allen reconstruction algorithms.
Based on MR !823 and the `NN_hack_gaudi_allen` branch.
Should go along https://gitlab.cern.ch/lhcb/Moore/-/merge_requests/609, MooreAnalysis!84 and https://gitlab.cern.ch/lhcb/MooreOnline/-/merge_requests/99https://gitlab.cern.ch/lhcb/Allen/-/merge_requests/685Type-Erased Allen2022-01-14T10:31:27+01:00Daniel Campora PerezType-Erased AllenThis MR type erases all Allen algorithms. It reduces compilation times, and enables running any Allen sequence specified in a python config or json file (see list of changes)
All the changes affect the backend, the frontend (that is, ...This MR type erases all Allen algorithms. It reduces compilation times, and enables running any Allen sequence specified in a python config or json file (see list of changes)
All the changes affect the backend, the frontend (that is, what the average user will see) is not modified, with the exception of requiring specifying `INSTANTIATE_ALGORITHM(algorithm_type)` in each algorithm source file.
List of changes
---
* All Allen algorithms are now type-erased. All algorithms require specifying an `INSTANTIATE_ALGORITHM(algorithm_type)` macro. Selection algorithms do this by their preexisting `INSTANTIATE_LINE`, which has been extended to also instantiate the algorithm.
* An `AlgorithmDB.h` containing a string to algorithm correspondence is generated at compile-time.
* Arguments are fully type-erased as well. They were missing the type size and scope, which now allows the memory manager to determine the correct `bytesize` to use for allocating / freeing and whether to use the `host` or `device` memory manager instance.
* The Allen store (`ArgumentManager`) has now become an `UnorderedStore` instance. It should be noted that this "store" keeps `ArgumentData` which contain pointers to the data. The memory where the data lives is managed by the memory manager as before.
* StreamLoader, SchedulerMachinery and `ConfiguredSequence.h` generation code were not required anymore and have been removed.
* By removing the need to generate a `ConfiguredSequence.h`, this MR reduces the compilation time and resource requirements significantly.
* `AlgorithmTraversalLibClang.py` now uses exclusively `c++` parsing in contrast to before, where `.cuh` files were being parsed with `-x cuda`. The reason for this is that in doing so the parsing is more homogeneous. Other options include `"-std=c++17", "-nostdinc++"`.
* Sequence is now generated by default at runtime (from the python configurations). This means that changes in the python configurations, even those done after compilation, will have an effect at runtime. It is still possible to generate a list of `.json` sequences using option `-DSEQUENCES` as before, and run off them specifying `--run-from-json 1` at runtime.
```bash
./Allen --sequence <my_sequence> # generates config in configuration/sequences/<my_sequence>.py and runs it
./Allen # runs hlt1_pp_default per default
```
Sequence generation changes
---
* The compile step for generating sequences now only generates a `.json` file. Please note that this generation still occurs at compile-time, although it is not a true compile requirement anymore. It can potentially be moved to runtime.
* The `.json` configuration contains the sequence under the field `"sequence"` in a predefined format. It expects `configured_algorithms`, `configured_arguments`, `configured_sequence_arguments` and `argument_dependencies`. As an example:
```json
"sequence": {
"configured_algorithms": [
["host_init_event_list::host_init_event_list_t", "initialize_event_lists"],
["host_data_provider::host_data_provider_t", "host_scifi_banks"],
],
"configured_arguments": [
["host", "initialize_event_lists__host_event_list_output_t"],
["device", "initialize_event_lists__dev_event_list_output_t"],
["host", "host_scifi_banks__host_raw_banks_t"],
["host", "host_scifi_banks__host_raw_offsets_t"],
["host", "host_scifi_banks__host_raw_bank_version_t"]
],
"configured_sequence_arguments": [
[["initialize_event_lists__host_event_list_output_t", "initialize_event_lists__dev_event_list_output_t"], []],
[["host_scifi_banks__host_raw_banks_t",
"host_scifi_banks__host_raw_offsets_t",
"host_scifi_banks__host_raw_bank_version_t"], []]
],
"argument_dependencies": {
"host_scifi_banks__host_raw_banks_t": ["initialize_event_lists__host_event_list_output_t"]
}
}
```
* `configured_algorithms` - It is of type `std::vector<ConfiguredAlgorithm>` (due to the `nlohmann-json` parsing POD-type requirements, it is parsed as a `std::vector<std::tuple<std::string, std::string>>` which is then converted with `std::make_from_tuple` to `ConfiguredAlgorithm`s - a similar method is used for the other types). `configured_algorithms` contains a list of algorithms to be executed with their configured names.
* `configured_arguments` - Type `std::vector<ConfiguredArgument>` (`std::vector<std::tuple<std::string, std::string>>`). It contains a list of argument names with their associated scope (either `host` or `device`). These are the arguments that populate the Allen store.
* `configured_sequence_arguments` - Type `std::vector<ConfiguredAlgorithmArguments>` (`std::vector<std::tuple<std::vector<std::string>, std::vector<std::vector<std::string>>>>`). It includes the list of arguments and input aggregates that will be used in each algorithm's invocation.
* `argument_dependencies` - Type `ArgumentDependencies` (`std::map<std::string, std::vector<std::string>>`). It contains the dependencies of each datatype.
Built on top of https://gitlab.cern.ch/lhcb/Allen/-/merge_requests/431Rosen MatevJonathan DaviesRosen Matevhttps://gitlab.cern.ch/lhcb/Allen/-/merge_requests/552Runtime sequence selection2021-09-25T18:31:01+02:00Roel AaijRuntime sequence selectionThis MR implements a proposal for runtime sequence selection.
Sequences are detected by looking at the `configuration/sequences` directory and assuming that all `.py` files are a sequence. For each sequence a separate shared library is ...This MR implements a proposal for runtime sequence selection.
Sequences are detected by looking at the `configuration/sequences` directory and assuming that all `.py` files are a sequence. For each sequence a separate shared library is built (including `GatherSelections.cu`). These libraries are built and installed in the directory `sequences` relative to the location of `libAllenLib.so`.
`StreamWrapper` has been refactored into the abstract base class `IStream` and for each processing thread a stream instance is created in `Allen.cpp`
At runtime, the location of `libAllenLib.so` is determined and used to get the location of the `sequences` directory. The correct shared library is then loaded and the factory function (`create_sequence`) is retrieved from the library. This is wrapped in a lambda that returns a `unique_ptr<IStream>`.
This feature is tested in all of the CI tests, because the sequence loading is always used. For the stack build, the feature is tested by running the Allen sequence without the GEC to calculate efficiencies in lhcb/MooreAnalysis!34.
TODO:
- [x] Update configuration of `RunAllen` in Moore to allow the name of the desired sequence to be configured from an options file; handled in lhcb/Moore!792
- [x] Decide in which order this should be merged relative to !429 and !532
Partially addresses #215; the TCK part will be a future MR
Should be tested together with lhcb/Moore!792 and lhcb/MooreAnalysis!34https://gitlab.cern.ch/lhcb/Allen/-/merge_requests/637follow LHCb ODIN creator changes2021-09-24T11:18:30+02:00Sevda Esenfollow LHCb ODIN creator changesto go with LHCb!3036to go with LHCb!3036https://gitlab.cern.ch/lhcb/Allen/-/merge_requests/393Reworked selections2021-02-18T12:55:04+01:00Daniel Campora PerezReworked selectionsThis MR redesigns from the ground up selections.
* Selections are now algorithms. A new type of algorithm `SelectionAlgorithm` is available.
* Input aggregates are now supported. Parser detects these types and generates code accordingly...This MR redesigns from the ground up selections.
* Selections are now algorithms. A new type of algorithm `SelectionAlgorithm` is available.
* Input aggregates are now supported. Parser detects these types and generates code accordingly.
* `GatherSelections` is an algorithm that uses input aggregates.
* Three line types have been added (OneTrackLine, TwoTrackLine, ODINLine).
* The RateChecker has been adapted to work with the new selections.
* All algorithms (that require it) have been given a `dev_event_list_t` as input.
* Algorithms that require it have been given a `dev_number_of_events_t` as input.
* Added "verbosity" property to all algorithms.
* `selections.md` documentation updated accordingly.
* Created a new implementation of a `dec_reporter`.
* Selection algorithms now have the following properties: a pre-scaler factor and hash string, and a post-scaler factor and hash string. The scaler factor is in the range [0-1] and determines how many events will be accepted. The hash string must not be empty and contains a string from which an `unsigned` hash is calculated which serves as the seed for the deterministic scaler.
To do:
- [x] Implement pre-scalers
- [x] Implement post-scalers
- [x] Fix warning in Debug build
- [x] Fix compilation with Gaudi (see [this failing ci-test](https://lhcb-nightlies.web.cern.ch/logs/build/nightly/lhcb-master-mr/1330/x86_64-centos7-gcc9-opt/Allen/) )
Implement the following lines:
- [x] PassThrough
- [x] ODINNoBias
- [x] ODINLumi
- [x] GECPassthrough
- [x] SingleHighPtMuon
- [x] LowPtMuon
- [x] DiMuonHighMass
- [x] DiMuonLowMass
- [x] LowPtDiMuon
- [x] DisplacedDiMuon
- [x] DiMuonSoft
- [x] D2KPi
- [x] D2PiPi
- [x] D2KK
- [x] DiMuonTrackEff
- [x] TrackMuonMVA
Note: The previous "ErrorEvent" line does not make sense in the context of the new selection framework, where each selection is an algorithm. Error handling deserves its own issue, should be discussed in depth and implemented in a separate MR.
GPU throughput of the sequence is reduced by 4% with this branch:
Prior throughput:
```
Quadro RTX 6000 │█████████████████████████████████ 166.02 kHz
GeForce RTX 2080 Ti │███████████████████████████████ 155.88 kHz
Tesla V100-PCIE-32GB │██████████████████████████████ 151.72 kHz
AMD EPYC 7502 32-Core │████ 22.44 kHz
Intel Xeon E5-2630 v4 │▌ 4.66 kHz
┼─┴─┼─┴─┼─┴─┼─┴─┼─┴─┼─┴─┼─┴─┼─┴─┼─┴─┼
0 20 40 60 80 100 120 140 160 180
```
New throughput:
```
Quadro RTX 6000 │███████████████████████████████████████████████ 159.90 kHz
GeForce RTX 2080 Ti │████████████████████████████████████████████ 149.55 kHz
Tesla V100-PCIE-32GB │██████████████████████████████████████████ 140.60 kHz
AMD EPYC 7502 32-Core │██████ 22.35 kHz
Intel Xeon E5-2630 v4 │█ 4.73 kHz
┼──┴──┼──┴──┼──┴──┼──┴──┼──┴──┼──┴──┼──┴──┼──┴──┼
0 20 40 60 80 100 120 140 160
```
Built on top of https://gitlab.cern.ch/lhcb/Allen/-/merge_requests/388.
Closes https://gitlab.cern.ch/lhcb/Allen/-/issues/131
Should go together with: https://gitlab.cern.ch/lhcb/Moore/-/merge_requests/652https://gitlab.cern.ch/lhcb/Allen/-/merge_requests/292add run changes to the main Allen loop2020-10-01T09:16:43+02:00Daniel Charles Craikadd run changes to the main Allen loop- `MEPProvider` and `MDFProvider` have both been modified to only fill a slice with events from a single run
- `InputProviders` now also return the run number on calls to `get_slice`
- `BinaryProvider` always reports a run number of `0` ...- `MEPProvider` and `MDFProvider` have both been modified to only fill a slice with events from a single run
- `InputProviders` now also return the run number on calls to `get_slice`
- `BinaryProvider` always reports a run number of `0` and does not split slices by run
- `run_slice` reports any change of run number by sending a `RUN` message to the main loop before the `SLICE` message
- Upon receiving a `RUN` message, the main loop will pause listening to I/O threads; wait for all streams to finish processing existing slices; call `updater->update` with the new run number; and then resume listening to I/O threads
- Splitting slices by run is disabled by the command-line argument `--disable-run-changes` (defaults to 1)https://gitlab.cern.ch/lhcb/Allen/-/merge_requests/281Allen TDR2020-08-03T18:30:23+02:00Daniel Campora PerezAllen TDR