FEST, HLT1 PRR and beam test

This branch collects all developments for FEST and the HLT1 PRR and has been rebased on master:

• New cmake configuration for Allen
• Creation of the InputProvider and OutputHandler factored out of Allen.cpp
• Fixes to output MDF writing
• Bank types configurable on the command line (should be improved to obtain the types from the sequence and configuration)
• Installation of headers and libAllenLib such that it can be used from other projects, e.g. MooreOnline
• Added a passthrough sequence

Suggested TODO:

• Fix the CI and tests
• Test MDF output for regressions
• Add copying of banks to device to passthrough sequence, which is needed to test data throughput to GPU with EB)
• Obtain the bank types that are needed from a combination of the sequence and the JSON configuration.
• Add an Allen::runtime exported target that can be used downstream to link against the device runtime (implemented using Allen::Backend)

FYI @dovombru @dcampora @mfontana @rzan

Goes with MooreOnline!70 (closed)

added FEST not for merge labels

assigned to @mfontana and @dovombru

changed the description

added 1 commit

• e6fae437 - Fixed formatting

Compare with previous version

added 1 commit

• 9bc6ff56 - Add copyright

Compare with previous version

added 1 commit

• a0398933 - Fix standalone build

Compare with previous version

added 4 commits

• d31ca53d - Update CI scripts for rename of BUILD_TESTS to BUILD_TESTING
• 0eb6b203 - Fix backend refactor
• 78feea46 - Move some HIP detection stuff to AllenDependencies
• 870a60e0 - Fix standalone build

Compare with previous version

added 1 commit

• 8328ab3b - Update backend library for use from downstream projects

Compare with previous version

changed the description

I have not had the time to look at the changes in detail yet, but these questions come up for me now:

• What exactly do you have in mind for improvements of the command-line arguments?
• Which types of tests do you have in mind for the MDF output?
• I'm not sure that splitting the branch by commits into separate branches is worth the effort, especially since it will have to be someone who is not the branch author. What is your worry with working on this branch (apart from it being nicer to have one branch per feature)?

marked the checklist item Add an Allen::runtime exported target that can be used downstream to link against the device runtime as completed

Command-line arguments

• A single struct with members that represent the final values of each parameter. So after parsing, sanity checks and updates because of the values of other parameters. That would imply hardcoding all the members, which is not ideal.
• A object that uses type erasure to store the arguments and exposes methods along the lines of template<typename R> R get(std::string_view par) const;.
• Move to boost::program_options (we already depend on Boost anyway) and pass the variables_map around.

Tests for the MDF output

The files that are written should be readable by a Gaudi application that includes the DecReport and SelReport decoders. It should also be possible to run HLT2 on it.

The branch

I don't have a preference for splitting the branch. It's quite a large set of changes, but whatever works to get things merged.

changed the description

@clemenci @rmatev This MR allows building Allen with the "modern" cmake approach.

added 1 commit

• 1b755b37 - Move device detection for stack build to AllenDependencies

Compare with previous version

I find the support for the new-style cmake configuration good, apart for a few comments.

You should review the need of FindXYZ.cmake modules, for example you should not use FindZMQ.cmake, but use pkg_check_modules(libzmq libzmq REQUIRED IMPORTED_TARGET).

removed not for merge label

changed the description

mentioned in merge request MooreOnline!70 (closed)

mentioned in issue #267 (closed)

unassigned @dovombru

mentioned in merge request !655 (merged)

mentioned in issue #151 (closed)

marked the checklist item Test MDF output for regressions as completed

marked the checklist item Test MDF output for regressions as incomplete

added 1 commit

• 1d6e34bf - temporarily removed gitlab-ci.yml; please cherry-pick commits 06353b3f...

Compare with previous version

added 136 commits

• 1d6e34bf...09a813a3 - 50 commits from branch master
• 2bd51f99 - First prototype that converts cmake include directories into target include directories.
• 48d1850d - Remove include_directories from muon/CMakeLists.txt
• 50c7e717 - Converted all CMakeLists include_directories into target_include_directories.
• dce837f2 - Unified Boost memory requirement. Fixed compilation with CUDA.
• 780e191d - Fixed formatting
• a99f4bd4 - Added copyright to missing files.
• 6922bacb - Fixed HIP backend too.
• 0ef2a68c - Remove leftover file
• f7967c38 - inline some methods
• cea06de4 - Fix stack build
• c692e9f0 - Add Catch2 again
• f437bba2 - WIP cmake modernization and device builds
• 4c1b818d - WIP CMake modernization
• 2bb4404a - Fix clang error
• 49460c33 - WIP cmake-modernization
• 4dc70514 - InputProvider function
• ee3ed5b7 - Prepare for EBProvider
• 43f034b0 - Untemplate providers
• db317643 - Fix test
• 42fca858 - Fix make_provider
• afd010f0 - Fix standalone build.
• 53f7220a - Move MEP stuff to AllenOnline
• 077c4cbe - Add lhcbproject.yml
• 1376a4dd - Move Property parsing for RawBank::BankType to separate file.
• e00d5023 - Also install Timer.h
• 470016da - Fix standalone build.
• 3fea6393 - Put provider functions in Allen namespace
• abc200fe - Small CMake cleanup
• f4e24858 - Split bank mapping into separate file.
• f47914bc - Remove non-stop option and improve io configuration
• 78200cfe - Move MEP and MPI to AllenOnline
• f9097bfd - Move AllenTypeTraits.cuh to be able to install it
• 701b81e0 - Remove MPI stuff
• 145acaa9 - Refactor backend to allow usage of headers in other projects
• ae5e56ab - Add missing file
• 77e6ba8b - Fix regression in MDFProvider
• f523eb6e - Initialize number of lines in output writers.
• 333bd6e0 - Fix spurious 0s after detector bank content when writing MDF files from MDF input.
• 26cc3ce5 - Update pregenerated sequences
• f1943365 - Fix data-not-there issue
• 21fe125d - Fix merge
• fae2fa0d - Fix SelReports merge
• 6e2ba77f - Fix parsing of kalman_velo_only
• fa3a36e1 - Fix allen.py
• 5c7f4a0d - Fix CPU build
• f866b72e - Add assert to event reading
• bf8e1b3a - Fix mdf subdirectory build with old cmake
• c58e5bfc - Set n_events in Provider.cpp
• b5e7daa3 - Update average bank sizes
• 4835d0bb - Print status messages about number of processed events and rate of processing
• a73ff831 - Fix reporting of number of events output
• b6d0871a - Fix reading n_events in MDFProvider
• dad889e4 - Put some extra types into the Allen namespace
• e2393f6e - Move BufferStatus into IInputProvider to avoid future name clashes
• c9e83e7b - Refactor creation of OutputHandler into separate function
• 98da2793 - Add start and stop to OutputHandler
• 4563bbb6 - Use LHCb version of RawBank::BankType parser
• eca9e2d5 - Changes needed to allow online running
• debc9684 - Add option to force boost::filesystem to be used
• eaf587a0 - Skip SV monitor for now
• ec9cf853 - Scripts uses for FEST
• 6ce2db2b - Update service cast helper to make it more generic
• 3e2d5345 - Add ReadOnly and IoMemory types to host memory registration with device runtime
• 2ce7a64e - Remove unused arguments from init_event_list and passthrough line
• 0b9abfcd - Remove unused command-line from Allen (moved to MEPProvider in AllenOnline
• 307278b6 - Add buffer index to BufferStatus struct
• 67c16835 - Add function to InputProvider to release provider buffers
• f57e758b - Don't try to write the histograms to a file without a filename
• ae84659e - Allow using MEPProvider for allen.py
• 9c92ecf6 - Fix formatting
• 922f4224 - Updates to telegraf/grafana monitoring scripts for PRR tests
• 54e582e1 - Fix test_mep_banks after move of MEPProvider to MooreOnline
• de9e3c02 - Add passthrough sequence that only copies data to the device
• f1066a35 - WIP make write_mep.py more usable
• f54f1d5c - Fix line ID in dec reports
• 3a960c6f - Fix rebase
• ccb11d51 - Fixed formatting
• 4e927d9c - Add copyright
• 127b6fc9 - Fix standalone build
• 177c2be0 - Update CI scripts for rename of BUILD_TESTS to BUILD_TESTING
• ebb7420f - Fix backend refactor
• 8bf1bfbc - Move some HIP detection stuff to AllenDependencies
• 7e24530e - Fix standalone build
• ef2cf8b4 - Update backend library for use from downstream projects
• 21a9683a - Move device detection for stack build to AllenDependencies
• 14cf7cb2 - Fix rebase

Compare with previous version

added 1 commit

• d03b606b - Add missing script

Compare with previous version

added 1 commit

• 2204b3d7 - Fix CPU build

Compare with previous version

added 1 commit

• adef3938 - Re-add missing mep argument to allen.py

Compare with previous version

added 1 commit

• 975bb4fa - Fix includes in InputProvider.h

Compare with previous version

added 1 commit

• d2984282 - Explicitly initialize std::optional

Compare with previous version

added 3 commits

• 703ddc5c - Fix padding of output banks and only write SelReports when they exist
• d548d3c2 - Allow output cancel to actually be hit
• d40d6ff1 - Always build tests in stack build and fix ALLEN_INSTALL_DIR

Compare with previous version

added 1 commit

• cb55421a - Don't count DAQ bank when summing up bank sizes

Compare with previous version

added 1 commit

• 5b4a0ecd - Do output per batch instead of per event

Compare with previous version

mentioned in merge request Moore!1149 (merged)

added 222 commits

• 5b4a0ecd...5e6d1c10 - 123 commits from branch master
• a59380dd - First prototype that converts cmake include directories into target include directories.
• 74eed519 - Remove include_directories from muon/CMakeLists.txt
• 2c5100c1 - Converted all CMakeLists include_directories into target_include_directories.
• f2aea781 - Unified Boost memory requirement. Fixed compilation with CUDA.
• f996d032 - Fixed formatting
• a189727c - Added copyright to missing files.
• dc0fa563 - Fixed HIP backend too.
• 6ff4ce2f - Remove leftover file
• 22c5aa19 - inline some methods
• 19bd70c5 - Fix stack build
• 4840b5cd - Add Catch2 again
• 0affbd52 - WIP cmake modernization and device builds
• fe98ce66 - WIP CMake modernization
• 831c8259 - Fix clang error
• 837172ea - WIP cmake-modernization
• e03ce63c - InputProvider function
• 2d15a5c7 - Prepare for EBProvider
• 47343a0e - Untemplate providers
• 21008c7b - Fix test
• cc39c88f - Fix make_provider
• ca587bba - Fix standalone build.
• 4cadcd55 - Move MEP stuff to AllenOnline
• c2c07316 - Add lhcbproject.yml
• 6500c068 - Move Property parsing for RawBank::BankType to separate file.
• 8cda11a3 - Also install Timer.h
• e625b6d2 - Fix standalone build.
• 108710d5 - Put provider functions in Allen namespace
• 326609d8 - Small CMake cleanup
• 51f4032c - Split bank mapping into separate file.
• b8cf1117 - Remove non-stop option and improve io configuration
• e88cc79b - Move MEP and MPI to AllenOnline
• c050e65d - Move AllenTypeTraits.cuh to be able to install it
• 812276db - Remove MPI stuff
• d78beb57 - Refactor backend to allow usage of headers in other projects
• 61c900fe - Add missing file
• 157238aa - Fix regression in MDFProvider
• c9045090 - Initialize number of lines in output writers.
• c28221ec - Fix spurious 0s after detector bank content when writing MDF files from MDF input.
• 459874ad - Update pregenerated sequences
• 1b3fd01e - Fix data-not-there issue
• c7f102de - Fix merge
• c64e1b37 - Fix SelReports merge
• 54f6cd10 - Fix parsing of kalman_velo_only
• b79990fa - Fix allen.py
• c138629d - Fix CPU build
• e48fd7f7 - Add assert to event reading
• ff4588c4 - Fix mdf subdirectory build with old cmake
• 0c7868b5 - Set n_events in Provider.cpp
• 5134deed - Update average bank sizes
• 1bcde43a - Print status messages about number of processed events and rate of processing
• f57041f1 - Fix reporting of number of events output
• c72bf55f - Fix reading n_events in MDFProvider
• c59af999 - Put some extra types into the Allen namespace
• 5989320e - Move BufferStatus into IInputProvider to avoid future name clashes
• 8429797f - Refactor creation of OutputHandler into separate function
• df62717c - Add start and stop to OutputHandler
• 77e0a337 - Use LHCb version of RawBank::BankType parser
• b21a1a03 - Changes needed to allow online running
• e084a0b3 - Add option to force boost::filesystem to be used
• adf39e62 - Skip SV monitor for now
• a95b1124 - Scripts uses for FEST
• ea4ea4bc - Update service cast helper to make it more generic
• 224336c4 - Add ReadOnly and IoMemory types to host memory registration with device runtime
• 606aaa26 - Remove unused arguments from init_event_list and passthrough line
• 64b4641b - Remove unused command-line from Allen (moved to MEPProvider in AllenOnline
• 96ce6045 - Add buffer index to BufferStatus struct
• 8d1590a3 - Add function to InputProvider to release provider buffers
• 70e12a8b - Don't try to write the histograms to a file without a filename
• 30567549 - Allow using MEPProvider for allen.py
• 94c3c645 - Fix formatting
• 8c3380d6 - Updates to telegraf/grafana monitoring scripts for PRR tests
• e62d7ce9 - Fix test_mep_banks after move of MEPProvider to MooreOnline
• f0cd99ab - Add passthrough sequence that only copies data to the device
• 8f0e547c - WIP make write_mep.py more usable
• 4215ad65 - Fix line ID in dec reports
• 96ff7b56 - Fix rebase
• 5930de5b - Fixed formatting
• 82ab6a49 - Add copyright
• 7e1707c8 - Fix standalone build
• 5ff0b133 - Update CI scripts for rename of BUILD_TESTS to BUILD_TESTING
• a8b4524d - Fix backend refactor
• 66fb49c4 - Move some HIP detection stuff to AllenDependencies
• 79dd1efe - Fix standalone build
• 42764cc2 - Update backend library for use from downstream projects
• b968b0ee - Move device detection for stack build to AllenDependencies
• 1f6c45c3 - Fix rebase
• d92e8487 - Add missing script
• 566b5396 - Fix CPU build
• 354ef77d - Re-add missing mep argument to allen.py
• 4fbb5942 - Fix includes in InputProvider.h
• 8639ed9d - Explicitly initialize std::optional
• b395642b - Fix padding of output banks and only write SelReports when they exist
• 324af395 - Allow output cancel to actually be hit
• 3dda886e - Always build tests in stack build and fix ALLEN_INSTALL_DIR
• 91d6e5da - Don't count DAQ bank when summing up bank sizes
• 5dd47867 - Do output per batch instead of per event
• 3151dceb - Fix sequence generation script for the stack
• cfb27917 - Updates/fixes after rebase
• 06075028 - Fix test_mdf_transpose

Compare with previous version

added 4 commits

• e9248090 - Add missing algorithms to build
• 023fda94 - Generate JSON files for all sequences by default
• 78ce80a3 - Remove pregenerated sequence
• 007b635f - Update Findcppgsl from Gaudi and use imported target

Compare with previous version