Commit 7b0626f9 authored by Simon Spannagel's avatar Simon Spannagel
Browse files

Merge branch 'moduletypes' into 'master'

Major Rework of Central Corryvreckan Components

See merge request !56
parents 6e940573 810103a9
Pipeline #578143 passed with stages
in 14 minutes and 3 seconds
......@@ -31,3 +31,6 @@ CMakeLists.txt.user
# Ignore conflict files:
......@@ -8,7 +8,7 @@ MACRO(corryvreckan_enable_default val)
# Common module definitions
MACRO(corryvreckan_module name)
MACRO(_corryvreckan_module_define_common name)
# Get the name of the module
......@@ -60,6 +60,43 @@ Create the header or provide the alternative class name as first argument")
SET_PROPERTY(SOURCE "${PROJECT_SOURCE_DIR}/src/core/module/dynamic_module_impl.cpp" APPEND PROPERTY OBJECT_DEPENDS "${CMAKE_CURRENT_SOURCE_DIR}/${_corryvreckan_module_class}.h")
# Put this at the start of every global module
MACRO(corryvreckan_global_module name)
_corryvreckan_module_define_common(${name} ${ARGN})
# Set the unique flag to true
# Put this at the start of every detector module
MACRO(corryvreckan_detector_module name)
_corryvreckan_module_define_common(${name} ${ARGN})
# Set the unique flag to false
# Append list of possible detector types as compile definition
MACRO(corryvreckan_detector_type name)
SET(extra_macro_args ${ARGN})
LIST(LENGTH extra_macro_args num_extra_args)
IF(${num_extra_args} GREATER 0)
STRING(REPLACE ";" "," TYPESLIST "${extra_macro_args}")
# Put this at the start of every detector module
MACRO(corryvreckan_dut_module name)
_corryvreckan_module_define_common(${name} ${ARGN})
# Set the unique flag to false
# Add sources to the module
MACRO(corryvreckan_module_sources name)
# Get the list of sources
......@@ -160,6 +160,11 @@ The detector configuration consists of a set of sections describing the detector
Each section starts with a header describing the name used to identify the detector; all names are required to be unique.
Every detector should contain all of the following parameters:
\item The \parameter{role} parameter is an array of strings indicating the function(s) of the respective detector. This can be \parameter{dut} or \parameter{reference}, the default is \parameter{none}. With the default, the respective detector participates in tracking but is neither used as reference plane for alignment and correlations, nur treated as DUT. As reference, the detector is used as anchor for relative alignments, and its position and orientation is used to produce correlation plots. As DUT, the detector is by default excluded from tracking, and all DUT-type modules are executed for this detector.
There always has to be exactly \emph{one} reference detector in the setup. For setups with a single detector only, the role should be configured as \parameter{dut, reference} for the detector to act as both.
\item The \parameter{type} parameter is a string describing the type of detector, e.g.\ \parameter{Timepix3} or \parameter{CLICpix2}. This value might be used by some modules to distinguish between different types.
\item The \parameter{position} in the world frame.
This is the position of the geometric center of the sensor given in world coordinates as X, Y and Z as defined in Section~\ref{sec:coordinate_systems}.
......@@ -84,6 +84,170 @@ Another (additional) location to write to can be specified on the command line u
\item \parameter{library_directories}: Additional directories to search for module libraries, before searching the default paths.
\section{Modules and the Module Manager}
\corry is a modular framework and one of the core ideas is to partition functionality in independent modules which can be inserted or removed as required.
These modules are located in the subdirectory \textit{src/modules/} of the repository, with the name of the directory the unique name of the module.
The suggested naming scheme is CamelCase, thus an example module name would be \textit{OnlineMonitor}.
A \emph{specifying} part of a module name should precede the \emph{functional} part of the name, e.g.\ \textit{EventLoaderCLICpix2} rather than \textit{CLICpix2EventLoader}.
There are htree different kind of modules which can be defined:
\item \textbf{Global}: Modules for which a single instance runs, irrespective of the number of detectors.
\item \textbf{Detector}: Modules which are concerned with only a single detector at a time.
These are then replicated for all required detectors.
\item \textbf{DUT}: Similar to the Detector modules, these modules run for a single detector.
However, they are only replicated if the respective detector is marked as DUT.
The type of module determines the constructor used, the internal unique name and the supported configuration parameters.
For more details about the instantiation logic for the different types of modules, see Section~\ref{sec:module_instantiation}.
\subsection{Files of a Module}
Every module directory should at minimum contain the following documents (with \texttt{ModuleName} replaced by the name of the module):
\item \textbf{CMakeLists.txt}: The build script to load the dependencies and define the source files of the library.
\item \textbf{}: Full documentation of the module.
\item \textbf{\textit{ModuleName}.h}: The header file of the module.
\item \textbf{\textit{ModuleName}.cpp}: The implementation file of the module.
These files are discussed in more detail below.
By default, all modules added to the \textit{src/modules/} directory will be built automatically by CMake.
If a module depends on additional packages which not every user may have installed, one can consider adding the following line to the top of the module's \textit{CMakeLists.txt}:
Contains the build description of the module with the following components:
\item On the first line either \parameter{CORRYVRECKAN_DETECTOR_MODULE(MODULE_NAME)}, \parameter{CORRYVRECKAN_DUT_MODULE(MODULE_NAME)} or \parameter{CORRYVRECKAN_GLOBAL_MODULE(MODULE_NAME)} depending on the type of module defined.
The internal name of the module is automatically saved in the variable \parameter{${MODULE_NAME}} which should be used as an argument to other functions.
Another name can be used by overwriting the variable content, but in the examples below, \parameter{${MODULE_NAME}} is used exclusively and is the preferred method of implementation.
For DUT and Detector modules, the type of detector this module is capable of handling can be specified by adding so-called type restrictions, e.g.\
The module will then only be instantiated for detectors of one of the given types. This is particularly useful for event loader modules which read a very specific file format.
\item The following lines should contain the logic to load possible dependencies of the module (below is an example to load Geant4).
Only ROOT is automatically included and linked to the module.
\item A line with \texttt{\textbf{CORRYVRECKAN\_MODULE\_SOURCES(\$\{MODULE\_NAME\} \textit{sources})}} defines the module source files. Here, \texttt{sources} should be replaced by a list of all source files relevant to this module.
\item Possible lines to include additional directories and to link libraries for dependencies loaded earlier.
\item A line containing \parameter{CORRYVRECKAN_MODULE_INSTALL(${MODULE_NAME})} to set up the required target for the module to be installed to.
A simple CMakeLists.txt for a module named \parameter{Test} which should run only on DUT detectors of type \emph{Timepix3} is provided below as an example.
# Define module and save name to MODULE_NAME
# Add the sources for this module
# Provide standard install target
The \file{} serves as the documentation for the module and should be written in Markdown format~\cite{markdown}.
It is automatically converted to \LaTeX using Pandoc~\cite{pandoc} and included in the user manual in Chapter~\ref{ch:modules}.
By documenting the module functionality in Markdown, the information is also viewable with a web browser in the repository within the module sub-folder.
The \file{} should follow the structure indicated in the \file{} file of the \parameter{Dummy} module in \dir{src/modules/Dummy}, and should contain at least the following sections:
\item The H1-size header with the name of the module and at least the following required elements: the \textbf{Maintainer}, the \textbf{Module Type} and the \textbf{Status} of the module.
The module type should be either \textbf{GLOBAL}, \textbf{DETECTOR}, \textbf{DUT}.
If the module is working and well-tested, the status of the module should be \textbf{Functional}.
By default, new modules are given the status \textbf{Immature}.
The maintainer should mention the full name of the module maintainer, with their email address in parentheses.
A minimal header is therefore:
# ModuleName
Maintainer: Example Author (<>)
Module Type: GLOBAL
Status: Functional
In addition, the \textbf{Detector Type} should be mentioned for modules of types \textbf{DETECTOR} and \textbf{DUT}.
\item An H3-size section named \textbf{Description}, containing a short description of the module.
\item An H3-size section named \textbf{Parameters}, with all available configuration parameters of the module.
The parameters should be briefly explained in an itemised list with the name of the parameter set as an inline code block.
\item An H3-size section named \textbf{Plots Created}, listing all plots created by this module.
\item An H3-size section with the title \textbf{Usage} which should contain at least one simple example of a valid configuration for the module.
\paragraph{\texttt{ModuleName}.h and \texttt{ModuleName}.cpp}
All modules should consist of both a header file and a source file.
In the header file, the module is defined together with all of its methods.
Brief Doxygen documentation should be added to explain what each method does.
The source file should provide the implementation of every method and also its more detailed Doxygen documentation.
Methods should only be declared in the header and defined in the source file in order to keep the interface clean.
\subsection{Module structure}
All modules must inherit from the \texttt{Module} base class, which can be found in \textit{src/core/module/Module.hpp}.
The module base class provides two base constructors, a few convenient methods and several methods which the user is required to override.
Each module should provide a constructor using the fixed set of arguments defined by the framework; this particular constructor is always called during by the module instantiation logic.
These arguments for the constructor differ for global and detector/DUT modules.
For global modules, the constructor for a \texttt{TestModule} should be:
TestModule(Configuration& config, std::vector<std::shared_ptr<Detector>> detectors): Module(std::move(config), detectors) {}
For detector and DUT modules, the first argument are the same, but the last argument is a \texttt{std::shared\_ptr} to the linked detector.
It should always forward this detector to the base class together with the configuration object.
Thus, the constructor of a detector module is:
TestModule(Configuration& config, std::shared_ptr<Detector> detector): Module(std::move(config), std::move(detector)) {}
In addition to the constructor, each module can override the following methods:
\item \parameter{initialise()}: Called after loading and constructing all modules and before starting the analysis loop.
This method can for example be used to initialize histograms.
\item \parameter{run(std::shared_ptr<Clipboard> clipboard)}: Called for every time frame or triggered event to be analyzed in the simulation. The argument represents a pointer to the clipboard where the event data is stored.
A status code is returned to signal the framework whether to continue processing data or to end the run.
\item \parameter{finalise()}: Called after processing all events in the run and before destructing the module.
Typically used to save the output data (like histograms).
Any exceptions should be thrown from here instead of the destructor.
\subsection{Module instantiation}
Modules are dynamically loaded and instantiated by the Module Manager.
They are constructed, initialized, executed and finalized in the linear order in which they are defined in the configuration file; for this reason the configuration file should follow the order of the real process.
For each section in the main configuration file (see~\ref{sec:config_parameters} for more details), a corresponding library is searched for which contains the module (the exception being the global framework section).
Module libraries are always named following the scheme \textbf{libCorryvreckanModule\texttt{ModuleName}}, reflecting the \texttt{ModuleName} configured via CMake.
The module search order is as follows:
\item Modules already loaded before from an earlier section header
\item All directories in the global configuration parameter \parameter{library_directories} in the provided order, if this parameter exists.
\item \item The internal library paths of the executable, that should automatically point to the libraries that are built and installed together with the executable.
These library paths are stored in \dir{RPATH} on Linux, see the next point for more information.
\item The other standard locations to search for libraries depending on the operating system.
Details about the procedure Linux follows can be found in~\cite{linuxld}.
If the loading of the module library is successful, the module is checked to determine if it is a global, detector or DUT module.
As a single module may be called multiple times in the configuration, with overlapping requirements (such as a module which runs on all detectors of a given type, followed by the same module but with different parameters for one specific detector, also of this type) the Module Manager must establish which instantiations to keep and which to discard.
The instantiation logic determines a unique name and priority, where a lower number indicates a higher priority, for every instantiation.
The name and priority for the instantiation are determined differently for the two types of modules:
\item \textbf{Global}: Name of the module, the priority is always \emph{high}.
\item \textbf{Detector/DUT}: Combination of the name of the module and the name of detector this module is executed for.
If the name of the detector is specified directly by the \parameter{name} parameter, the priority is \emph{high}.
If the detector is only matched by the \parameter{type} parameter, the priority is \emph{medium}.
If the \parameter{name} and \parameter{type} are both unspecified and the module is instantiated for all detectors, the priority is \emph{low}.
In the end, only a single instance for every unique name is allowed.
If there are multiple instantiations with the same unique name, the instantiation with the highest priority is kept.
If multiple instantiations with the same unique name and the same priority exist, an exception is raised.
\section{Logging and Verbosity Levels}
\corry is designed to identify mistakes and implementation errors as early as possible and to provide the user with clear indications about the problem.
......@@ -5,6 +5,17 @@ echo -e "\nPreparing code basis for a new module:\n"
# Ask for module name:
read -p "Name of the module? " MODNAME
# Ask for module type:
echo -e "Type of the module?\n"
select yn in "global" "detector" "dut"; do
case $yn in
global ) type=1; break;;
detector ) type=2; break;;
dut ) type=3; break;;
echo "Creating directory and files..."
......@@ -53,6 +64,32 @@ opt=-i
if [ "$platform" == "Darwin" ]; then opt="-i \"\""; fi
# Change to detector or dut module type if necessary:
if [ "$type" != 1 ]; then
# Prepare sed commands to change to per detector module
# Change module type in CMakeLists
if [ "$type" == 3 ]; then
command="sed $opt 's/_GLOBAL_/_DUT_/g' $MODDIR/$MODNAME/CMakeLists.txt"
command="sed $opt 's/_GLOBAL_/_DETECTOR_/g' $MODDIR/$MODNAME/CMakeLists.txt"
eval $command
# Change header file
command="sed ${opt} \
-e 's/param detectors.*/param detector Pointer to the detector for this module instance/g' \
-e 's/std::vector<Detector\*> detectors/Detector\* detector/g' \
eval $command
# Change implementation file
command="sed ${opt} \
-e 's/std::vector<Detector\*> detectors/Detector\* detector/g' \
-e 's/move(detectors)/move\(detector\)/g' \
eval $command
# Print a summary of the module created:
echo "Name: $MODNAME"
# Create core library
......@@ -14,6 +13,7 @@ ADD_LIBRARY(CorryvreckanCore SHARED
# Link the dependencies
......@@ -5,13 +5,11 @@
using namespace corryvreckan;
void Clipboard::put(std::string name, Objects* objects) {
m_data[name] = objects;
m_data.insert(ClipboardData::value_type(name, objects));
void Clipboard::put(std::string name, std::string type, Objects* objects) {
m_dataID.push_back(name + type);
m_data[name + type] = objects;
m_data.insert(ClipboardData::value_type(name + type, objects));
void Clipboard::put_persistent(std::string name, double value) {
......@@ -38,20 +36,20 @@ bool Clipboard::has_persistent(std::string name) {
void Clipboard::clear() {
for(auto& id : m_dataID) {
Objects* collection = m_data[id];
for(Objects::iterator it = collection->begin(); it != collection->end(); it++)
for(auto set = m_data.cbegin(); set != m_data.cend();) {
Objects* collection = set->second;
for(Objects::iterator it = collection->begin(); it != collection->end(); ++it) {
delete m_data[id];
delete collection;
set = m_data.erase(set);
std::vector<std::string> Clipboard::listCollections() {
std::vector<std::string> collections;
for(auto& name : m_dataID) {
for(auto& dataset : m_data) {
return collections;
......@@ -98,11 +98,10 @@ namespace corryvreckan {
std::vector<std::string> listCollections();
// Container for data, list of all data held
std::map<std::string, Objects*> m_data;
typedef std::map<std::string, Objects*> ClipboardData;
// List of available data collections
std::vector<std::string> m_dataID;
// Container for data, list of all data held
ClipboardData m_data;
// Persistent clipboard storage
std::unordered_map<std::string, double> m_persistent_data;
......@@ -22,6 +22,7 @@ using namespace ROOT::Math;
using namespace corryvreckan;
Detector::Detector(const Configuration& config) : m_role(DetectorRole::NONE) {
// Role of this detector:
auto roles = config.getArray<std::string>("role", std::vector<std::string>{"none"});
for(auto& role : roles) {
......@@ -67,8 +67,8 @@ namespace corryvreckan {
~Detector() {}
// Functions to retrieve basic information
std::string type() { return m_detectorType; }
std::string name() { return m_detectorName; }
const std::string type() const { return m_detectorType; }
const std::string name() const { return m_detectorName; }
// Detector role and helper functions
bool isReference();
......@@ -10,23 +10,55 @@
using namespace corryvreckan;
Module::Module(Configuration config, std::vector<Detector*> detectors) {
m_name = config.getName();
m_config = config;
m_detectors = detectors;
std::stringstream det;
for(auto& d : m_detectors) {
det << d->name() << ", ";
LOG(TRACE) << "Module determined to run on detectors: " << det.str();
Module::Module(Configuration config, std::shared_ptr<Detector> detector)
: Module(std::move(config), std::vector<std::shared_ptr<Detector>>{detector}) {}
Module::~Module() {}
Module::Module(Configuration config, std::vector<std::shared_ptr<Detector>> detectors)
: m_config(std::move(config)), m_detectors(std::move(detectors)) {}
* @throws InvalidModuleActionException If this method is called from the constructor
* This name is guaranteed to be unique for every single instantiation of all modules
std::string Module::getUniqueName() const {
std::string unique_name = get_identifier().getUniqueName();
if(unique_name.empty()) {
throw InvalidModuleActionException("Cannot uniquely identify module in constructor");
return unique_name;
Module::~Module() {}
void Module::set_identifier(ModuleIdentifier identifier) {
identifier_ = std::move(identifier);
ModuleIdentifier Module::get_identifier() const {
return identifier_;
Detector* Module::get_detector(std::string name) {
auto it = find_if(m_detectors.begin(), m_detectors.end(), [&name](Detector* obj) { return obj->name() == name; });
* @throws InvalidModuleActionException If this method is called from the constructor or destructor
* @warning Cannot be used from the constructor, because the instantiation logic has not finished yet
* @warning This method should not be accessed from the destructor (the file is then already closed)
* @note It is not needed to change directory to this file explicitly in the module, this is done automatically.
TDirectory* Module::getROOTDirectory() const {
// The directory will only be a null pointer if this method is executed from the constructor or destructor
if(directory_ == nullptr) {
throw InvalidModuleActionException("Cannot access ROOT directory in constructor or destructor");
return directory_;
void Module::set_ROOT_directory(TDirectory* directory) {
directory_ = directory;
std::shared_ptr<Detector> Module::get_detector(std::string name) {
auto it = find_if(
m_detectors.begin(), m_detectors.end(), [&name](std::shared_ptr<Detector> obj) { return obj->name() == name; });
if(it == m_detectors.end()) {
throw ModuleError("Device with detector ID " + name + " is not registered.");
......@@ -34,13 +66,12 @@ Detector* Module::get_detector(std::string name) {
return (*it);
Detector* Module::get_reference() {
auto it = find_if(m_detectors.begin(), m_detectors.end(), [](Detector* obj) { return obj->isReference(); });
return (*it);
std::shared_ptr<Detector> Module::get_reference() {
return m_reference;
Detector* Module::get_dut() {
auto it = find_if(m_detectors.begin(), m_detectors.end(), [](Detector* obj) { return obj->isDUT(); });
std::shared_ptr<Detector> Module::get_dut() {
auto it = find_if(m_detectors.begin(), m_detectors.end(), [](std::shared_ptr<Detector> obj) { return obj->isDUT(); });
if(it == m_detectors.end()) {
return nullptr;
......@@ -49,7 +80,8 @@ Detector* Module::get_dut() {
bool Module::has_detector(std::string name) {
auto it = find_if(m_detectors.begin(), m_detectors.end(), [&name](Detector* obj) { return obj->name() == name; });
auto it = find_if(
m_detectors.begin(), m_detectors.end(), [&name](std::shared_ptr<Detector> obj) { return obj->name() == name; });
if(it == m_detectors.end()) {
return false;
......@@ -11,6 +11,7 @@
#include <string>
#include "ModuleIdentifier.hpp"
#include "core/clipboard/Clipboard.hpp"
#include "core/config/Configuration.hpp"
#include "core/detector/Detector.hpp"
......@@ -49,6 +50,7 @@ namespace corryvreckan {
* - Module::finalise(): for finalising the module at the end
class Module {
friend class ModuleManager;
......@@ -57,11 +59,18 @@ namespace corryvreckan {
Module() = delete;
* @brief Base constructor for modules
* @brief Base constructor for detector modules
* @param config Configuration for this module
* @param detector Pointer to the detector this module is bound to
Module(Configuration config, std::shared_ptr<Detector> detector);
* @brief Base constructor for unique modules
* @param config Configuration for this module
* @param detectors List of detectors this module should be aware of
Module(Configuration config, std::vector<Detector*> detectors);
Module(Configuration config, std::vector<std::shared_ptr<Detector>> detectors);
* @brief Essential virtual destructor.
......@@ -75,10 +84,11 @@ namespace corryvreckan {
Module& operator=(const Module&) = delete;
* @brief Get the name of this module
* @return Name of the module
* @brief Get the unique name of this module
* @return Unique name
* @note Can be used to interact with ROOT objects that require an unique name
std::string getName() { return m_name; }
std::string getUniqueName() const;
* @brief Get the configuration for this module
......@@ -100,7 +110,7 @@ namespace corryvreckan {
* Does nothing if not overloaded.
virtual StatusCode run(Clipboard*) { return Success; }
virtual StatusCode run(std::shared_ptr<Clipboard>) { return Success; }
* @brief Finalise the module after the event sequence
......@@ -110,6 +120,12 @@ namespace corryvreckan {
virtual void finalise(){};
* @brief Get ROOT directory which should be used to output histograms et cetera
* @return ROOT directory for storage
TDirectory* getROOTDirectory() const;
// Configuration of this module
Configuration m_config;
......@@ -118,7 +134,7 @@ namespace corryvreckan {
* @brief Get list of all detectors this module acts on
* @return List of all detectors for this module
std::vector<Detector*> get_detectors() { return m_detectors; };
std::vector<std::shared_ptr<Detector>> get_detectors() { return m_detectors; };
* @brief Get the number of detectors this module takes care of
......@@ -132,20 +148,20 @@ namespace corryvreckan {
* @return Pointer to the requested detector
* @throws ModuleError if detector with given name is not found for this module
Detector* get_detector(std::string name);
std::shared_ptr<Detector> get_detector(std::string name);
* @brief Get the reference detector for this setup
* @return Pointer to the reference detector
Detector* get_reference();
std::shared_ptr<Detector> get_reference();
* @brief Get the device under test
* @return Pointer to the DUT detector. A nullptr is returned if no DUT is found.
* FIXME This should allow retrieval of a vector of DUTs
Detector* get_dut();
std::shared_ptr<Detector> get_dut();
* @brief Check if this module should act on a given detector
......@@ -155,11 +171,31 @@ namespace corryvreckan {
bool has_detector(std::string name);