Commit 176c98aa authored by Simon Spannagel's avatar Simon Spannagel
Browse files

Merge branch 'documentation' into 'master'

Update of Documentation

See merge request simonspa/corryvreckan!41
parents a9270b43 0b0399ab
Pipeline #485183 passed with stages
in 5 minutes and 3 seconds
......@@ -100,11 +100,21 @@ IF(LATEX_COMPILER)
\chapter{Configuration Files}
The framework is configured with human-readable key/value based configuration files.
The configuration format consists of section headers within $[$ and $]$ brackets, and a global section without header at the start.
Each of these sections contains a set of key/value pairs separated by the \texttt{=} character.
Comments are indicated using the hash symbol (\texttt{\#}).
The framework has the following two required layers of configuration files:
\item The \textbf{main} configuration: The most important configuration file and the file that is passed directly to the binary.
Contains both the global framework configuration and the list of modules to instantiate together with their configuration.
\item The \textbf{detector} configuration passed to the framework to determine the geometry.
Describes the detector setup, containing the position, orientation and type of all detectors along with additional properties crucial for the reconstruction.
In the following paragraphs, the available types and the unit system are explained and an introduction to the different configuration files is given.
\section{Parsing types and units}
The \corry framework supports the use of a variety of types for all configuration values.
The module requesting the configuration key specifies how the value type should be interpreted.
An error will be raised if either the key is not specified in the configuration file, the conversion to the desired type is not possible, or if the given value is outside the domain of possible options.
Please refer to the module documentation in Chapter~\ref{ch:modules} for the list of module parameters and their types.
Parsing the value roughly follows common-sense, but a few special rules do apply:
\item If the value is a \textbf{string}, it may be enclosed by a single pair of double quotation marks (\texttt{"}), which are stripped before passing the value to the modules.
If the string is not enclosed by quotation marks, all whitespace before and after the value is erased.
If the value is an array of strings, the value is split at every whitespace or comma (\texttt{,}) that is not enclosed in quotation marks.
\item If the value is a \textbf{boolean}, either numerical (\texttt{0}, \texttt{1}) or textual (\texttt{false}, \texttt{true}) representations are accepted.
\item If the value is a \textbf{relative path}, that path will be made absolute by adding the absolute path of the directory that contains the configuration file where the key is defined.
\item If the value is an \textbf{arithmetic} type, it may have a suffix indicating the unit.
The list of base units is shown in Table~\ref{tab:units}.
If no units are specified, values will always be interpreted in the base units of the framework.
In some cases this can lead to unexpected results.
E.g. specifying a bias voltage as \texttt{bias\_voltage = 50} results in an applied voltage of \SI{50}{\mega\volt}.
Therefore it is strongly recommended to always specify units in the configuration files.
The internal base units of the framework are not chosen for user convenience but for maximum precision of the calculations and in order to avoid the necessity of conversions in the code.
\caption{List of units supported by \corry}
\textbf{Quantity} & \textbf{Default unit} & \textbf{Auxiliary units} \\
\multirow{6}{*}{\textit{Length}} & \multirow{6}{*}{mm (millimeter)} & nm (nanometer) \\
& & um (micrometer) \\
& & cm (centimeter) \\
& & dm (decimeter) \\
& & m (meter) \\
& & km (kilometer) \\
\multirow{4}{*}{\textit{Time}} & \multirow{4}{*}{ns (nanosecond)} & ps (picosecond) \\
& & us (microsecond) \\
& & ms (millisecond) \\
& & s (second) \\
\multirow{3}{*}{\textit{Energy}} & \multirow{4}{*}{MeV (megaelectronvolt)} & eV (electronvolt) \\
& & keV (kiloelectronvolt) \\
& & GeV (gigaelectronvolt) \\
\textit{Temperature} & K (kelvin) & --- \\
\multirow{2}{*}{\textit{Charge}} & e (elementary charge) & C (coulomb) \\
& & ke (kiloelectrons) \\
\multirow{2}{*}{\textit{Voltage}} & \multirow{2}{*}{MV (megavolt)} & V (volt) \\
& & kV (kilovolt) \\
\multirow{2}{*}{\textit{Angle}} & \multirow{2}{*}{rad (radian)} & deg (degree) \\
& & mrad (milliradian) \\
Combinations of base units can be specified by using the multiplication sign \texttt{*} and the division sign \texttt{/} that are parsed in linear order (thus $\frac{V m}{s^2}$ should be specified as $V*m/s/s$).
The framework assumes the default units (as given in Table~\ref{tab:units}) if the unit is not explicitly specified.
It is recommended to always specify the unit explicitly for all parameters that are not dimensionless as well as for angles.
Examples of specifying key/values pairs of various types are given below:
# All whitespace at the front and back is removed
first_string = string_without_quotation
# All whitespace within the quotation marks is preserved
second_string = " string with quotation marks "
# Keys are split on whitespace and commas
string_array = "first element" "second element","third element"
# Integer and floats can be specified in standard formats
int_value = 42
float_value = 123.456e9
# Units can be passed to arithmetic type
energy_value = 1.23MeV
time_value = 42ns
# Units are combined in linear order
acceleration_value = 1.0m/s/s
# Thus the quantity below is the same as 1.0deg*kV*K/m/s
random_quantity = 1.0deg*kV/m/s*K
# Relative paths are expanded to absolute
# Path below will be /home/user/test if the config file is in /home/user
output_path = "test"
# Booleans can be represented in numerical or textual style
my_switch = true
my_other_switch = 0
\section{Main configuration}
The main configuration consists of a set of sections specifying the modules used.
All modules are executed in the \emph{linear} order in which they are defined.
There are a few section names which have a special meaning in the main configuration, namely the following:
\item The \textbf{global} (framework) header sections: These are all zero-length section headers (including the one at the beginning of the file) and all sections marked with the header \texttt{[Corryvreckan]} (case-insensitive).
These are combined and accessed together as the global configuration, which contain all parameters of the framework itself (see Section~\ref{sec:framework_parameters} for details).
All key-value pairs defined in this section are also inherited by all individual configurations as long the key is not defined in the module configuration itself. This is encouraged for module parameters used in multiple modules.
\item The \textbf{ignore} header sections: All sections with name \texttt{[Ignore]} are ignored.
Key-value pairs defined in the section as well as the section itself are discarded by the parser.
These section headers are useful for quickly enabling and disabling individual modules by replacing their actual name by an ignore section header.
All other section headers are used to instantiate modules of the respective name.
Installed module libraries are loaded automatically at startup.
Parameters defined under the header of a module are local to that module and are not inherited by other modules.
An example for a valid albeit illustrative \corry main configuration file is:
# Key is part of the empty section and therefore the global configuration
string_value = "example1"
# The location of the detector configuration is a global parameter
detectors_file = "testbeam_setup.conf"
# The Corryvreckan section is also considered global and merged with the above
another_random_string = "example2"
# Stop after one thousand events:
number_of_events = 1000
# First run "ModuleA"
# This module takes no parameters
# Ignore this section:
my_key = "my_value"
int_value = 2
vector_of_doubles = 23.0, 45.6, 78.9
\section{Detector configuration}
The detector configuration consists of a set of sections describing the detectors in the setup.
Each section starts with a header describing the name used to identify the detector; all names are required to be unique.
Every detector should contain all of the following parameters:
\item The \parameter{type} parameter is a string describing the type of detector, e.g.\ \parameter{Timepix3} or \parameter{CLICpix2}. This value might be used by some modules to distinguish between different types.
\item The \parameter{position} in the world frame.
This is the position of the geometric center of the sensor given in world coordinates as X, Y and Z as defined in Section~\ref{sec:coordinate_systems}.
\item An \parameter{orientation_mode} that determines the way that the orientation is applied.
This can be either \texttt{xyz}, \texttt{zyx} or \texttt{zxz}, where \textbf{\texttt{xyz} is used as default if the parameter is not specified}. Three angles are expected as input, which should always be provided in the order in which they are applied.
\item The \texttt{xyz} option uses extrinsic Euler angles to apply a rotation around the global $X$ axis, followed by a rotation around the global $Y$ axis and finally a rotation around the global $Z$ axis.
\item The \texttt{zyx} option uses the \textbf{extrinsic Z-Y-X convention} for Euler angles, also known as Pitch-Roll-Yaw or 321 convention. The rotation is represented by three angles describing first a rotation of an angle $\phi$ (yaw) about the $Z$ axis, followed by a rotation of an angle $\theta$ (pitch) about the initial $Y$ axis, followed by a third rotation of an angle $\psi$ (roll) about the initial $X$ axis.
\item The \texttt{zxz} uses the \textbf{extrinsic Z-X-Z convention} for Euler angles instead. This option is also known as the 3-1-3 or the "x-convention" and the most widely used definition of Euler angles~\cite{eulerangles}.
It is highly recommended to always explicitly state the orientation mode instead of relying on the default configuration.
\item The \parameter{orientation} to specify the Euler angles in logical order (e.g. first $X$, then $Y$, then $Z$ for the \texttt{xyz} method), interpreted using the method above (or with the \texttt{xyz} method if the \parameter{orientation_mode} is not specified). An example for three Euler angles would be
orientation_mode = "zyx"
orientation = 45deg 10deg 12deg
which describes the rotation of \SI{45}{\degree} around the $Z$ axis, followed by a \SI{10}{\degree} rotation around the initial $Y$ axis, and finally a rotation of \SI{12}{\degree} around the initial $X$ axis.
All supported rotations are extrinsic active rotations, i.e. the vector itself is rotated, not the coordinate system. All angles in configuration files should be specified in the order they will be applied.
\item The \parameter{number_of_pixels} parameter represents a two-dimensional vector with the number of pixels in the active matrix in the column and row direction, respectively.
\item The \parameter{pixel_pitch} is a two-dimensional vector defining the size of a single pixel.
\item The intrinsic resolution of the detector can be specified using the \parameter{resolution} parameter, a two-dimensional vector holding the position resolution for the column and row directions.
\item The \parameter{time_offset} can be used to shift the individual detector time frames of reference to e.g.\ account for time of flight effects between different detector planes by adding a fixed offset.
\item Pixels to be masked in the offline analysis can be placed in a separate file specified by the \parameter{mask_file} parameter explained in detail in Section~\ref{sec:masking}.
\item A region of interest in the given detector can be defined using the \parameter{roi} parameter. More details on this functionality cna be found in Section~\ref{sec:roi}.
An example configuration file describing a setup with one CLICpix2 detector (named \parameter{016_CP_PS} and two Timepix3~\cite{timepix} detectors (\parameter{W0013_D04}and \parameter{W0013_J05}) is the following:
number_of_pixels = 256, 256
orientation = 9deg, 9deg, 0deg
orientation_mode = "xyz"
pixel_pitch = 55um, 55um
position = 0um, 0um, 10mm
type = "Timepix3"
mask_file = "mask_016_CP_PS.conf"
number_of_pixels = 128,128
orientation = -0.02deg, 0.0deg, -0.015deg
orientation_mode = "xyz"
pixel_pitch = 25um, 25um
position = -0.9mm, 0.21mm, 106.0mm
type = "CLICpix2"
number_of_pixels = 256, 256
orientation = -9deg, 9deg, 0deg
orientation_mode = "xyz"
pixel_pitch = 55um, 55um
position = 0um, 0um, 204mm
type = "Timepix3"
\subsection{Masking Pixels Offline}
Mask files can be provided to individual detectors, which allow the specification of pixels ot be masked in the reconstruction.
The following syntax is within the mask file:
\item \command{c COL}: masking all pixels in column \parameter{COL}
\item \command{r ROW}: masking all pixels in row \parameter{ROW}
\item \command{p COL ROW}: masking the single pixel at address \parameter{COL, ROW}
It should be noted that the individual event loader modules have to take care of discarding masked pixels manually, the \corry framework only parses the mask file and attaches the mask information to the respective detector. The event loader modules should thus always query the detector object for masks before adding new pixels to the data collections.
\subsection{Defining a Region of Interest}
The region of interest (ROI) feature of each detector allows marking tracks or clusters to be within a certain region on the respective detector.
This information can be used in analyses to restrict the selection of tracks or clusters to certain regions of the device, e.g.\ to exclude known bad regions from the calculation of efficiencies.
The ROI is defined as a polynomial in local pixel coordinates of the device using the \parameter{roi} keyword. A rectangle could, for example, be defined by providing the four corners of the shape via
roi = [1, 1], [1, 120], [60, 120], [60, 1]
Internally, a winding number algorithm is used to determine whether a certain local position is within or outside the given polynomial shape.
Two functions are provided by the detector API:
// Returns "true" if the track is found to be within the ROI
bool isWithinROI(const Track* track);
// Returns "true" if the cluster as well as all its pixels are found to be within the ROI
bool isWithinROI(const Cluster* cluster);
\chapter{Correlating Different Devices}
\todo{describe how different devices can be correlated}
\section{Triggered Devices}
\todo{if event has no time information, event definition is the trigger}
\section{Triggerless Devices}
\todo{event needs to be defined by the metronome}
\section{Mixing Devices}
\todo{set triggered device first, start/stop information from frame is taken as event (metronome information replacement) and subsequent untriggered devices use this time window to define the event}
\chapter{Extending the \corry Framework}
\section{Writing Additional Modules}
use \command{} script in \dir{etc/}.
\chapter{The \corrybold Framework}
This chapter provides some crucial information about the framework, its executables and the available configuration parameters which are processed on a global level.
\section{The \texttt{corry} Executable}
The \corry executable \texttt{corry} functions as the interface between the user and the framework.
It is primarily used to provide the main configuration file, but also allows to add and overwrite options from the main configuration file.
This is both useful for quick testing as well as for batch processing of many runs to be reconstructed.
The executable handles the following arguments:
\item \texttt{-c <file>}: Specifies the configuration file to be used for the reconstruction, relative to the current directory.
This is the only \textbf{required} argument, the simulation will fail to start if this argument is not given.
\item \texttt{-l <file>}: Specify an additional location such as a file to forward log output to. This is used as additional destination alongside the standard output and the location specified in the framework parameters described in Section~\ref{sec:framework_parameters}.
\item \texttt{-v <level>}: Sets the global log verbosity level, overwriting the value specified in the configuration file described in Section~\ref{sec:framework_parameters}.
Possible values are \texttt{FATAL}, \texttt{STATUS}, \texttt{ERROR}, \texttt{WARNING}, \texttt{INFO} and \texttt{DEBUG}, where all options are case-insensitive.
The module specific logging level introduced in Section~\ref{sec:logging_verbosity} is not overwritten.
\item \texttt{-o <option>}: Passes extra options which are added and overwritten in the main configuration file.
This argument may be specified multiple times, to add multiple options.
Options are specified as key/value pairs in the same syntax as used in the configuration files (refer to Chapter~\ref{ch:configuration_files} for more details), but the key is extended to include a reference to a configuration section in shorthand notation.
There are two types of keys that can be specified:
\item Keys to set \textbf{framework parameters}. These have to be provided in exactly the same way as they would be in the main configuration file (a section does not need to be specified). An example to overwrite the standard output directory would be \command{corry -c <file> -o output_directory="run123456"}.
\item Keys for \textbf{module configurations}. These are specified by adding a dot (\texttt{.}) between the module and the actual key as it would be given in the configuration file (thus \textit{module}.\textit{key}). An example to overwrite the information written by the FileWiter module would be \command{corry -c <file> -o FileWriter.onlyDUT="true"}.
Note that only the single argument directly following the \texttt{-o} is interpreted as the option. If there is whitespace in the key/value pair this should be properly enclosed in quotation marks to ensure the argument is parsed correctly.
No interaction with the framework is possible during the reconstruction. Signals can however be send using keyboard shortcuts to terminate the run, either gracefully or with force. The executable understand the following signals:
\item \texttt{CTRL+C} (SIGINT): Request a graceful shutdown of the reconstruction. This means the currently processed event is finished, while all other events requested in the configuration file are ignored. After finishing the event, the finalization stage is executed for every module to ensure all modules finish properly.
\item \texttt{CTRL+\textbackslash} (SIGQUIT): Forcefully terminates the framework. It is not recommended to use this signal as it will normally lead to the loss of all generated data. This signal should only be used when graceful termination is for any reason not possible.
\section{The Clipboard}
The clipboard is the framework's infrastructure for temporary storing information during the event processing.
Every module can access the clipboard and both read and write information.
Collections or individual elements on the clipboard are accessed via their name, and is internally stored as map.
The clipboard consists of two parts, a temporary storage and a persistent storage space.
\subsection{Temporary Data Storage}
The temporary data storage is only available during the processing of a single event.
It is automatically cleared at the end of the event processing and has to be populated with new data in the new event to be processed.
The temporary storage acts as the main data structure to communicate information between different modules and can hold multiple collections of \corry objects such as pixel hits, clusters or tracks.
\subsection{Persistent Storage}
The persistent storage is not cleared at the end of each event processing and can be used to store information used in multiple events.
Currently this storage only allows for the caching of double-precision floating point numbers.
\section{Global Framework Parameters}
The \corry framework provides a set of global parameters which control and alter its behavior. These parameters are inherited by all modules.
The currently available global parameters are:
\item \parameter{detectors_file}: Location of the file describing the detector configuration described in Section~\ref{sec:detector_config}.
The only \textit{required} global parameter: the framework will fail to start if it is not specified.
\item \parameter{detectors_file_updated}: Location of the file that the (potentially) updated detector configuration should be written into. If this file does not already exist, it will be created. If the same file is given as for \parameter{detectors_file}, the file is overwritten with the updated values.
\item \parameter{histogram_file}: Location of the file where the ROOT output histograms of all modules will be written to. The file extension \texttt{.root} will be appended if not present. Directories within the ROOT file will be created automatically for all modules.
\item \parameter{number_of_events}: Determines the total number of events the framework should process, negative numbers allow processing of all data available.
After reaching the specified number of events, reconstruction is stopped.
Defaults to $-1$.
\item \parameter{number_of_tracks}: Determines the total number of tracks the framework should reconstruct, negative numbers indicate no limit on the number of reconstructed tracks.
After reaching the specified number of events, reconstruction is stopped.
Defaults to $-1$.
\item \parameter{run_time}: Determines the wall-clock time of data acquisition the framework should reconstruct up until. Negative numbers inidcate no limit on the time slice to reconstruct.
Defaults to $-1$.
\item \parameter{log_level}: Specifies the lowest log level which should be reported.
Possible values are \texttt{FATAL}, \texttt{STATUS}, \texttt{ERROR}, \texttt{WARNING}, \texttt{INFO} and \texttt{DEBUG}, where all options are case-insensitive.
Defaults to the \texttt{INFO} level.
More details and information about the log levels, including how to change them for a particular module, can be found in Section~\ref{sec:logging_verbosity}.
Can be overwritten by the \texttt{-v} parameter on the command line (see Section~\ref{sec:executable}).
\item \parameter{log_format}: Determines the log message format to display.
Possible options are \texttt{SHORT}, \texttt{DEFAULT} and \texttt{LONG}, where all options are case-insensitive.
More information can be found in Section~\ref{sec:logging_verbosity}.
\item \parameter{log_file}: File where the log output should be written to in addition to printing to the standard output (usually the terminal).
Another (additional) location to write to can be specified on the command line using the \texttt{-l} parameter (see Section~\ref{sec:executable}).
\item \parameter{library_directories}: Additional directories to search for module libraries, before searching the default paths.
\section{Logging and Verbosity Levels}
\corry is designed to identify mistakes and implementation errors as early as possible and to provide the user with clear indications about the problem.
The amount of feedback can be controlled using different log levels which are inclusive, i.e.\ lower levels also include messages from all higher levels.
The global log level can be set using the global parameter \parameter{log_level}.
The log level can be overridden for a specific module by adding the \parameter{log_level} parameter to the respective configuration section.
The following log levels are supported:
\item \textbf{FATAL}: Indicates a fatal error that will lead to direct termination of the application.
Typically only emitted in the main executable after catching exceptions as they are the preferred way of fatal error handling.
An example of a fatal error is an invalid configuration parameter.
\item \textbf{STATUS}: Important information about the status of the reconstruction.
Is only used for messages which have to be logged in every run such as initial information on the module loading, opened data files and the current progress of the run.
\item \textbf{ERROR}: Severe error that should not occur during a normal well-configured reconstruction.
Frequently leads to a fatal error and can be used to provide extra information that may help in finding the problem (for example used to indicate the reason a dynamic library cannot be loaded).
\item \textbf{WARNING}: Indicate conditions that should not occur normally and possibly lead to unexpected results.
The framework will however continue without problems after a warning.
A warning is for example issued to indicate that a calibration file for a certain detector cannot be found and that the reconstruction is therefore performed with uncalibrated data.
\item \textbf{INFO}: Information messages about the reconstruction process.
Contains summaries of the reconstruction details for every event and for the overall run.
Should typically produce maximum one line of output per event and module.
\item \textbf{DEBUG}: In-depth details about the progress of the reconstruction such as information on every cluster formed or on track fitting results.
Produces large volumes of output per event, and should therefore only be used for debugging the reconstruction.
\item \textbf{TRACE}: Messages to trace what the framework or a module is currently doing.
Unlike the \textbf{DEBUG} level, it does not contain any direct information about the physics but rather indicates which part of the module or framework is currently running.
Mostly used for software debugging or determining performance bottlenecks in the simulations.
It is not recommended to set the \parameter{log_level} higher than \textbf{WARNING} in a typical reconstruction as important messages may be missed.
Setting too low logging levels should also be avoided since printing many log messages will significantly slow down the simulation.
The logging system supports several formats for displaying the log messages.
The following formats are supported via the global parameter \parameter{log_format} or the individual module parameter with the same name:
\item \textbf{SHORT}: Displays the data in a short form.
Includes only the first character of the log level followed by the configuration section header and the message.
\item \textbf{DEFAULT}: The default format.
Displays system time, log level, section header and the message itself.
\item \textbf{LONG}: Detailed logging format.
Displays all of the above but also indicates source code file and line where the log message was produced.
This can help in debugging modules.
\section{Coordinate Systems}
Local coordinate systems for each detector and a global frame of reference for the full setup are defined.
The global coordinate system is chosen as a right-handed Cartesian system, and the rotations of individual devices are performed around the geometrical center of their sensor.
Local coordinate systems for the detectors are also right-handed Cartesian systems, with the x- and y-axes defining the sensor plane.
The origin of this coordinate system is the center of the lower left pixel in the grid, i.e.\ the pixel with indices (0,0).
This simplifies calculations in the local coordinate system as all positions can either be stated in absolute numbers or in fractions of the pixel pitch.
\corry is a
This section describes the currently available Corryvreckan modules in detail.
This chapter describes the currently available Corryvreckan modules in detail.
The description comprises a description of the module implemented as well as possible configuration parameters along with their defaults.
Furthermore, a list of (potentially optional) output plots is provided.
For inquiries about certain modules or their documentation, the respective maintainers should be contacted directly.
\chapter{Using \corry as Online Monitor}
\section{Additional Tools \& Resources}
\chapter{Additional Tools \& Resources}
The following section briefly describes tools provided with the Corryvreckan framework.
% Add command to display allpix squared
% Temporary TODO commands
% \newcommand{\comment}[1]{#1} % DRAFT
\newcommand{\comment}[1]{} % FINAL
......@@ -13,6 +18,11 @@
% Typeset framework parameter and escape underscores:
% Typeset directory and file names
% Define ini format used in the converted Markdown files
......@@ -34,10 +44,11 @@
% Command to add all modules
......@@ -62,11 +73,11 @@
% Create addreferences command (overwritten for HTML in config)
% Command to add the license (overwritten for HTML in config)
\renewcommand{\arraystretch}{1.5}% Spread rows out...
......@@ -2,7 +2,7 @@
\documentclass[fontsize=12pt, parskip=half]{scrartcl}
......@@ -64,7 +64,7 @@
\title{Corryvreckan User Manual} % Title
\title{\corrybold User Manual} % Title
\author{Morag Williams (\href{}{})\\
Daniel Hynds (\href{}{})\\
......@@ -76,37 +76,49 @@
\maketitle % Insert the title, author and date
\maketitle % Insert the title, author and date
% Table Of Contents
% intro
% Time correlation
% Time correlation
% modules
% additional tools and resources