Commit 0eb03d97 authored by Simon Spannagel's avatar Simon Spannagel
Browse files

Merge branch 'rework_configuration' into 'master'

Allow Overwriting of Detector Parameters from Commandline

See merge request !76
parents 3ff1cfb7 1b5f70e3
Pipeline #637068 passed with stages
in 19 minutes and 10 seconds
......@@ -17,16 +17,20 @@ This is the only \textbf{required} argument, the simulation will fail to start i
\item \texttt{-v <level>}: Sets the global log verbosity level, overwriting the value specified in the configuration file described in Section~\ref{sec:framework_parameters}.
Possible values are \texttt{FATAL}, \texttt{STATUS}, \texttt{ERROR}, \texttt{WARNING}, \texttt{INFO} and \texttt{DEBUG}, where all options are case-insensitive.
The module specific logging level introduced in Section~\ref{sec:logging_verbosity} is not overwritten.
\item \texttt{-o <option>}: Passes extra options which are added and overwritten in the main configuration file.
\item \texttt{-o <option>}: Passes extra framework or module options which are added and overwritten in the main configuration file.
This argument may be specified multiple times, to add multiple options.
Options are specified as key/value pairs in the same syntax as used in the configuration files (refer to Chapter~\ref{ch:configuration_files} for more details), but the key is extended to include a reference to a configuration section in shorthand notation.
There are two types of keys that can be specified:
Options are specified as key/value pairs in the same syntax as used in the configuration files described in Chapter~\ref{ch:configuration_files}, but the key is extended to include a reference to a configuration section or instantiation in shorthand notation.
There are three types of keys that can be specified:
\begin{itemize}
\item Keys to set \textbf{framework parameters}. These have to be provided in exactly the same way as they would be in the main configuration file (a section does not need to be specified). An example to overwrite the standard output directory would be \command{corry -c <file> -o output_directory="run123456"}.
\item Keys for \textbf{module configurations}. These are specified by adding a dot (\texttt{.}) between the module and the actual key as it would be given in the configuration file (thus \textit{module}.\textit{key}). An example to overwrite the information written by the FileWiter module would be \command{corry -c <file> -o FileWriter.onlyDUT="true"}.
\item Keys to set \textbf{framework parameters}. These have to be provided in exactly the same way as they would be in the main configuration file (a section does not need to be specified). An example to overwrite the standard output directory would be \texttt{corry -c <file> -o output\_directory="run123456"}.
\item Keys for \textbf{module configurations}. These are specified by adding a dot (\texttt{.}) between the module and the actual key as it would be given in the configuration file (thus \textit{module}.\textit{key}). An example to overwrite the deposited particle to a positron would be \texttt{corry -c <file> -o Tracking4D.timingCut="100ns"}.
\item Keys to specify values for a particular \textbf{module instantiation}. The identifier of the instantiation and the name of the actual key are split by a dot (\texttt{.}), in the same way as for keys for module configurations (thus \textit{identifier}.\textit{key}). The unique identifier for a module can contains one or more colons (\texttt{:}) to distinguish between various instantiations of the same module. The exact name of an identifier depends on the name of the detector and the optional input and output name. Those identifiers can be extracted from the logging section headers. An example to change the temperature of propagation for a particular instantiation for a detector named \textit{dut} could be \texttt{corry -c <file> -o Clustering4D:dut.timingCut="1us"}.
\end{itemize}
Note that only the single argument directly following the \texttt{-o} is interpreted as the option. If there is whitespace in the key/value pair this should be properly enclosed in quotation marks to ensure the argument is parsed correctly.
\item \texttt{-g <option>}: Passes extra detector options which are added and overwritten in the detector configuration file.
This argument can be specified multiple times, to add multiple options.
The options are parsed in the same way as described above for module options, but only one type of key can be specified to overwrite an option for a single detector.
These are specified by adding a dot (\texttt{.}) between the detector and the actual key as it would be given in the detector configuration file (thus \textit{detector}.\textit{key}). An example to change the orientation of a particular detector named \texttt{detector1} would be \texttt{corry -c <file> -g detector1.orientation=0deg,0deg,45deg}.
\end{itemize}
No interaction with the framework is possible during the reconstruction. Signals can however be send using keyboard shortcuts to terminate the run, either gracefully or with force. The executable understand the following signals:
......
......@@ -12,6 +12,7 @@ ADD_LIBRARY(CorryvreckanCore SHARED
config/ConfigReader.cpp
config/Configuration.cpp
config/exceptions.cpp
config/OptionParser.cpp
module/Module.cpp
module/ModuleManager.cpp
)
......
/**
* @file
* @brief Implementation of config manager
* @copyright Copyright (c) 2017 CERN and the Allpix Squared authors.
* @copyright Copyright (c) 2017 CERN and the Corryvreckan authors.
* This software is distributed under the terms of the MIT License, copied verbatim in the file "LICENSE.md".
* In applying this license, CERN does not waive the privileges and immunities granted to it by virtue of its status as an
* Intergovernmental Organization or submit itself to any jurisdiction.
......@@ -24,115 +24,162 @@ using namespace corryvreckan;
/**
* @throws ConfigFileUnavailableError If the main configuration file cannot be accessed
*/
ConfigManager::ConfigManager(std::string file_name) : file_name_(std::move(file_name)) {
LOG(TRACE) << "Using " << file_name_ << " as main configuration file";
ConfigManager::ConfigManager(std::string file_name,
std::initializer_list<std::string> global,
std::initializer_list<std::string> ignore) {
// Check if the file exists
std::ifstream file(file_name_);
std::ifstream file(file_name);
if(!file) {
throw ConfigFileUnavailableError(file_name_);
throw ConfigFileUnavailableError(file_name);
}
// Convert main file to absolute path
file_name_ = corryvreckan::get_absolute_path(file_name_);
// Initialize global base configuration with absolute file name
global_base_config_ = Configuration("", file_name_);
file_name = corryvreckan::get_canonical_path(file_name);
LOG(TRACE) << "Reading main configuration";
// Read the file
reader_.add(file, file_name_);
ConfigReader reader(file, file_name);
// Convert all global and ignored names to lower case and store them
auto lowercase = [](const std::string& in) {
std::string out(in);
std::transform(out.begin(), out.end(), out.begin(), ::tolower);
return out;
};
std::transform(global.begin(), global.end(), std::inserter(global_names_, global_names_.end()), lowercase);
std::transform(ignore.begin(), ignore.end(), std::inserter(ignore_names_, ignore_names_.end()), lowercase);
// Initialize global base configuration
global_config_ = reader.getHeaderConfiguration();
// Store all the configurations read
for(auto& config : reader.getConfigurations()) {
// Skip all ignored sections
std::string config_name = config.getName();
std::transform(config_name.begin(), config_name.end(), config_name.begin(), ::tolower);
if(ignore_names_.find(config_name) != ignore_names_.end()) {
continue;
}
// Merge all global section with the global config
if(global_names_.find(config_name) != global_names_.end()) {
global_config_.merge(config);
continue;
}
module_configs_.push_back(config);
}
// Reading detector file
auto detector_file_names = global_config_.getPathArray("detectors_file", true);
LOG(TRACE) << "Reading detector configurations";
for(auto& detector_file_name : detector_file_names) {
std::ifstream detector_file(detector_file_name);
ConfigReader detector_reader(detector_file, detector_file_name);
auto detector_configs = detector_reader.getConfigurations();
detector_configs_.splice(detector_configs_.end(),
std::list<Configuration>(detector_configs.begin(), detector_configs.end()));
}
}
/**
* @warning Only one header can be added in this way to define its main name
* The global configuration is the combination of all sections with a global header.
*/
void ConfigManager::setGlobalHeaderName(std::string name) {
global_default_name_ = name;
std::transform(name.begin(), name.end(), name.begin(), ::tolower);
global_names_.emplace(std::move(name));
}
void ConfigManager::addGlobalHeaderName(std::string name) {
std::transform(name.begin(), name.end(), name.begin(), ::tolower);
global_names_.emplace(std::move(name));
Configuration& ConfigManager::getGlobalConfiguration() {
return global_config_;
}
/**
* The global configuration is the combination of all sections with a global header.
* Load all extra options that should be added on top of the configuration in the file. The options loaded here are
* automatically applied to the module instance when these are added later.
*/
Configuration ConfigManager::getGlobalConfiguration() {
// Copy base config and set name
Configuration global_config = global_base_config_;
global_config.setName(global_default_name_);
// Add all other global configuration
for(auto& global_name : global_names_) {
auto configs = reader_.getConfigurations(global_name);
for(auto& config : configs) {
global_config.merge(config);
bool ConfigManager::loadModuleOptions(const std::vector<std::string>& options) {
bool optionsApplied = false;
// Parse the options
for(auto& option : options) {
module_option_parser_.parseOption(option);
}
// Apply global options
optionsApplied = module_option_parser_.applyGlobalOptions(global_config_) || optionsApplied;
// Apply module options
for(auto& config : module_configs_) {
optionsApplied = module_option_parser_.applyOptions(config.getName(), config) || optionsApplied;
}
return global_config;
}
void ConfigManager::addIgnoreHeaderName(std::string name) {
std::transform(name.begin(), name.end(), name.begin(), ::tolower);
ignore_names_.emplace(std::move(name));
return optionsApplied;
}
/**
* Option is split in a key / value pair, an error is thrown if that is not possible. When the key contains at least one dot
* it is interpreted as a relative configuration with the module identified by the first dot. In that case the option is
* applied during module loading when either the unique or the configuration name match. Otherwise the key is interpreted as
* global key and is added to the global header.
* Load all extra options that should be added on top of the detector configuration in the file. The options loaded here are
* automatically applied to the detector instance when these are added later and will be taken into account when possibly
* loading customized detector models.
*/
void ConfigManager::parseOption(std::string line) {
line = corryvreckan::trim(line);
auto key_value = ConfigReader::parseKeyValue(line);
auto key = key_value.first;
auto value = key_value.second;
auto dot_pos = key.find('.');
if(dot_pos == std::string::npos) {
// Global option, add to the global base config
global_base_config_.setText(key, value);
} else {
// Other identifier bound option is passed
auto identifier = key.substr(0, dot_pos);
key = key.substr(dot_pos + 1);
identifier_options_[identifier].push_back(std::make_pair(key, value));
}
}
bool ConfigManager::loadDetectorOptions(const std::vector<std::string>& options) {
bool optionsApplied = false;
bool ConfigManager::applyOptions(const std::string& identifier, Configuration& config) {
if(identifier_options_.find(identifier) == identifier_options_.end()) {
return false;
// Create the parser
OptionParser detector_option_parser;
// Parse the options
for(auto& option : options) {
detector_option_parser.parseOption(option);
}
for(auto& key_value : identifier_options_[identifier]) {
config.setText(key_value.first, key_value.second);
// Apply detector options
for(auto& config : detector_configs_) {
optionsApplied = detector_option_parser.applyOptions(config.getName(), config) || optionsApplied;
}
return true;
return optionsApplied;
}
bool ConfigManager::hasConfiguration(const std::string& name) {
return reader_.hasConfiguration(name);
/**
* All special global and ignored sections are not included in the list of module configurations.
*/
std::list<Configuration>& ConfigManager::getModuleConfigurations() {
return module_configs_;
}
/**
* The list of detector configurations is read from the configuration defined in 'detector_file'
*/
std::list<Configuration>& ConfigManager::getDetectorConfigurations() {
return detector_configs_;
}
/**
* All special global and ignored sections are removed before returning the rest of the configurations. The list of normal
* sections is used by the ModuleManager to instantiate all the required modules.
* @warning A previously stored configuration is directly invalidated if the same unique name is used again
*
* An instance configuration is a specialized configuration for a particular module instance. If an unique name already
* exists the previous record is deleted and a new configuration record corresponding to the replaced instance is added.
*/
std::vector<Configuration> ConfigManager::getConfigurations() const {
std::vector<Configuration> result;
for(auto& config : reader_.getConfigurations()) {
// ignore all global and ignores names
std::string config_name = config.getName();
std::transform(config_name.begin(), config_name.end(), config_name.begin(), ::tolower);
if(global_names_.find(config_name) != global_names_.end() ||
ignore_names_.find(config_name) != ignore_names_.end()) {
continue;
}
result.push_back(config);
Configuration& ConfigManager::addInstanceConfiguration(const ModuleIdentifier& identifier, const Configuration& config) {
std::string unique_name = identifier.getUniqueName();
// Check uniqueness
if(instance_name_to_config_.find(unique_name) != instance_name_to_config_.end()) {
instance_configs_.erase(instance_name_to_config_[unique_name]);
}
return result;
// Add configuration
instance_configs_.push_back(config);
Configuration& ret_config = instance_configs_.back();
instance_name_to_config_[unique_name] = --instance_configs_.end();
// Add identifier key to config
ret_config.set<std::string>("identifier", identifier.getIdentifier());
// Apply instance options
module_option_parser_.applyOptions(unique_name, ret_config);
return ret_config;
}
/**
* The list of instance configurations can contain configurations with duplicate names, but the instance configuration is
* guaranteed to have a configuration value 'identifier' that contains an unique identifier for every same config name
*/
std::list<Configuration>& ConfigManager::getInstanceConfigurations() {
return instance_configs_;
}
/**
* @file
* @brief Interface to the main configuration and its normal and special sections
* @copyright Copyright (c) 2017 CERN and the Allpix Squared authors.
* @copyright Copyright (c) 2017 CERN and the Corryvreckan authors.
* This software is distributed under the terms of the MIT License, copied verbatim in the file "LICENSE.md".
* In applying this license, CERN does not waive the privileges and immunities granted to it by virtue of its status as an
* Intergovernmental Organization or submit itself to any jurisdiction.
......@@ -16,6 +16,8 @@
#include "ConfigReader.hpp"
#include "Configuration.hpp"
#include "OptionParser.hpp"
#include "core/module/ModuleIdentifier.hpp"
namespace corryvreckan {
......@@ -35,8 +37,12 @@ namespace corryvreckan {
/**
* @brief Construct the configuration manager
* @param file_name Path to the main configuration file
* @param global List of sections representing the global configuration (excluding the empty header section)
* @param ignore List of sections that should be ignored
*/
explicit ConfigManager(std::string file_name);
explicit ConfigManager(std::string file_name,
std::initializer_list<std::string> global = {},
std::initializer_list<std::string> ignore = {"Ignore"});
/**
* @brief Use default destructor
*/
......@@ -59,70 +65,63 @@ namespace corryvreckan {
/// @}
/**
* @brief Set the name of the global header and add to the global names
* @param name Name of a global header that should be used as the name
*/
// TODO [doc] Should only set the name and do not add it
void setGlobalHeaderName(std::string name);
/**
* @brief Add a global header name
* @param name Name of a global header section
* @brief Get the global configuration
* @return Reference to global configuration
*/
// TODO [doc] Rename to addGlobalHeader
void addGlobalHeaderName(std::string name);
Configuration& getGlobalConfiguration();
/**
* @brief Get the global configuration
* @return Global configuration
* @brief Get all the module configurations
* @return Reference to list of module configurations
*/
Configuration getGlobalConfiguration();
std::list<Configuration>& getModuleConfigurations();
/**
* @brief Add a header name to fully ignore
* @param name Name of a header to ignore
* @brief Add a new module instance configuration and applies instance options
* @param identifier Identifier for this module instance
* @param config Instance configuration to store
* @return Reference to stored instance configuration
*/
// TODO [doc] Rename to ignoreHeader
void addIgnoreHeaderName(std::string name);
Configuration& addInstanceConfiguration(const ModuleIdentifier& identifier, const Configuration& config);
/**
* @brief Parse an extra configuration option
* @param line Line with the option
* @brief Get all the instance configurations
* @return Reference to list of instance configurations
*/
void parseOption(std::string line);
std::list<Configuration>& getInstanceConfigurations();
/**
* @brief Apply all relevant options to the passed configuration
* @param identifier Identifier to select the options to apply
* @param config Configuration option where the options should be applied to
* @return True if the configuration was changed because of applied options
* @brief Load module options and directly apply them to the global configuration and the module configurations
* @param options List of options to load and apply
* @return True if any actual options where applied
*
* @note Instance configuration options are applied in \ref ConfigManager::addInstanceConfiguration instead
*/
bool applyOptions(const std::string& identifier, Configuration& config);
bool loadModuleOptions(const std::vector<std::string>& options);
/**
* @brief Return if section with given name exists
* @param name Name of the section
* @return True if at least one section with that name exists, false otherwise
* @brief Get all the detector configurations
* @return Reference to list of detector configurations
*/
bool hasConfiguration(const std::string&);
std::list<Configuration>& getDetectorConfigurations();
/**
* @brief Get all configurations that are not global or ignored
* @return List of all normal configurations
* @brief Load detector specific options
* @param options List of options to load and apply
* @return True if any actual options where applied
*/
std::vector<Configuration> getConfigurations() const;
bool loadDetectorOptions(const std::vector<std::string>& options);
private:
std::string file_name_;
std::set<std::string> global_names_{};
std::set<std::string> ignore_names_{};
ConfigReader reader_;
OptionParser module_option_parser_;
Configuration global_base_config_;
std::string global_default_name_{};
std::set<std::string> global_names_{};
std::list<Configuration> module_configs_;
Configuration global_config_;
std::set<std::string> ignore_names_{};
std::list<Configuration> detector_configs_;
std::map<std::string, std::vector<std::pair<std::string, std::string>>> identifier_options_{};
std::list<Configuration> instance_configs_;
std::map<std::string, std::list<Configuration>::iterator> instance_name_to_config_;
};
} // namespace corryvreckan
......
/**
* @file
* @brief Implementation of config reader
* @copyright Copyright (c) 2017 CERN and the Allpix Squared authors.
* @copyright Copyright (c) 2017 CERN and the Corryvreckan authors.
* This software is distributed under the terms of the MIT License, copied verbatim in the file "LICENSE.md".
* In applying this license, CERN does not waive the privileges and immunities granted to it by virtue of its status as an
* Intergovernmental Organization or submit itself to any jurisdiction.
......@@ -101,7 +101,7 @@ void ConfigReader::add(std::istream& stream, std::string file_name) {
// Convert file name to absolute path (if given)
if(!file_name.empty()) {
file_name = corryvreckan::get_absolute_path(file_name);
file_name = corryvreckan::get_canonical_path(file_name);
}
// Build first empty configuration
......@@ -127,7 +127,16 @@ void ConfigReader::add(std::istream& stream, std::string file_name) {
}
// Check if section header or key-value pair
if(line.front() == '[' && line[line.length() - 1] == ']') {
if(line.front() == '[') {
// Line should be a section header with an alphanumeric name
size_t idx = 1;
for(; idx < line.length() - 1; ++idx) {
if(isalnum(line[idx]) == 0 && line[idx] != '_') {
break;
}
}
std::string remain = corryvreckan::trim(line.substr(idx + 1));
if(line[idx] == ']' && (remain.empty() || remain.front() == '#')) {
// Ignore empty sections if they contain no configurations
if(!conf.getName().empty() || conf.countSettings() > 0) {
// Add previous section
......@@ -135,8 +144,12 @@ void ConfigReader::add(std::istream& stream, std::string file_name) {
}
// Begin new section
section_name = std::string(line, 1, line.length() - 2);
section_name = std::string(line, 1, idx - 1);
conf = Configuration(section_name, file_name);
} else {
// Section header is not valid
throw ConfigParseError(file_name, line_num);
}
} else if(isalpha(line.front()) != 0) {
// Line should be a key / value pair with an equal sign
try {
......
/**
* @file
* @brief Provides a reader for configuration files
* @copyright Copyright (c) 2017 CERN and the Allpix Squared authors.
* @copyright Copyright (c) 2017 CERN and the Corryvreckan authors.
* This software is distributed under the terms of the MIT License, copied verbatim in the file "LICENSE.md".
* In applying this license, CERN does not waive the privileges and immunities granted to it by virtue of its status as an
* Intergovernmental Organization or submit itself to any jurisdiction.
......
/**
* @file
* @brief Implementation of configuration
* @copyright Copyright (c) 2017 CERN and the Allpix Squared authors.
* @copyright Copyright (c) 2017 CERN and the Corryvreckan authors.
* This software is distributed under the terms of the MIT License, copied verbatim in the file "LICENSE.md".
* In applying this license, CERN does not waive the privileges and immunities granted to it by virtue of its status as an
* Intergovernmental Organization or submit itself to any jurisdiction.
......@@ -25,12 +25,23 @@ bool Configuration::has(const std::string& key) const {
return config_.find(key) != config_.cend();
}
unsigned int Configuration::count(std::initializer_list<std::string> keys) const {
if(keys.size() == 0) {
throw std::invalid_argument("list of keys cannot be empty");
}
unsigned int found = 0;
for(auto& key : keys) {
if(has(key)) {
found++;
}
}
return found;
}
std::string Configuration::getName() const {
return name_;
}
void Configuration::setName(const std::string& name) {
name_ = name;
}
std::string Configuration::getFilePath() const {
return path_;
}
......@@ -93,12 +104,12 @@ std::string Configuration::path_to_absolute(std::string path, bool canonicalize_
// Set new path
path = directory + "/" + path;
}
// Normalize path only if we have to check if it exists
// NOTE: This throws an error if the path does not exist
if(canonicalize_path) {
path = corryvreckan::get_absolute_path(path);
}
path = corryvreckan::get_canonical_path(path);
}
return path;
}
......
......@@ -62,6 +62,13 @@ namespace corryvreckan {
*/
bool has(const std::string& key) const;
/**
* @brief Check how many of the given keys are defined
* @param keys Keys to check for existence
* @return number of existing keys from the given list
*/
unsigned int count(std::initializer_list<std::string> keys) const;
/**
* @brief Get value of a key in requested type
* @param key Key to get value of
......@@ -143,19 +150,19 @@ namespace corryvreckan {
std::vector<std::string> getPathArray(const std::string& key, bool check_exists = false) const;
/**
* @brief Set value for a key in a given type with units
* @brief Set value for a key in a given type
* @param key Key to set value of
* @param val Value to assign to the key
* @param units List of possible output units
*/
template <typename T> void set(const std::string& key, const T& val, std::initializer_list<std::string> units);
template <typename T> void set(const std::string& key, const T& val);
/**
* @brief Set value for a key in a given type
* @brief Store value for a key in a given type, including units
* @param key Key to set value of
* @param val Value to assign to the key
* @param units List of possible output units
*/
template <typename T> void set(const std::string& key, const T& val);
template <typename T> void set(const std::string& key, const T& val, std::initializer_list<std::string> units);
/**
* @brief Set list of values for a key in a given type
......@@ -211,11 +218,6 @@ namespace corryvreckan {
*/
std::string getName() const;
/**
* @brief Set name of the configuration header
*/
void setName(const std::string& name);
/**
* @brief Get path to the file containing the configuration if it has one
* @return Absolute path to configuration file or empty if not linked to a file
......
......@@ -91,7 +91,7 @@ namespace corryvreckan {
auto node = parse_value(str);
for(auto& child : node->children) {
if(child->children.empty()) {
throw std::invalid_argument("matrix has less than two dimensions");
throw std::invalid_argument("matrix has less than two dimensions, enclosing brackets might be missing");
}
std::vector<T> array;
......
/**
* @file
* @brief Implementation of option parser