SW ROD

This is the implementation of SW ROD readout application of the ATLAS TDAQ system that is based on the Phase-I SW ROD Architectural Analysis and Design document, which addresses the requirements provided by detector community, which are described in the Phase-I SW ROD User Requirements document. The package is distributed as part of the ATLAS TDAQ software release. Custom SW ROD components development shall follow the TDAQ software development procedure.

Introduction

SW ROD is envisaged to act in the ATLAS data flow chain as the data-handling interface between the FELIX readout system and the ATLAS High-Level Trigger (HLT). SW ROD implements ROB fragment building and formatting, which in the Run-1/2 systems were done by the detector specific Readout Driver (ROD) components. SW ROD also furnishes the same buffering and HLT request processing capabilities as provided by the Readout System (ROS) of the legacy DAQ.

SW ROD Application

SW ROD functionality is provided by multiple homogeneous SW processes running on a cluster of commodity computers. All processes originate from the same binary executable, called swrod_application, but they diverge by using different configuration parameters such as a set of FELIX E-Links for receiving data, data processing algorithms, HLT request processing parameters and so on.

The SW ROD application is fully integrated with the TDAQ online infrastructure. It implements the Run Control Finite State Machine (FSM) and is configured using the standard TDAQ Configuration service. It also implements the Event Monitoring Sampler interface and publishes its operational statistics to the TDAQ Information Service.

Integrating Custom Processing into the SW ROD Application

The SW ROD Application provides a framework for executing detector specific code that is required for:

• Extracting TTC information (L1ID and BCID) from input data packets.
• Verifying integrity of input data packets.
• Applying custom detector-specific processing to fully aggregated ROB fragments.

The corresponding procedures must be implemented by a custom detector specific shared library, which should provide three functions with well-defined signatures as described in the following sections. Such a library is advertised to the SW ROD Application via an object of the SwRodCustomProcessingLib OKS configuration class that must declare the names of the custom functions. This object must be linked with each data channel, aka ROD, that is represented by an instance of the SwRodDataChannel OKS class.

NOTE: Custom functions must be declared using extern "C" specifier that guarantees that the functions run-time symbolic references will be the same as the function names.

TTC Information Extraction Function

A function that implements TTC information extraction is mandatory and should be implemented along the lines demonstrated by the following example:

extern "C"
std::tuple<uint32_t, uint32_t, uint16_t> testTriggerInfoExtractor(const uint8_t * data, uint32_t size) {
  if (size < 2) {
    throw swrod::Exception(ERS_HERE, "Data packet is too short");
  }
  uint8_t type = data[0]>>6;
  if (type == 1) {
    return std::tuple(
        *(data+1),     // L1ID
        0xff,          // L1ID width is 8 bits
        0xffff);       // no BCID
  } else if (type == 2) {
    if (size < 4) {
      throw swrod::Exception(ERS_HERE, "Data packet is too short");
    }
    return std::tuple(
        *reinterpret_cast<uint16_t*>(data+2)),  // L1ID
        0xffff,                                 // L1ID width is 16 bits
        *reinterpret_cast<uint16_t*>(data)>>4); // BCID
  } else {
    throw swrod::Exception(ERS_HERE, "Data packet has invalid type");
  }
}

A TTC information extraction function has two parameters: a pointer to the memory block that contains data packet to be processed and the size of this data packet in bytes. The first parameter points to the beginning of the data packet payload meaning that any meta-data, like for example FELIX header or communication protocol header, are intentionally omitted. This function must return a 3-tuple with the following values extracted from the given data packet:

• l1id: 32-bit value that contains all available bits of the extended L1ID.
• l1id_mask: 32-bit mask that defines how many bits of extended L1ID are provided by the first value. The bits, which are present in the L1ID value must be set to 1 in this mask, while all other bits must be set to 0. For example if the data packet contains only the 8 least significant bits of the extended L1ID then the mask value must be set to 0xFF. If a full extended L1ID is present in the given data packet, then the mask value must be set to 0xFFFFFFFF.
• bcid: 16-bit BCID value from the given data packet. If no BCID is present in the data packet the value must be set to 0xFFFF.

This function may also throw swrod::Exception if TTC information cannot be reliably extracted from the given data packet. Any implementation of the SW ROD fragment builder algorithm must handle such an exception gracefully. A custom TTC information extraction function may use existing swrod::Exception class that is declared in the swrod/exceptions.h file or it can declare a custom exception class that in this case must inherit from swrod::Exception.

Data Integrity Check Function

Optionally, a custom plugin library can provide a function to be used to check integrity of a given data packet. If data packet format contains a CRC value that was calculated using a known algorithm, then such a function can apply the same algorithm to this packet and compare its result with that value. The following pseudo-code illustrates this idea:

extern "C"
std::optional<bool> testDataIntegrityChecker(const uint8_t * data, uint32_t size) {
  if (not contains_checksum_value(packet)) {
    return std::nullopt; // packet has no checksum
  }
  else {
    return (calculate_checksum(packet) == get_checksum_value(packet));
  }
}

It is up to the fragment building algorithm implementation how to use this function. For performance reasons the existing algorithm implementations call this function only in case of TTC information mismatch between data and L1A packets.

Custom Processing

The last function that may be provided by a custom plugin library is also optional and should be implemented only if a detector specific processing must be applied to the ROB fragments produced by the fragment building algorithm. In this case such a function has to implement a factory for detector-specific class that implements the swrod::CustomProcessor interface. The following code demomnstrates a possible implementation of this function:

extern "C"
swrod::CustomProcessor * createTestCustomProcessor(const boost::property_tree::ptree & config) {
    return new TestCustomProcessor(config.get<int64_t>("CustomLongIntAttribute"));
}

Note: The config object passed to this function contains configuration parameters declared in the corresponding instance of the SwRodDataChannel OKS class. It is possible to add an arbitrary number of custom parameters to the config object by creating a new OKS class that inherits from the SwRodDataChannel and declares these parameters as its attributes. To retrieve the value of a custom attribute one should use the boost::property_tree::ptree::get() function specifying the corresponding attribute type as template parameter and the attribute name as the function argument.

A custom class that inherits the from the swrod::CustomProcessor interface must implement the processROBFragment(swrod::ROBFragment & ) pure virtual function. For scalability, SW ROD may create multiple instances of the swrod::CustomProcessor class as defined by the WorkersNumber attribute of the corresponding SwRodCustomProcessor configuration object. Each instance runs in a dedicated thread, meaning that an implementation of the SwRodCustomProcessor interface does not need to handle thread safety unless it accesses global resources. It is guaranteed that each object of the custom processing class will be used by exactly one thread, so there is no need to protect access to the object's member variables.

Note: To activate custom processing an instance of the SwRodCustomProcessor class must be created in the SWROD configuration and linked to the SwRodRob objects via the Consumers relationship.

The processROBFragment(swrod::ROBFragment & ) function is called for every ROB fragment produced by the fragment building algorithm of the corresponding ROB. The ROB fragment is passed to this function as a reference to an instance of the swrod::ROBFragment class. This class declares several constant attributes. that describe the read-only properties of the fragment. The other, non-read-only attributes can be freely modified by the custom processing procedure:

• m_detector_type: 32-bit value that will be set to the Detector Event Type field of the ROD header.
• m_rod_minor_version: 16-bit value that will be set to the lower 2 bytes of the Format Version Number field of the ROD header.
• m_status_front: if set to true the ROD fragment status words will be placed right after the ROD header, otherwise they will be put at the end of the ROD fragment. Default value is true.
• m_status_words: an originally empty vector of status words. Custom processor may add any number of 32-bit words to this vector. All these words will be added to the ROD fragment status section.
• m_data: this is a vector containing the fragment payload, which may be split into multiple memory blocks. Normally, the number of memory blocks equals the number of data-receiving threads, but this is not always guaranteed, as it may depend on the specific implementation of the fragment-building algorithm Each memory block is managed by an instance of the swrod::ROBFragment::DataBlock class, which provides several functions for accessing and modifying the data within the block. The format of the data in a memory block depends on the fragment-building algorithm used to generate the ROB fragment. A detailed description of the data format for fragments produced by existing SW ROD algorithms is provided in the following sections.

ROB Fragment Memory Management

For performance reasons, fragment-building algorithms use pre-allocated memory blocks of fixed size to store ROB fragments they produce. The size of these blocks is determined using the SwRodFragmentBuilder::MaxMessageSize and MemoryPool::PageSize configuration parameters. Different algorithms may calculate this size in different ways, as explained in the following sections. This approach minimizes the memory footprint of SW ROD applications. The goal is to set the memory block size to the smallest value sufficient to accommodate the majority (e.g., ~99%) of the ROB fragments produced by the corresponding detector component. Rare, larger fragments will be split across multiple memory blocks.

The memory pool can be configured using the the MemoryPool configuration class. Each SwRodApplication object must be linked to an instance of the MemoryPool, which defines default configuration for all ROBs handled by the given SW ROD application. This configuration can be overridden for a specific ROB by linking another instance of the MemoryPool with the corresponding SwRodRob object. The MemoryPool class has two parameters:

• PageSize: This parameter defines the default size of individual memory pages allocated by this memory pool. Note that this value can be overridden by fragment builder implementation, in which case this must be explained in the algorithm's description.
• NumberOfPages: This parameter defines the number of pages that will be pre-allocated before a new run is started. The maximum number of pages that can be allocated by the memory pool is unlimited.

Custom processing function may need to reformat ROB fragment's payload in a way that would require to increase its size. In such a case it is strongly recommended to adjust the SwRodFragmentBuilder::MaxMessageSize and MemoryPool::PageSize parameters of the SW ROD configuration such that the memory blocks allocated by the algorithm will have enough spare space to accommodate the extra amount of data (at least for the bulk of the fragments). If modifying the configuration is not possible, then the custom processing function may allocate a new contiguous memory block of sufficient size using either the standard C++ new operator or a custom memory management routine and copy the reformatted data into the new block. The following example demonstrates this process.

class TestCustomProcessor: public swrod::CustomProcessor {
public:
    explicit TestCustomProcessor(uint64_t tag) {
        m_tag = tag;
    }
    
    void processROBFragment(swrod::ROBFragment & fragment) override {
        fragment.m_status_words.push_back(0x0a);
        fragment.m_status_words.push_back(0x0b);
        fragment.m_status_words.push_back(0x0c);
        fragment.m_rod_minor_version = 15;
        fragment.m_detector_type = 0x222;
        fragment.m_status_front = false;

        for (auto & block : fragment.m_data) {
            swrod::GBTChunk::Header header{0, 0, 0, 0xffffffff};
            if (!block.append(header, &m_tag, sizeof(m_tag))) {
                // The memory block is not large enough to accommodate extra data
                // Allocate a new memory block of sufficient size, copy the
                // original data there and add the custom tag at the end

                // The size is in 4-byte words
                uint32_t size = block.dataSize() +
                        ((sizeof(m_tag) + sizeof(swrod::GBTChunk::Header) + 3)>>2);
                uint32_t * data = new uint32_t[size];
                memcpy(data, block.dataBegin(), block.dataSize()<<2);

                block = swrod::ROBFragment::DataBlock(data, size, block.dataSize());
                block.append(header, &m_tag, sizeof(m_tag));
            }
        }
    }
private:
    uint64_t m_tag;
};

A custom processing function may also modify the size of the m_data vector by either adding new memory blocks or removing obsolete ones. For example, if m_data vector contains multiple memory blocks that need to be completely restructured, it may be more efficient to allocate a single memory block of sufficient size and copy the contents of the original memory blocks into the new one after applying the necessary transformations. Finally, the original memory blocks must be replaced with the new block in the m_data vector.

Using Custom Plugin Test application

The swrod package provides an application for validating and profiling custom plugins. This application, called swrod_custom_plugin_test, can be used as follows:

• On the first invocation, the following parameters must be specified:

  • The name of the data file to be used as the data source.
  • The name of the shared library that implements the plugin to be tested.
  • The names of the three custom functions implemented by this plugin. Optionally, the -o option can be used to save the new configuration to a specified JSON file.

• If a JSON configuration file has been generated, the test application can be started using this file as the sole input parameter by specifying it with the -i command-line option.

Fragment Building Algorithms

A SW ROD fragment-building algorithm is designed to collect data packets that share the same unique identifier — such as the L1 Trigger ID (L1ID) — from a given set of input links and combine them with the corresponding TTC information into an instance of the swrod::ROBFragment class.

This class provides a serialize() function, which can be used to convert a swrod::ROBFragment object into a series of contiguous memory chunks. These chunks store the content of the ROB fragment, formatted according to the ATLAS Event Format specification. The structure of an ATLAS-compliant ROB fragment is shown in the following table.

The contents of the ROB and ROD headers and trailer are defined by the ATLAS Event Format specification and are independent of the fragment-building algorithm implementation. The ROD Data section of the fragment consists of aggregated data chunks received from the detector Front-End electronics, which use a detector-specific data format internally. The method used to aggregate these data chunks depends on the algorithm implementation.

The swrod package provides two fragment building algorithms that can be used to handle data received from FELIX:

• GBT mode algorithm: This algorithm aggregates data chunks from all E-links associated with a given SwRodDataChannel object into a single ROD Data block, aligning them using their L1IDs. It takes at most one data chunk from a given E-link for each ROB fragment. Receiving more than one data chunk in a row with the same L1ID from the same E-link is considered an error. Additionally, the algorithm expects each individual data chunk to have a size that is a multiple of 4 bytes. If this condition is not met, padding zeros are added at the end of the chunk to ensure 4-byte alignment.
• FULL mode algorithm: This algorithm treats each data chunk from any E-link associated with the given SwRodDataChannel as a fully built ROD Data block, which may optionally include ROD Header and ROD Trailer elements. Like the GBT mode, it expects each incoming data chunk to be a multiple of 4 bytes in size. If this condition is not met, padding zeros are added at the end of the chunk to ensure 4-byte alignment.

Both the GBT and FULL mode algorithms can operate in either TTC-driven or data-driven mode. In data-driven mode, no Level-1 Accept (L1A) packets are received. The mode of operation is determined by the state of the L1AHandler relationships in the SwRodConfiguration and SwRodModule configuration objects. If both relationships are empty, the event builders do not subscribe to L1A messages and run in data-driven mode. Otherwise, they subscribe to the TTC E-Link to receive L1 Accept messages and use them for fragment building.

Additional fragment-building algorithms can be implemented and integrated into SW ROD via its plugin mechanism.

SwRodFragmentBuilder configuration class

This is an abstract class that defines several parameters that are common for any fragment building algorithm implementation. These parameters are:

• BuildersNumber: The number of dedicated threads, which are notified when a new ROB fragment data payload is ready and take care of creating a new instance of the swrod::ROBFragment class for this data and passing it to the ROB Fragment Consumers. These threads are used to disentangle fragment builders from fragment consumers and reduce a possible impact of slow consumers on fragment building performance. If BuildersNumber number is set to zero, then no building threads are used, in which case one of the fragment builder's working threads will execute the above procedure.
• DropCorruptedPackets: This parameter defines what to do with data chunks that cannot be unanimously attributed to any ROB fragment due to containing corrupt data or arriving too late. If this parameter is set to 1 then such data chunks will be discarded. Otherwise, the chunks will be assigned to the currently being built ROB fragments. In such a case the error bits, that explain the origin of the error, will be set to the swrod_status byte of the chunk's local header as well as to status word of the ROB fragment header.
• FlushBufferAtStop: Defines fragment builder behavior during Stop-of-Run procedure. If set to 1, the algorithm stops data processing immediately upon receiving SoR command. The data which were present in the internal buffers are flushed. If set to 0, the algorithm keeps processing data from its internal buffers until they get empty.
• L1AWaitTimeout: Building a ROB fragment requires information from the corresponding L1 Accept packet. The timeout defines the maximum time in milliseconds to wait for L1A packet to arrive. If the required L1A packet does not arrive within this timeout the ROB fragment is considered as fully built but it will miss a proper Trigger Type information in the ROB header, where the corresponding attribute will be set to zero.
• ReadyQueueSize: The size of the queue that is used to hold references to completely aggregated data blocks. This queue is used by the building threads that are defined via BuildersNumber parameters. The building threads take the references from this queue and use them to create new instances of the swrod::ROBFragment class. If BuildersNumber is set to zero, this parameter is ignored as no queue will be used in this case.
• ResynchTimeout: This value defines time in milliseconds to wait for building the ROB fragment that corresponds to the last L1ID produced before the Trigger was put on hold. This timeout is used for the stopless recovery procedure to make sure that the fragment builder will be properly synchronized with the rest of the read-out system when the Trigger is resumed. If the given ROB fragment is not built within this timeout, the stopless recovery procedure will continue but its result will not be guaranteed.
• PacketDumpPath: The name of a directory to store files with dumps of corrupted packets. A corrupted packet will be dumped either if L1 ID cannot be reliable extracted from it or the DropCorruptedPackets parameter is set to true.
• PacketDumpLimit: The maximum number of packets to dump per run.
• LinkLoggingLimit: The maximum number of data corruption incidents reported for a single E-Link. When this limit is reached further incidents will not be reported.

GBT Fragment Building Algorithm

The ROD Data element produced by this algorithm is split into several contiguous memory blocks, with each block containing copies of data packets received from a subset of the E-Links associated with the given ROB in the SW ROD configuration. In most cases, the number of blocks equals the number of data-receiving threads defined by the WorkersNumber attribute of the corresponding SwRodModule configuration object. However, the number of blocks may be greater if the default memory page size is insufficient to accommodate data from all input E-Links. In such cases, additional memory blocks will be automatically allocated by the fragment builder.

Each memory block contains a sequence of data packets received from the associated E-Links, with each packet preceded by a 64-bit header structured as follows:

• size: 16-bit value that contains a total size (in 4-byte words) of the data packet including the size of the header itself.
• felix_status: 8-bit status of the data packet provided by FELIX.
• swrod_status: 8-bit status of the data packet assigned by the fragment building algorithm. This status may contain any combination of the following flags:
  • swrod::GBTChunk::Status::Ok (0): no errors detected for this packet.
  • swrod::GBTChunk::Status::Corrupt (1): custom TTC information extraction has thrown exception for this packet.
  • swrod::GBTChunk::Status::CRCError (2): custom data integrity checking function has returned false for this packet.
  • swrod::GBTChunk::Status::L1IdMismatch (4): the packet's L1 ID does not match L1 ID from the corresponding L1A packet.
  • swrod::GBTChunk::Status::BCIdMismatch (8): the packet's BC ID does not match BC ID from the corresponding L1A packet.
• link_id: 32-bit detector resource ID that identifies the origin of this data packet.

Note: The link_id field contains DetectorResourceId value, which corresponds to the FELIX E-Link ID as defined by the respective instance of the SwRodInputLink configuration class.

SwRodGBTModeBuilder configuration class

This OKS configuration class inherits from the SwRodFragmentBuilder and adds a few configuration parameters that are specific for the FULL mode algorithm, namely:

• BufferSize: defines the maximum size of the main aggregation buffer in terms of the number of ROB fragments. This buffer size determines how many fragments can be built simultaneously. The parameter also indirectly affects the data input timeout, which is the time to wait before terminating the aggregation of a ROB fragment that is still missing data chunks from one or more input links. The timeout value can be explicitly set in the SW ROD configuration using the DataReceivingTimeout. However, incomplete fragments may still be produced by the GBT algorithms if the aggregation buffer size is insufficient. This can occur when the BufferSize, divided by the current input data rate, is smaller than the DataReceivingTimeout value. In such cases, the effective timeout (in milliseconds) can be calculated using the following formula:

EffectiveTimeout = min(DataReceivingTimeout, BufferSize / InputRate(kHz))

• DataReceivingTimeout: Defines a timeout in milliseconds for ROB fragment data aggregation. If the specified number of milliseconds elapses after receiving the first data chunk for a particular ROB fragment, that fragment is considered built and will be passed to the fragment consumers, even if it does not contain data chunks from all the E-links associated with the given ROB. By default, this attribute is set to zero, which disables the time-based timeout. In this case, the effective timeout will be determined by the buffer size, as explained in the previous paragraph.
• L1AInitiatesTimeout: If this value is set to true, the data receiving timeout will be activated when the corresponding L1A packet is received. Otherwise, the timeout will be triggered by the first data packet with the specific L1ID.
• MinimumBufferSize: Defines the minimum size of the main aggregation buffer in terms of the number of ROB fragments. The buffer will always attempt to shrink to this value (but not below it) whenever the number of concurrently built fragments decreases.
• RecoveryDepth: If an incoming data chunk contains L1ID and BCID values that do not match the algorithm's expectations, the algorithm will check if this mismatch can be explained by assuming that a number of previous data packets from the same E-link have been missed. To verify this hypothesis, the algorithm will attempt to match the packet's L1ID and BCID to the corresponding values in the current L1 Accept packets. The RecoveryDepth parameter defines the maximum number of L1A packets to be checked during this procedure, as this is the only reliable way to terminate the recovery if the error was caused by data corruption (e.g., a bit flip).

The MaxMessageSize attribute of the base SwRodFragmentBuilder class class has a specific purpose in this algorithm: it defines the maximum size, in bytes, for a single data packet that may arrive from an individual input link. Any data packet exceeding this size will be discarded. The algorithm also uses this value to calculate the size of the memory blocks that will be used for ROB fragments. This size will be set to the maximum of the MaxMessageSize value and the PageSize value of the MemoryPool object associated with the given SwRodRob instance. If the SwRodRob's MemoryConfiguration relationship is empty, the MemoryPool object associated with the SwRodApplication object will be used instead.

Memory Management Configuration

This section provides an example that demonstrates how to choose the optimal values of the SwRodGBTModeBuilder::MaxMessageSize and MemoryPool::PageSize parameters to ensure the best performance and minimize memory footprint of GBT mode algorithm.

SwRodGBTModeBuilder::MaxMessageSize

As explained in the previous paragraph, the primary purpose of this parameter is to define the maximum size of an individual data packet (a packet received from a single E-Link) that will be accepted by the algorithm. For example, if the maximum number of hits a single data packet may contain and the size of each hit are known, the value for this parameter can be easily calculated by multiplying these two values and adding the size of the fixed portion of the data packet format (e.g., L1ID, BCID, etc.).

For instance, if the maximum number of hits a single data packet can contain is 500, the hit size is 3 bytes, and the size of the fixed portion of the data packet is 20 bytes, then this parameter should be set to:

500*3 + 20 = 1520

MemoryPool::PageSize

This parameter should be used to minimize the memory footprint of the SW ROD. The optimal value of this parameter can be calculated by multiplying the following three values:

• The size of a single hit
• The maximum number of hits in the bulk of the data packets
• The number of input E-Links for the current ROB

For example, if it is known that 99.9% of data packets contain no more than 5 hits and the number of E-Links used to provide data for the current ROB is 150, then the PageSize parameter should be set to the following value:

(5 * 3 + 20) * 150 = 5250

Here, 20 bytes represents the size of the fixed portion of a data packet. With this configuration, the majority of the ROB fragments produced by the current algorithm will fit into a single memory block. Rare fragments, which contain unusually large data packets, will be automatically split into multiple data blocks.

Note: If the MaxMessageSize exceeds the PageSize, the former will override the latter, setting the page size to the greater value. This ensures that a single data packet will never need to be split across multiple memory pages. With this approach, this situation is avoided, as any data packet larger than the PageSize will also exceed the MaxMessageSize and therefore be discarded.

Error Handling

The default GBT fragment-building algorithm implements error handling as described by the SW ROD Error Use Cases page. The algorithm may produce incomplete ROB fragments, i.e. fragments that are missing data chunks from one or several E-Links. This can occur if data from these E-Links did not arrive within the defined timeout, as specified in the algorithm configuration, or if the data was not present in the corresponding E-Link data streams.

FULL Mode Fragment Building Algorithm

This algorithm receives data packets from a set of E-Links defined by the corresponding SW ROD configuration. It assumes that each data packet contains fully built ROD Data element and may optionally also contain the ROD Header and ROD Trailer elements. The presence of the latter is controlled by the RODHeaderPresent attribute of the SwRodFullModeBuilder configuration object. The algorithm combines these data packets into objects of the swrod::ROBFragment class. If RODHeaderPresent attribute is set to true the algorithm performs a few consistency checks of the ROD fragment header and sets the error bits in the ROB header status word if any of these checks fail. Complete information about the error bits that may appear in the ROB header status words can be found in the following page.

The following attributes of the base SwRodFragmentBuilder class are specifically relevant for this algorithm:

• DropCorruptedPackets: This parameter is considered only if the RODHeaderPresent parameter is set to 1. In this case, the algorithm will discard any fragment that contains errors in its header or is too short to contain complete ROD Header and ROD Trailer elements.
• MaxMessageSize: The maximum size (in bytes) of a single data packet that may arrive from an individual input link. A data packet exceeding this size will be truncated to MaxMessageSize bytes. If the MaxMessageSize value is greater than the PageSize value of the MemoryPool object associated with the given SwRodRob instance, data packets larger than the PageSize will be split into multiple data blocks. If the SwRodRob's MemoryConfiguration relationship is empty, the MemoryPool object associated with the SwRodApplication object will be used.

SwRodFullModeBuilder configuration class

This OKS configuration class inherits from the SwRodFragmentBuilder and adds the RODHeaderPresent parameter. If this parameter is set to 1, the algorithm assumes that each incoming data packet already contains the ROD header and ROD trailer elements, and thus will not add them. Otherwise, the algorithm will generate the ROD header and ROD trailer itself.

Memory Management Configuration

This section contains an example that demonstrates how to choose the optimal values of the SwRodGBTModeBuilder::MaxMessageSize and MemoryPool::PageSize parameters to assure the best performance and minimize memory footprint of FULL mode fragment-building algorithm.

SwRodFullModeBuilder::MaxMessageSize

Unlike the GBT mode algorithm the FULL mode algorithm never discards incoming data packets, but may truncate them if they are excessively large. This is where the MaxMessageSize parameter comes into play. It defines the maximum size of an individual data packet that the algorithm will accept without truncation. If a data packet exceeds the MaxMessageSize value, the algorithm will truncate it to that size.

To calculate the appropriate value for this parameter, one can multiply the maximum number of hits a single data packet may contain by the size of each hit and then add the size of the fixed portion of the data packet format (e.g., ROD Header and ROD Trailer).

For example, if the maximum number of hits is 500 and the hit size is 3 bytes and the fixed portion of a data packet is equal to 50 bytes, then the MaxMessageSize parameter should be set to the following value:

500*3 + 50 = 1550

MemoryPool::PageSize

This parameter is used to minimize the memory footprint of the SW ROD application. The optimal value of this parameter can be calculated by multiplying the size of a single hit by the maximum number of hits that the majority of data packets may contain, and then adding the size of the fixed portion of the data packet format.

For example, if it is known that 99.9% of data packets contain no more than 100 hits, and the size of the fixed portion of a data packet is 20 bytes, then this parameter should be set to:

100 * 5 + 20 = 520

The goal is for the majority of ROB fragments produced by the algorithm to fit into a single memory block. Rare data packets that are larger than the PageSize will automatically be split into multiple data blocks.

Note: The PageSize value can be set to be equal to or greater than the MaxMessageSize, ensuring that every fragment of the given ROB will always consist of a single memory block.

Configuring SW ROD Application

A SW ROD application must be configured using OKS configuration service. For convenience all OKS classes that can be used for this purpose have their names started with SwRod prefix. These classes are defined in the daq/schema/swrod.schema.xml OKS schema file that must be included by any SW ROD OKS configuration. A fully functional example of a SW ROD configuration can be found in the data/SwRodSegment.data.xml file located in the swrod package.

The SW ROD configuration schema enables splitting a SW ROD application configuration across multiple files, making a clear distinction between the detector-specific and TDAQ-specific parts, thus simplifying maintenance. The following diagram illustrates a recommended approach for handling the SW ROD configuration. The green boxes represent classes to be instantiated in the detector-specific portion of the configuration, while the yellow boxes denote the classes used for creating the TDAQ portion. Further details will be provided in the following sections.

Note: SwRodApplication, SwRodModule and SwRodRob classes inherit from the legacy read-out configuration classes to make the new SW ROD-based readout configuration compatible with the legacy ROS-based one.

Detector Specific Configuration

Detector-specific portion of a SW ROD configuration is expected to contain the objects of the following three classes:

• SwRodInputLink: The objects of this class define a set of E-Links for receiving data.
• SwRodDataChannel: These objects define a mapping of E-Links to ROBs.
• SwRodCustomProcessingLib: This class defines configuration of a detector specific custom processing plugin.

SwRodInputLink class

This class is used to describe a set of input E-Links for a particular detector and implements the mapping of FELIX-specific E-Link IDs to detector specific Resource IDs. This class has three attributes:

• FelixId: The ID of this link as defined by the FELIX system. This ID must be unique within ATLAS.
• DetectorResourceId: The ID of the detector read-out element connected to this FELIX link. This ID must be unique for a given sub-detector.
• DetectorResourceName: A human-readable name for the detector read-out element.

It is recommended to place all instances of this class in a specific OKS configuration file (or a set of files), which then can be effectively shared by SW ROD configuration segments.

SwRodDataChannel class

An object of this class defines a set of input links for a given ATLAS data channel (ROD) as well as a custom processing plugin that has to be used for this channel. It has the following relationships:

• Contains: Inherited from ResourceSetAND class, this relationship must contain references to the objects of the SwRodInputLink class that represent the corresponding E-Links.
• CustomLib: A reference to an instance of the SwRodCustomProcessingLib class.

This class also includes several attributes that influence the procedure manipulating ECR counters maintained by the FELIX cards, such as:

• TTCControllerName: Defines the name of the segment controller used to determine whether the SW ROD application must send the ECR reset command to the FELIX cards when a new run starts. By default, this is set to "RootController". When the SW ROD application receives the PrepareForRun command, it checks the controller's state. If the controller is not yet in the RUNNING state (as when a new run is starting), the SW ROD application sends the ECR reset command. If the value of this parameter is empty, the command is not sent.
• UpdateECRCounter: If set to 1, the SW ROD application updates the ECR counters of the FELIX cards it is subscribed to with the last known ECR value during the the TTC Restart, Stopless Recovery and Resynchronise procedures.

It is recommended that all instances of this class be placed in a separate OKS configuration file, which should also include files defining the objects of the SwRodInputLink class.

SwRodCustomProcessingLib class

This class should be used for configuring custom detector plugins for the SW ROD. It has the following attributes:

• LibraryName: Specifies the name of the shared library that implements the custom plugin.
• TrigInfoExtractor: The name of the mandatory function used to extract trigger information from the incoming data.
• DataIntegrityChecker: The name of the optional function responsible for performing data integrity checks. If the plugin does not provide this function, the attribute should be left empty.
• ProcessorFactory: The name of the optional custom processor factory function. This function creates custom processors that handle the data as it flows through the system. If the plugin does not provide a processor factory, this attribute should be left empty.

Each instance of the SwRodDataChannel class, which represents a data channel for the SW ROD application, must be linked to an appropriate instance of the SwRodCustomProcessingLib class. This linkage ensures that the necessary custom processing logic is applied to the data channel's data, as defined by the custom processing library.

TDAQ Specific Configuration

This portion of the SW ROD configuration defines:

• Computers where the SW ROD applications will be running
• The mapping of the SW ROD data channels (RODs) to the respective SW ROD applications
• The HLT request handling and event monitoring parameters

SwRodApplication Class

An instance of the SwRodApplication class serves as the entry point to the SW ROD configuration, fulfilling several roles:

• Defines the standard Run Control application parameters for the swrod_application process.
• Points to an instance of the SwRodConfiguration class, which specifies the TDAQ-specific portion of the SW ROD configuration.
• Contains a set of SwRodRob objects that define the standard ATLAS ROB-to-ROD mapping. RODs are represented by the corresponding instances of the SwRodDataChannel class, as defined by the detector-specific portion of the SW ROD configuration.

SwRodConfiguration Class

The SwRodConfiguration class declares the following parameters:

• Plugins: A list of SwRodPluginLib objects, which reference shared libraries that provide implementation of the core SW ROD interfaces. At a minimum, this list must contain a reference to the libswrod_core_impl.so library, which provides default implementations of these interfaces.
• L1AHandler: A reference to an instance of a class that inherits SwRodL1AInputHandler interface. By default, this relationship should point to an instance of the SwRodDefaultL1AHandler class, which provides the default implementation of the L1 accept message handler.
• Consumers: A list of objects implementing the SwRodFragmentConsumer interface. Consumers from this list will receive all fragments for all ROBs produced by the current SW ROD application. Note that the order of objects in this list matters. ROB fragments will be passed to the consumers in the same order as they are linked to the SwRodConfiguration instance. For example, if the SwRodEventSampler consumer precedes the SwRodCustomProcessor, any monitoring task connected to the current SW ROD application will receive ROB fragments without custom processing applied.
• InputMethod: A reference to an object implementing SwRodInputMethod interface. For getting data from FELIX, one should reference an instance of the SwRodFelixInput class.

SwRodFragmentConsumer interface implementations

The SwRodFragmentConsumer is an abstract base class for any resource that processes fully built ROB fragments. It declares three attributes:

• Type: A string ID representing the specific fragment builder type used to create an instance of the respective consumer at runtime. Each subclass of SwRodFragmentConsumer must provide a unique type name for its instantiation.
• WorkersNumber: Specifies the number of worker threads for this consumer.
• CPU: Defines the CPU affinity for the worker threads of this consumer. It is a string parameter that contains CPU core numbers separated by commas and may include ranges. For example: 0,5,7,9-11.

SW ROD provides several implementations of the SwRodFragmentConsumer interface, each serving a different purpose and configurable through corresponding OKS classes:

• SwRodFileWriter: This implementation writes data produced by the SW ROD application to the standard ATLAS raw data file using the ATLAS event format. It can be used for individual ROBs or for the whole SW ROD application. When used at the application level, it combines fragments from different ROBs with the same L1ID into a single ATLAS event.
• SwRodHLTRequestHandler: Implements the standard High-Level Trigger (HLT) to Readout Subsystem (ROS) communication protocol. From the perspective of the HLT, a SW ROD application is indistinguishable from a ROS application.
• SwRodEventSampler: This implementation creates an Event Sampler for the SW ROD application. It collates fragments from all ROBs handled by the SW ROD application into a single ATLAS event based on their L1IDs and provides these events to monitoring applications through the TDAQ Event Monitoring interface. Monitoring applications can connect to the Event Sampler using its SamplerType (always set to "SWROD") and SamplerName (set to the ID of the SW ROD application).
• SwRodCustomProcessor: Applies detector-specific custom processing to fully built ROB fragments.

Fragment consumers in SW ROD can be attached at two levels:

• Per ROB: This is done via the SwRodRob::Consumers relationship, where consumers are specific to individual ROBs. This allows for fragment handling at a granular level, with each ROB having its own set of consumers that process its fragments.
• Per Application: This is done via the SwRodConfiguration::Consumers relationship, where consumers are shared across the entire SW ROD application. This is useful when you need to process all fragments produced by the application in a unified way.

While both approaches are supported, there are certain implementation-specific limitations that need to be considered when configuring specific consumers. These limitations may include constraints on how consumers handle data, how fragments are passed to them, and potential conflicts when attaching multiple consumers at different levels. The following table outlines these limitations.

SwRodInputMethod interface implementations

The SwRodInputMethod is an abstract class that defines interface for getting data into SW ROD. SW ROD provides several implementations of this interface:

• SwRodInternalDataGenerator: This is an in-memory data generator used for testing.
• SwRodNetioInput: This implementation can be used to receive data via NetIO protocol. This class is kept for backward compatibility only and shall no longer been used. Use the SwRodFelixInput instead.
• SwRodNetioNextInput: This implementation can be used to receive data via netio-next protocol. This class is kept for backward compatibility only and shall no longer been used. Use the SwRodFelixInput instead.
• SwRodFelixInput: This implementation should be used to receive data from FELIX. It is implkemendted using the FelixClient interface.

The SwRodFelixInput has several attributes which can be used to configure FelixClient communication interface, including:

• DataNetwork: The name or IP address of the network that shall be used for receiving data
• FelixBusGroupName: The FELIX Bus group name
• FelixBusDirectory: The name of the file system directory which is used by the FELIX Bus for storing its internal data.
• FelixBusInterface: This is the legacy attributed that was used with the previous FELIX Bus implementation and must no longer be used.
• FelixBusTimeout: The timeout (in milliseconds) for link resolution via FELIX bus.

SwRodRob Class

Each instance of the SwRodRob class provides configuration for a specific ROB or in another words defines portion of the detector readout from which data must be combined to a single fragment (ROB fragment). This class has the following parameters:

• Id: The ROB identifier, which must be unique across all SW ROD and ROS components. This ID is used by HLT to request the corresponding ROB fragments.
• FragmentBuilder: A reference to an instance of a class that inherits SwRodFragmentBuilder interface. This object provides configuration for the data aggregation algorithm, which is used to build fragments for the given ROB.
• Consumers: A list of objects implementing SwRodFragmentConsumer interface. Consumers from this list will receive all fragments that are built for the given ROB.
• Contains: This relationship is inherited from the ResourceSetAND class and is used to reference an instance of the SwRodDataChannel class. This relationship provides the sole link between the detector and the TDAQ-specific portions of the SW ROD application configuration.

Note: For compatibility with the legacy read-out system configuration Contains is a multi-value relationship that potentially could reference more than one Resource object. However, for a valid SW ROD configuration this relationship must point to exactly one unique SwRodDataChannel instance.

SwRodModule Class

This class provides data receiving configuration for a given set of SwRodRob objects, which are referenced via its Contains relationship. This class has the following attributes:

• WorkersNumber: The number of threads for receiving input data.
• CPU: A string that defines CPU affinity of the data receiving threads. A value may contain CPU numbers separated by commas and may optionally include ranges. For example: 0,5,7,9-11. If this string is empty, then no affinity will be assigned.
• L1AHandler: If this relationship is not empty then a dedicated instance of L1 Accept receiver will be created to be used exclusively by the ROBs that belong to this SwRodModule. If this relationship is empty, the ROBs will use the global L1 Accept receiver that is created with respect to the configuration object referenced by the SwRodConfiguration. If the L1AHandler relationship of this object is empty as well, the fragment building algorithms of the ROBs belonging to the current SwRodModule will operate in data-driven mode.
• Contains: This relationship is inherited from the ResourceSetAND class and is used to reference objects of the SwRodRob class, which will share input parameters defined by the SwRodModule instance.

Customizing SW ROD application

SW ROD declares three abstract interfaces for its main components:

• DataInput: This interface can be used for receiving input data from a given source, e.g. from FELIX.
• ROBFragmentBuilder: This interface can be used to implement an algorithm of building ROB fragments from the data received via the DataInput interface.
• ROBFragmentConsumer: This interface can used for implementing specific processing of fully built ROB fragments.

Default implementations of these interfaces are provided by the libswrod_core_impl.so library and can be used via the corresponding classes in the SW ROD configuration. SW ROD also provides a way to integrate custom implementations of these interfaces into the standard SW ROD application.

Making a Custom Interface Implementation

The following example demonstrates a simple implementation of the ROBFragmentConsumer interface, which counts the incoming ROB fragments.

class ROBFragmentCounter : public swrod::ROBFragmentConsumer {
public:
    ROBFragmentCounter(const boost::property_tree::ptree & config, const swrod::Core & core) 
     : m_counter(0),
       m_ROB_id(-1) {
       m_output_frequency = config.get<uint32_t>("OutputFrequency");
       if (config.count("RobConfig")) {
           m_ROB_id = config.get<uint32_t>("RobConfig.Id");
       }
    }

    void insertROBFragment(const std::shared_ptr<swrod::ROBFragment> & fragment) override {
       if ((++m_counter % m_output_frequency) == 0) {
           std::cout << m_counter << " fragments have been built for ROB " << m_ROB_id << std::endl;
       };
       forwardROBFragment(fragment);
    }

    void runStarted(const RunParams & run_params) {
       m_counter = 0;
    }
private:
    uint32_t m_output_frequency;
    uint32_t m_ROB_id;
    uint64_t m_counter;
};

An instance of this consumer can be used at the level of an individual ROB as well as at the level of the entire SW ROD application. Each time a new ROB fragment is produced, it is passed to the instance of the ROBFragmentCounter class via the insertROBFragment() function. In this example the implementation of this function increments the fragment counter and then forwards the given ROB fragment to the other consumers by calling the forwardROBFragment() function. Additionally, every m_output_frequency fragments the function prints the fragment counter to the standard output. The value of the m_output_frequency parameter is taken from the OKS configuration. The following section explains how that is implemented.

Configuring Custom Interface Implementation

The ROBFragmentCounter class constructor takes a reference to the boost::property_tree::ptree instance that represents parameters taken from the corresponding OKS class. For the new consumer type a new OKS class that inherits from the SwRodFragmentConsumer must be declared. This can be done using the following procedure:

• Run the OKS schema editor and create a new OKS schema file
• Add include of the daq/schema/swrod.schema.xml OKS schema file into the new file
• Create a new class (in this case ROBFragmentCounter) inheriting it from the SwRodFragmentConsumer class
• Add a new attribute(s) (in this case the attributed called OutputFrequency) to the new class
• Save the new schema file

The new schema file must be included by the SW ROD configuration. After that one can create a new instance of the ROBFragmentCounter class and add it either to a SwRodRob or to a SwRodConfiguration instances depending on whether counting has to be done at the level of an individual ROB or for the entire SW ROD application.

Note: ROBFragmentCounter constructor implementation uses the "RobConfig" configuration object that is obtained by calling config.get_child("RobConfig") function with the given boost::property_tree::ptree instance. The "RobConfig" parameter is available only if consumer configuration object was linked to an instance of the SwRodRob class via its Contains relationship. In this case this parameter will contain the corresponding SwRodRob configuration. For an instance of the consumer that is attached to the SwRodConfiguration object, the "RobConfig" configuration parameter will not be set and an attempt to call the config.get_child("RobConfig") function result in an exception.

Registering Custom Interface Implementation with the SW ROD

Finally, the SW ROD application must be made aware of the new interface implementation in order to be able to use it at runtime. To achieve this the corresponding interface implementation class must be registered with the swrod::Core singleton, as shown in the following example.

using namespace swrod;

namespace {
    Factory<ROBFragmentConsumer>::Registrator __reg_custom plugin_(
            "ROBFragmentCounter",
            [](const boost::property_tree::ptree& config, const Core& core) {
                return std::make_shared<ROBFragmentCounter>(config, core);
            });
}

This code creates a new factory object that will be used for creating new instances of the ROBFragmentCounter class. The factory will be registered with the "ROBFragmentCounter" name. This name will be used in the OKS configuration as described in the next section. Note that the base class of the new component (ROBFragmentConsumer) must be used as template parameter of the Factory class.

The code must be compiled and linked together with the ROBFragmentCounter class implementation into a shared library that will be dynamically loaded by SW ROD application. Let's assume that such a library is called libswrod_custom_test.so. To make this library known to the SW ROD application a new instance of the SwRodPluginLib class must be created in the corresponding OKS configuration and the shared library name has to be set to its LibraryName attribute. One can either use a full path-name of the shared library or use a short file-name and add the library location to the LD_LIBRARY_PATH environment variable. Finally, the new instance of the SwRodPluginLib class must be linked with the SwRodConfiguration object via the Plugins relationship.

Testing SW ROD Custom Processing library

A custom implementation of the DataInput interface can be used to validate detector-specific custom processing plugins. This chapter explains how this can be done.

Implementing internal data generator for SW ROD

The simplest way of providing custom input to SW ROD is to implement internal data generator that produces data with desired formatting in memory of the SW ROD application. The swrod package contains an example of such generator in test/core/InternalDataGenerator.h(cpp) files. One can customize internal data generation by declaring a new class that inherits from the swrod::test::InternalDataGenerator and overrides the generatePacket(InputLinkId link, uint32_t l1id, uint16_t bcid) virtual function. This function is called for every new packet to be produced and is expected to pass the packet to the dataReceived(InputLinkId link, const uint8_t * data, uint32_t size, uint8_t status) function of the swrod::DataInput interface, as shown in the following example.

 void MyDataGenerator::generatePacket(InputLinkId link, uint32_t l1id, uint16_t bcid) {
    // For efficiency m_packet memory block had been preallocated in the constructor
    // Here we just calculate the size of the new packet. 
    // It must not exceed the size of the m_packet memory block
    uint32_t new_packet_size = ...;  
    
    // set TTC values to the appropriate places of the new packet, for example
    *((uint32_t*) (m_packet + 2)) = l1id;
    *((uint16_t*) (m_packet + 6)) = bcid;   
    
    dataReceived(link, m_packet, new_packet_size, 0);
}

Note that if a custom implementation produces packets of fixed size, it can calculate the packet size only once in the class constructor. A custom implementation that needs to generate packets of varying size must calculate a new packet size each time a new packet is produced. Finally, the new packet must be passed to the dataReceived(...) function, which in turn will pass it to the SW ROD fragment builder.

The swrod::test::InternalDataGenerator class also provides another virtual function called beforeStart(). This function is called each time a new run is about to be started and can be used to reset the internal counters of the custom data generator.

The new data generator must be advertised to the SW ROD by creating and registering a new object factory, as shown in the following example.

namespace {
    Factory<DataInput>::Registrator __reg__(
            "MyDataGenerator",
            [](const boost::property_tree::ptree& config, const Core& ) {
                return std::make_shared<MyDataGenerator>(config);
            });
}

Finally, the new generator class must be compiled into a shared library and the library name must be set to the LibraryName attribute of the new instance of the SwRodPluginLib object created in OKS configuration. This OKS object must be linked with the SwRodConfiguration via the Plugins relationship. This will make the new input method implementation known to the SW ROD application.

In order to use the new generator a new instance of the SwRodInternalDataGenerator class must be created and its Type attribute must be set to the same "MyDataGenerator" string, that was used for registering the corresponding class with the SW ROD plugins factory. Finally, this object has to be linked either with the SwRodConfiguration or SwRodModule via the InputMethod relationship.

Note: An internal data generator must be used in conjunction with InternalL1AGenerator class provided by the swrod package. The InternalL1AGenerator produces L1A packets which are used as seeds for data packets generation.