Skip to content

Reset ObjectCount to Avoid Exploding TProcessID Table

Simon Spannagel requested to merge fix_tref_object_count into master

This MR re-adds the reset of the TProcessID object count after processing of each event. This

The MR also adds the following documentation to the manual:

Object History, TRefs and PointerWrappers

Allpix Squared uses ROOTs TRef objects to store the persistent links of the simulation object history. These references act similar to C pointers and allow accessing the referenced object directly without additional bookkeeping or lookup of indices. Furthermore they persist when being written to a ROOT file and read back to memory. ROOT implements this via a central lookup table that keeps track of the referenced objects and their location in memory as described in the ROOT documentation.

This approach comes with some drawbacks, especially in multithreaded environments. Most importantly the lookup table is a global object, which means mutexes are required for accessing it. Multiple threads generating or using TRef references will have to share this mutex and will consequently be subject to significant waiting for lock release. Furthermore generating more and more TRef relations over the course of a simulation will increase the size of the central reference table. This table is initialized with a fixed size, and once the number of TRef objects outgrows this pre-allocated space, new memory has to be acquired, leading to a reallocation of memory for the entire new size of the table. With potentially millions of entries, this very quickly becomes a very computationally very expensive operation, slowing down the simulation significantly.

Allpix Squared solves these limitations by wrapping the TRef objects into a class called PointerWrapper. It contains both a direct, but transitional C pointer and a TRef to the referenced object. The latter, however, is only generated when required, i.e. if the object holding the PointerWrapper as well as referenced object are going to be written to file. This is achieved by first going through all relevant objects, marking them for storage:

for(auto& object : objects) {
    object.markForStorage();
}

Now, the required history references can be identified and TRef objects are generated only for relations between two objects that are both marked for storage:

for(auto& object : objects) {
    object.petrifyHistory();
}

Objects can now be written to file and will contain the persistent reference to the related object.

This approach solves the above problems. File writing has to be performed single-threaded anyway, so generating TRef objects on the same thread does not lead to additional locking of the central reference table mutex in root. In addition, TRef entries are only generated and stored in the table for objects that require it - all references to objects not to be stored will be nullptr in either case since the target object is not available anymore when reading in the data. Since now the generation of TRef objects and access to the reference table is performed by a single thread and one single event at a time, it is also possible to reset the ROOT-internal object ID of TRef references after the event has been processed. The subsequent event will reuse the same IDs again, preventing a continuous growth of the reference table and related memory re-allocation issues.

As a consequence, when reading objects back from file, the TRef has to be converted back to a C memory pointer, again to avoid locking access to the central reference table when looking up the memory location from there. This is performed similarly to the generation of history relations, and here only relations to valid TRefs are loaded, other relations will hold a nullptr:

for(auto& object : objects) {
    object.loadHistory();
}
Edited by Simon Spannagel

Merge request reports