ATLASSIM-3150: Enable use of static Geant4 in Athena using "BigLibrary" pattern to improve simulation performance
This provides an implementation of the "BigLibrary" pattern for Athena simulation/Geant4-using packages as outlined in the long running ATLASSIM-3150 ticket, which should be consulted for the full details/discussions/results.
The primary goal was to improve throughput of the simulation via utilising Geant4 static libraries with private symbols - enabling the removal of function calls through the PLT (so-called "trampolines") and potential additional optimization by the compiler as Geant4 symbols do not need to be seen outside of the shared library that uses Geant4. The cost of this approach is that only one shared library in any link chain or process (e.g. via dlopen
) can link to/use symbols from the Geant4 static libraries. For Athena, this means all libraries/components that use Geant4 must be compiled into a single shared library to provide this single point of contact/use of Geant4 within an Athena process (whether linked to or loaded as a Python extension). The same approach is already in use by CMS.
As implemented in this MR, the approach for Athena is to:
- Build all Geant4-using libraries and components as CMake
OBJECT
libraries with PIC enabled usingathena_add_library
.- The target names in
athena_add_library
have_obj
appended to distinguish them - Any
_obj
target that links to other Geant4-using libraries in Athena links to the new_obj
targets. This forwards CMake usage requirements for compilation, whilst deferring linking/composition to the followingBigSimulation
step
- The target names in
- Add a new
BigSimulation
package underSimulation
that- Composes a
BigSimulationLib
shared library from all_obj
libraries usingathena_add_library
- Composes a
BigSimulation
component from all_obj
libraries and components usingathena_add_library
-
BigSimulation
has-Wl,--exclude-libs,ALL
link flags added to avoid export of symbols from any linked static libs. This is the key step to gain speedup from use of static libraries -
BigSimulationLib
is a slight artifact of the testing/development process that incrementally added packages into the big libraries. By using CMakeALIAS
targets that aliased the old non-Object target names to theBigSimulationLib
target, non-merged packages did not require any CMake changes. It currently only appears linked to by a handful of ROOT dictionary libraries discussed below.
- Composes a
- A handful of packages required update of their Python configuration to create the component(s) from the main Athena config manager, but I understand this is now the recommend method anyway.
The resultant build is fully compatible with both Geant4 shared or static libraries, and so does not require a co-working change in atlas/atlasexternals unless wanted. All CTests that pass for the current master also pass for this MR built with either static/shared Geant4. With Geant4 shared libraries, there is a runtime cost of ~10s at the initialisation stage (only fully testing in a Sim_tf.py
transform), but there is no observable difference in the event loop runtime. This initialization cost is also present with static Geant4 libraries, but the event loop is of order 7% faster (measured with ttbar events). Output hit files from Sim_tf.py
runs with each of these builds have been compared using acmd.py diff-root
with no differences reported, at least over 100 ttbar events.
This has been started as a Draft both given the scale of the changes to review, and to discuss/iterate on the one remaining development item, that of ROOT dictionaries for packages that use Geant4. These are limited to the packages/dicts:
-
LArG4CodeEnums
defined inLArCalorimeter/LArG4/LArG4Code/CMakeLists.txt
-
LArG4GenShowerLibDict
defined inLArCalorimeter/LArG4/LArG4GenShowerLib/CMakeLists.txt
-
LArWheelSolidCheckerDict
defined inSimulation/G4Utilities/Geo2G4/CMakeLists.txt
Discussion in ATLASSIM-3150 has identified an initial way forward to work with these in BigSimulation
and will be tested and pushed here for further tests/discussion.