ATLASSIM-3150: Enable use of static Geant4 in Athena using "BigLibrary" pattern to improve simulation performance

Benjamin Morgan requested to merge bmorgan/athena:atlassim-3150 into master

This provides an implementation of the "BigLibrary" pattern for Athena simulation/Geant4-using packages as outlined in the long running ATLASSIM-3150 ticket, which should be consulted for the full details/discussions/results.

The primary goal was to improve throughput of the simulation via utilising Geant4 static libraries with private symbols - enabling the removal of function calls through the PLT (so-called "trampolines") and potential additional optimization by the compiler as Geant4 symbols do not need to be seen outside of the shared library that uses Geant4. The cost of this approach is that only one shared library in any link chain or process (e.g. via dlopen) can link to/use symbols from the Geant4 static libraries. For Athena, this means all libraries/components that use Geant4 must be compiled into a single shared library to provide this single point of contact/use of Geant4 within an Athena process (whether linked to or loaded as a Python extension). The same approach is already in use by CMS.

As implemented in this MR, the approach for Athena is to:

  1. Build all Geant4-using libraries and components as CMake OBJECT libraries with PIC enabled using athena_add_library.
    • The target names in athena_add_library have _obj appended to distinguish them
    • Any _obj target that links to other Geant4-using libraries in Athena links to the new _obj targets. This forwards CMake usage requirements for compilation, whilst deferring linking/composition to the following BigSimulation step
  2. Add a new BigSimulation package under Simulation that
    • Composes a BigSimulationLib shared library from all _obj libraries using athena_add_library
    • Composes a BigSimulation component from all _obj libraries and components using athena_add_library
    • BigSimulation has -Wl,--exclude-libs,ALL link flags added to avoid export of symbols from any linked static libs. This is the key step to gain speedup from use of static libraries
    • BigSimulationLib is a slight artifact of the testing/development process that incrementally added packages into the big libraries. By using CMake ALIAS targets that aliased the old non-Object target names to the BigSimulationLib target, non-merged packages did not require any CMake changes. It currently only appears linked to by a handful of ROOT dictionary libraries discussed below.
  3. A handful of packages required update of their Python configuration to create the component(s) from the main Athena config manager, but I understand this is now the recommend method anyway.

The resultant build is fully compatible with both Geant4 shared or static libraries, and so does not require a co-working change in atlas/atlasexternals unless wanted. All CTests that pass for the current master also pass for this MR built with either static/shared Geant4. With Geant4 shared libraries, there is a runtime cost of ~10s at the initialisation stage (only fully testing in a transform), but there is no observable difference in the event loop runtime. This initialization cost is also present with static Geant4 libraries, but the event loop is of order 7% faster (measured with ttbar events). Output hit files from runs with each of these builds have been compared using diff-root with no differences reported, at least over 100 ttbar events.

This has been started as a Draft both given the scale of the changes to review, and to discuss/iterate on the one remaining development item, that of ROOT dictionaries for packages that use Geant4. These are limited to the packages/dicts:

  • LArG4CodeEnums defined in LArCalorimeter/LArG4/LArG4Code/CMakeLists.txt
  • LArG4GenShowerLibDict defined in LArCalorimeter/LArG4/LArG4GenShowerLib/CMakeLists.txt
  • LArWheelSolidCheckerDict defined in Simulation/G4Utilities/Geo2G4/CMakeLists.txt

Discussion in ATLASSIM-3150 has identified an initial way forward to work with these in BigSimulation and will be tested and pushed here for further tests/discussion.

Edited by Benjamin Morgan

Merge request reports