Generate configuration in a separate process to reduce memory usage
Generating the configuration can result in several GBs of RSS overhead which is kept for the remainder of the process. In practice this is no longer needed. It's not obvious how to deallocate this memory due to Configurables being singletons so it's easier to generate the configuration in a subprocess and then pass it back to the parent via shared memory.
Examples:
HLT2 MC Reconstruction
Before:
After:
test_davinci_tupling_All (prebuilt functor cache)
Before:
After
Large-ish DaVinci job from RD (prebuilt functor cache)
Job was killed after ~300 seconds.
Before:
After:
Large-ish DaVinci job from RD (no functor cache)
Job was killed after ~600 seconds.
Before:
After:
