Generate configuration in a separate process to reduce memory usage

Generating the configuration can result in several GBs of RSS overhead which is kept for the remainder of the process. In practice this is no longer needed. It's not obvious how to deallocate this memory due to Configurables being singletons so it's easier to generate the configuration in a subprocess and then pass it back to the parent via shared memory.

Examples:

HLT2 MC Reconstruction

Before:

After:

test_davinci_tupling_All (prebuilt functor cache)

Before:

After

Large-ish DaVinci job from RD (prebuilt functor cache)

Job was killed after ~300 seconds.

Before:

After:

Large-ish DaVinci job from RD (no functor cache)

Job was killed after ~600 seconds.

Before:

After:

added RTA label

requested review from @clemenci

@cburr I was told this MR would be rather beneficial for MC production. I am just preparing release for block 5 and 6, do you have any time estimation when this MR could be ready so it could be included?

added 3 commits

• 4fce433c...bd4d042f - 2 commits from branch 2024-patches
• 7756c4d9 - Generate configuration in a separate process to reduce memory usage

Compare with previous version

resolved all threads

added 1 commit

• 5f3b1032 - Generate configuration in a separate process to reduce memory usage

Compare with previous version

/ci-test

marked this merge request as ready

added ci-test-triggered label

• [2024-12-06 10:53] Validation started with lhcb-2024-patches-mr#1870

@clemenci Would you have any further questions to this MR? If not, then please approve.

approved this merge request

resolved all threads

mentioned in commit 0aa5963d

merged

@cburr Do you have time to prepare MR for master today? I am pushing to have a full release on master either later today or tomorrow.

@clemenci to me he would do one last sync

What would be the timescale for that? I guess it is not really doable to have it within a day or two, so this would be then included in some release later.

He mentioned he would try to do it on Friday. @clemenci is it feasible to do today or should I just cherry pick it myself?

I had a chat with @clemenci and from RTA side we are still having a few things going into 2024-patches (some special runs waiting to be processed) plus possible fixes for EoY Sprucing (deadline this Wednesday). I think it would make more sense to wait for those developments and then have the last synchronization.

mentioned in merge request !4804 (merged)

mentioned in commit ffbe3273

mentioned in commit 2c4348f0

mentioned in commit 92223e2d

picked the changes into the branch cherry-pick-0aa5963d with commit 9e7f3ada

mentioned in commit 9e7f3ada

mentioned in merge request !4823 (merged)

mentioned in issue Moore#896 (closed)

picked the changes into the branch cherry-pick-0aa5963d with commit b13afd68

mentioned in commit b13afd68

mentioned in merge request !4828 (merged)

picked the changes into the branch cherry-pick-0aa5963d with commit 3dd04b64

mentioned in commit 3dd04b64

mentioned in merge request !4832 (merged)

mentioned in commit db7a403e

picked the changes into the branch cherry-pick-0aa5963d with commit 4d0546ec

mentioned in commit 4d0546ec

mentioned in merge request !4845 (merged)

picked the changes into the branch cherry-pick-0aa5963d-2 with commit 4eaa6e6c

mentioned in commit 4eaa6e6c

mentioned in merge request !4846 (merged)

mentioned in commit f839f3ab

mentioned in commit 1a1037e3

mentioned in commit 412b3344

mentioned in commit 928e89de

mentioned in commit d8fbba46

mentioned in merge request !4774 (merged)

picked the changes into the branch cherry-pick-0aa5963d with commit 918c6881

mentioned in commit 918c6881

mentioned in merge request !4955 (merged)

mentioned in commit f33ad1ea