Skip to content

introduce ExecDriver (plus other EventLoop updates)

Nils Erik Krumnack requested to merge krumnack/athena:exec_driver into main

As the main feature this introduces ExecDriver. The idea behind ExecDriver is that it runs the actual job in a new executable (replacing the current one). This is meant to release all the memory that gets allocated by our configuration helpers. That is over 1GB (mostly ROOT-python dictionaries), so this is a significant savings.

There are a couple more updates that work with this:

  • on merging, if there is only one file I create a hard link instead of merging, which is a lot more efficient.
  • BatchDriver now respects maxEvents (in a fashion), which makes it easier to do tests with LocalDriver and ExecDriver.
  • add an option to FullCPAlgorithmsTest_elJob.py to use the new ExecDriver

Please note that the entire design is more of a quick hack than a fully thought out solution. We are simultaneously reworking the entire EL driver infrastructure and I don't want to make a very thoughtful solution for something that will be replaced in the next 2-3 months.

cc @ravinab @tdado

Edited by Nils Erik Krumnack

Merge request reports