introduce ExecDriver (plus other EventLoop updates)
As the main feature this introduces ExecDriver. The idea behind ExecDriver is that it runs the actual job in a new executable (replacing the current one). This is meant to release all the memory that gets allocated by our configuration helpers. That is over 1GB (mostly ROOT-python dictionaries), so this is a significant savings.
There are a couple more updates that work with this:
- on merging, if there is only one file I create a hard link instead of merging, which is a lot more efficient.
- BatchDriver now respects maxEvents (in a fashion), which makes it easier to do tests with LocalDriver and ExecDriver.
- add an option to
FullCPAlgorithmsTest_elJob.py
to use the newExecDriver
Please note that the entire design is more of a quick hack than a fully thought out solution. We are simultaneously reworking the entire EL driver infrastructure and I don't want to make a very thoughtful solution for something that will be replaced in the next 2-3 months.
Edited by Nils Erik Krumnack