TQSystematicsManager uses excessive amount of memory
The memory consumption of the TQSystematicsManager seems somewhat excessive (orders of magnitude larger than on-disk size of input files). It should be possible to bring this down significantly.
The memory consumption of the TQSystematicsManager seems somewhat excessive (orders of magnitude larger than on-disk size of input files). It should be possible to bring this down significantly.
It looks like loading sample folders is responsible for most of this footprint, and it can be reduced significantly by lazy loading them (using tag lazy
= true), or even not using caching at all (tag useCache
= false). In my case:
lazy | caching | mem (GB) | time (mm:ss) |
---|---|---|---|
false | true | 2.5 | 1:05 |
true | true | 1.7 | 0:43 |
true | false | 0.7 | 1:20 |
Lazy loading actually seems to improve performance, so I've turned it on by default. I also made a slight optimization such that when caching is turned off, the empty cache is no longer checked against every time.
created branch 100-tqsystematicsmanager-uses-excessive-amount-of-memory
mentioned in merge request !221 (closed)
mentioned in merge request !222 (closed)
Hi Aaron! That's quite surprising. In principle, it's possible that lazy loading is more efficient, since it only loads the parts of the sample folder "on-demand". However, I would have expected the TQSystematicsManager to access the full sample folder during its run, such that the lazy loading would only produce overhead. Do you know which parts of the sample folder are loaded / not loaded? This might hint towards a bug somewhere... Regards, Carsten
I believe the memory is allocated during the loadSampleFolder
call, so in principle the answer is in that function.