Fixes to internals / speedup and reduce memoryusage

I got a very interesting test case from @fbury where bamboo gets slow and uses a lot of memory (when using tensorflow is used, but probably not due to that), and ran into a few things that could be improved already, so collecting the patches here.