Skip to content

Add machinery for tracking dynamic allocations and enable its use in the scheduler

Olli Lupton requested to merge allocation_tracker into master

This adds a couple of tools that date back to me looking at the custom allocators/memory pool for HLT1. They intercept calls to the system memory allocators and record some statistics and timing information.

As-is, if you pass --preload libAllocationTrackerPreload.so to gaudirun.py and run a job using the HLTControlFlowMgr then a lightweight TimingAllocationTracker will run, which prints a very brief summary:

TimingAllocationTracker INFO Dynamic allocation information: 106.758 allocations/evt yielded 10.2461 MiB/evt taking 1.03977% of processing time

and adds very little overhead (I didn't see a significant slowdown in a few re-runs, so I guess <% cost -- but that's just adding/removing the --preload option. Other overhead should be small.).

There is an alternative StacktraceAllocationTracker, which uses the same machinery but records a lot more detail. Unfortunately it's very slow, because it captures a backtrace on each dynamic allocation [no doubt this could be improved], but the output can be informative:

StacktraceAllocationTracker            INFO Dynamic allocation information:
StacktraceAllocationTracker            INFO *(.*?)\*\s+Gaudi::Functional::details::put<\1, \1, void>\(DataObjectHandle<\1> const&, \1&&\) [100.042 bytes/alloc, ~3.91125 allocs/evt, ~3.36226% alloc time]
StacktraceAllocationTracker            INFO *DataObjectHandle<AnyDataWrapper<(.*?)\s*> >::put\(\1&&\) const [641.679 bytes/alloc, ~32.145 allocs/evt, ~33.5534% alloc time]
StacktraceAllocationTracker            INFO *FixTESPathDetails::fullTESLocation[abi:cxx11](std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >) [libGaudiAlgLib.so, 29.6009 bytes/alloc, ~0.145625 allocs/evt, ~0.0863742% alloc time]
StacktraceAllocationTracker            INFO *LinkManager::newInstance() [libGaudiKernel.so, 32 bytes/alloc, ~37.0156 allocs/evt, ~26.053% alloc time]
StacktraceAllocationTracker            INFO *UpdateManagerSvc::reserve(Gaudi::Time const&) const [libDetDescSvc.so, 24 bytes/alloc, ~0.930625 allocs/evt, ~0.450633% alloc time]
StacktraceAllocationTracker            INFO clone [libc.so.6, 2.0951e+06 bytes/alloc, ~4.98938 allocs/evt, ~5.12019% alloc time]
StacktraceAllocationTracker            INFO .AlgWrapper::execute(EventContext&, gsl::span<AlgState, -1l>) const [libHLTScheduler.so, 8.85888 bytes/alloc, ~0.925625 allocs/evt, ~0.851995% alloc time]
StacktraceAllocationTracker            INFO ..Gaudi::Functional::MergingMultiTransformer<std::tuple<LHCb::HltSelReports, ObjectVector<LHCb::HltObjectSummary> > (Gaudi::Functional::details::vector_of_const_<DataObject*> const&), Gaudi::Functional::Traits::BaseClass_t<Gaudi::Algorithm> >::execute(EventContext const&) const [libSelReports.so, 48 bytes/alloc, ~0.03 allocs/evt, ~0.0154656% alloc time]
StacktraceAllocationTracker            INFO ...ObjectVector<LHCb::HltObjectSummary>* Gaudi::Functional::details::put<ObjectVector<LHCb::HltObjectSummary>, ObjectVector<LHCb::HltObjectSummary>, void>(DataObjectHandle<ObjectVector<LHCb::HltObjectSummary> > const&, ObjectVector<LHCb::HltObjectSummary>&&) [libHltDAQ.so, 64 bytes/alloc, ~0.02875 allocs/evt, ~0.0248388% alloc time]
StacktraceAllocationTracker            INFO ...SelReportsMaker::operator()(Gaudi::Functional::details::vector_of_const_<DataObject*> const&) const [libSelReports.so, 156.594 bytes/alloc, ~0.063125 allocs/evt, ~0.0935622% alloc time]
StacktraceAllocationTracker            INFO ....GaudiUtils::VectorMap<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, float, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, float> > >::insert(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, float const&) [libHltDAQ.so, ]
StacktraceAllocationTracker            INFO .....void std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, float>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, float> > >::_M_realloc_insert<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, float> const&>(__gnu_cxx::__normal_iterator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, float>*, std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, float>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, float> > > >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, float> const&) [libHltDAQ.so, 40 bytes/alloc, ~0.034375 allocs/evt, ~0.0150459% alloc time]
StacktraceAllocationTracker            INFO ....LHCb::HltObjectSummary::HltObjectSummary(LHCb::HltObjectSummary const&) [libHltDAQ.so, 51.2453 bytes/alloc, ~0.06625 allocs/evt, ~0.0335732% alloc time]

(or overly verbose -- see stacktrace.log for a full example).

Note that the first entries, starting with *, are patterns matched against the function name (they're framework related things that otherwise show up under every algorithm).

cc: @chasse @nnolte @graven (in case the Gaudi-related regexes starting with * inspire you ;-))

Edited by Olli Lupton

Merge request reports