Regression in peak memory usage of histogram saving
!1477 (merged) introduced a regression (now in v37 and also ported back to v36) where the peak memory allocation during saving ROOT histograms is determined by the total memory footprint of the JSON representation for all histograms. This is quite problematic for realistic LHCb online monitoring workloads where we have many histograms with many bins.
A test has been extended to check for this in !1504
Below is a quick fix for v36-patches
but a mergeable fix needs to be found.
diff --git GaudiKernel/include/Gaudi/BaseSink.h GaudiKernel/include/Gaudi/BaseSink.h
index 19326ac5c..ba9741d37 100644
--- GaudiKernel/include/Gaudi/BaseSink.h
+++ GaudiKernel/include/Gaudi/BaseSink.h
@@ -69,6 +69,12 @@ namespace Gaudi::Monitoring {
return sortedEntities;
}
+ std::map<std::string, std::map<std::string, Hub::Entity const*>> const sortedEntitiesPtrs() {
+ std::map<std::string, std::map<std::string, Hub::Entity const*>> sortedEntities;
+ applytoAllEntities( [&sortedEntities]( auto& ent ) { sortedEntities[ent.component][ent.name] = &ent; } );
+ return sortedEntities;
+ }
+
private:
/// deciding whether a given name matches the list of regexps given
/// empty list means everything matches
diff --git GaudiKernel/include/Gaudi/Histograming/Sink/Base.h GaudiKernel/include/Gaudi/Histograming/Sink/Base.h
index a499489c1..b1102bce2 100644
--- GaudiKernel/include/Gaudi/Histograming/Sink/Base.h
+++ GaudiKernel/include/Gaudi/Histograming/Sink/Base.h
@@ -53,9 +53,10 @@ namespace Gaudi::Histograming::Sink {
// run may be mixed with new one
TFile histoFile( m_fileName.value().c_str(), "UPDATE" );
// get all entities, sorted by component and name
- std::map<std::string, std::map<std::string, nlohmann::json>> const sortedEntities = sortedEntitiesAsJSON();
+ auto const sortedEntities = sortedEntitiesPtrs();
for ( auto& [component, entityMap] : sortedEntities ) {
- for ( auto& [name, j] : entityMap ) {
+ for ( auto& [name, entity] : entityMap ) {
+ auto j = entity->toJSON();
auto dim = j.at( "dimension" ).template get<unsigned int>();
auto type = j.at( "type" ).template get<std::string>();
// cut type after last ':' if there is one. The rest is precision parameter that we do not need here
Edited by Rosen Matev