Skip to content

Use nlohmann::json instead of YAML::Node for in-memory condition format

Marco Clemencic requested to merge use-json-conditions-in-memory into master

As discussed in LHCb#241 (closed) and LHCb!3768 (merged), YAML::Node uses a lot of memory for relatively simple data.

With a relatively simple test I managed to dump all conditions we load in LoadDDDB as a single YAML file of 57MB and loading all 16919 entries required 2.3GB of RSS. The same kind of exercise with nlohmann::json requires a much more reasonable 80MB.

This MR aims to change the way we keep conditions in memory from YAML::Node instances to nlohmann::json objects. For ConditionsOverlay I replaced YAML::Node with its string representation.

A couple of helper functions (taken from LHCb!3768 (merged)) are used to convert between the two formats.

Note that the on-disk format of the conditions stays as YAML, which is more practical (for humans) to read and write than JSON.

YAML::Node is still used in a few places:

  • to construct some objects from conditions (JSON converted to YAML on the fly)
  • conditions overrides (it acts before YAML data is converted to JSON)
  • both JSON and YAML are supported in couple of cases

It must be merged at the same time as LHCb!3768 (merged).

Edited by Marco Clemencic

Merge request reports