Skip to content

Packing: Allow dependencies to be anonymized

Gerhard Raven requested to merge anonymous-dependencies into master

The packing packs two distinct groups of objects:

  1. all entries in containers which are explicitly requested to be packed
  2. entries which are solely required to (recursively) satisfy dependencies of items scheduled to be packed

Note that 2 will result in additional containers from the TES being packed, but not their entire content: only the subset of entries in them that happen to be 'referred to' by items in 1 or (recursively) 2 will be packed. Prior to this MR, the unpacking is unable to distinguish between these two case, so one can explicitly request a container to be unpacked, but depending on why it was packed, what one gets is either the complete container (case 1), or it a subset (case 2).

This MR adds the option to 'anonymize' the containers in category 2 during packing. If this option is enabled, it will no longer possible to explicitly request the containers in category 2 to be unpacked. These containers will only be unpacked if during unpacking of containers in category 1 there happen to be dependencies which need to be satisfied. In that case, the unpacking automatically unpacks the required data, and puts them in an automatically generated TES location (which should not be directly used, it just exist to manage the lifetime of the data, and to re-establish the references from the objects using this data).

The above has two important consequences:

I. it will no longer be needed to populate the ANNSvc with the names of the dependent locations, making the encoding keys much more stable, and thus reducing the chance of generating new keys

II. analysis will no longer be able to explicitly (presumably accidentally) use one of the dependent locations. This is IMO a good thing, as the decision which subset has been packed into them heavily depends on the which trigger lines fired, and what the trigger lines which fired requested to be packed. Note that all the data currently packed without anonymizing the dependencies is still available, it just implies that there is no direct way to navigate to this data -- it can only be reached by following the dependencies starting at objects in containers which are in category 1.

Note that at this point, the default behavior remains 'as is' -- to enable it, the AnonymizeDependencies property of SelectivePacker must be set to True. Also, the unpackers have gained a property WriteEmptyContainerIfBufferNotFound which defaults to True which, in case a (known!) container is requested but not present in the packed data, will create an empty container. This matches the current behavior, and some tests rely on this. Eventually, both these options should be switched to False.

Must be applied in conjunction with Moore!2227 (merged) DaVinci!889 (merged)

Edited by Gerhard Raven

Merge request reports