Skip to content

Persistreco at Sprucing level does not work as expected with version=1.1

The IFT group use persistreco Sprucing lines with persistreco_version: 1.1 for which the expected locations are here

What we see is that the FittedVeloTracks at location /Event/Rec/Track/BestVelo are not persisted - these are the only objects missing. Looking at the unpacker algs in the Sprucing job log

 NONLAZY_OR: SprucePbPbUPCOutput                                             #=20      Sum=20          Eff=|( 100.0000 +- 0.00000 )%|
       ProtoParticleUnpacker/Unpack_Hlt2__Event_HLT2_Rec_ProtoP_Long              #=20      Sum=20          Eff=|( 100.0000 +- 0.00000 )%|
       ProtoParticleUnpacker/Unpack_Hlt2__Event_HLT2_Rec_ProtoP_Downstream        #=20      Sum=20          Eff=|( 100.0000 +- 0.00000 )%|
       ProtoParticleUnpacker/Unpack_Hlt2__Event_HLT2_Rec_ProtoP_Upstream          #=20      Sum=20          Eff=|( 100.0000 +- 0.00000 )%|
       ProtoParticleUnpacker/Unpack_Hlt2__Event_HLT2_Rec_ProtoP_Neutrals          #=20      Sum=20          Eff=|( 100.0000 +- 0.00000 )%|
       TrackUnpacker/Unpack_Hlt2__Event_HLT2_Rec_Track_BestLong                   #=20      Sum=20          Eff=|( 100.0000 +- 0.00000 )%|
       TrackUnpacker/Unpack_Hlt2__Event_HLT2_Rec_Track_BestDownstream             #=20      Sum=20          Eff=|( 100.0000 +- 0.00000 )%|
       TrackUnpacker/Unpack_Hlt2__Event_HLT2_Rec_Track_BestUpstream               #=20      Sum=20          Eff=|( 100.0000 +- 0.00000 )%|
       RecVertexUnpacker/Unpack_Hlt2__Event_HLT2_Rec_Vertex_Primary               #=20      Sum=20          Eff=|( 100.0000 +- 0.00000 )%|
       CaloHypoUnpacker/Unpack_Hlt2__Event_HLT2_Rec_Calo_Electrons                #=20      Sum=20          Eff=|( 100.0000 +- 0.00000 )%|
       CaloHypoUnpacker/Unpack_Hlt2__Event_HLT2_Rec_Calo_Photons                  #=20      Sum=20          Eff=|( 100.0000 +- 0.00000 )%|
       CaloHypoUnpacker/Unpack_Hlt2__Event_HLT2_Rec_Calo_MergedPi0s               #=20      Sum=20          Eff=|( 100.0000 +- 0.00000 )%|
       CaloHypoUnpacker/Unpack_Hlt2__Event_HLT2_Rec_Calo_SplitPhotons             #=20      Sum=20          Eff=|( 100.0000 +- 0.00000 )%|
       RecSummaryUnpacker/Unpack_Hlt2__Event_HLT2_Rec_Summary                     #=20      Sum=20          Eff=|( 100.0000 +- 0.00000 )%|

It is clear that the version 1.1 is being configured due to the absence of Ttracks (which would be expected with the default version 1.0)

We have confirmed that the /Event/Rec/Track/BestVelo location is present in the HLT2 input however it should be noted that most events (~80%) have 0 size for this container.

I have a minimal reproducer below that can be run quickly in debug mode with

lb-run Moore/v55r16p4 lbexec sprucingIFT:pass_production --export job_opts sprucingIFT.yaml >& log &

(Moore v55r16p4 is the version we used in production but the same happens with v56r0)

sprucingIFT.py


## Options for concurrent Sprucing of PbPb data in 2024

from Moore import Options, run_moore
from Moore.lines import SpruceLine

from PyConf.reading import upfront_decoder
from Moore.streams import Stream, Streams
from Moore.persistence.hlt2_tistos import list_of_full_stream_lines

###################### SPRUCING ON Ion STREAM ####################

passthrough_reco_lines = [
    "Hlt2PbPbUPC",
]

def make_pass_reco_only_line(hlt2_linename):
    filter = f"{hlt2_linename}Decision"
    pass_linename = hlt2_linename.replace("Hlt2", "Spruce")
    print(f"{filter} {pass_linename}")
    with upfront_decoder.bind(source="Hlt2"):
        pass_line = SpruceLine(
            name=pass_linename,
            hlt2_filter_code=filter,
            algs=[],
            persistreco=True)
    return pass_line


def make_pass_prod_streams():

    streams = [
            Stream(
                "ionpass",
                lines=[make_pass_reco_only_line(line) for line in passthrough_reco_lines])
        ]

    ionstreams=Streams(streams=streams)
    ionstreams.print()
    return ionstreams


def pass_production(options: Options):

    with list_of_full_stream_lines.bind(lines=[]):
        config = run_moore(options, make_pass_prod_streams, public_tools=[])
        return config

sprucingIFT.yaml


input_files: ['/eos/lhcb/wg/dpa/wp1/temp/310067_00130006_0000_iondebug.raw']
input_type: 'RAW'
simulation: False
output_file : 'spruce_IFT.{stream}.dst'
output_type : 'ROOT'
output_manifest_file : "spruce_IFT.tck.json"

process: Spruce
input_raw_format : 0.5
data_type : Upgrade

geometry_version: run3/2024.Q1.2-v00.00
conditions_version: master

first_evt: 1
persistreco_version: 1.1
input_process: Hlt2
evt_max: 20
output_level: 1

Im afraid my persistency knowledge has been defeated after a few days of trying to debug here :( We have never seen issues with persistreco in Sprucing when using version 1.0. Expert persistency help is required 🙏

@mveghel, @graven

(cc @ngrieser, @oboenteg, @sbelin, @sstahl, @cburr)

Edited by Nicole Skidmore