Use rawbank views for sprucing

To go with DaVinci!618 (merged), Moore!1137 (merged), MooreAnalysis!65 (merged)

Make Sprucing use RawBank::View to access persistreco objects rawbanks from previous passes

Motivation

This is a move towards the extensive use of "views" which will allow the Sprucing AND HLT2 to customise which rawbanks get persisted in a NEW RawEvent without mutilating the input RawEvent by using eg. BankKillers

Sprucing will now have a new RawEvent "Event/{stream}" where "stream"=default if none given and will not rely on the "DAQ/RawEvent"

This functionality does not exist for HLT2 yet either, customising which raw banks are output only appears to work when you use Brunel MC input with input_raw_format = 4.3. Running HLT1->HLT2 where input_raw_format = 0.3 means everything is in /DAQ/RawEvent and you cannot customise which (detector for instance) rawbanks are persisted

https://gitlab.cern.ch/lhcb-dpa/project/-/issues/124

Whats changed

Hlt/HltDAQ/src/component/HltPackedDataDecoder.cpp

• Hardcoded to take View from "/Event/DAQ/RawBanks/DstData" achieved through LHCb__UnpackRawEvent - this is temporary!

Hlt/HltDAQ/src/component/HltPackedDataWriter.cpp

• Now uses MergingTransformer such that a "OutputRawEvent" and "OutputView" are accessible
• Had to change the way 2 errors are reported
• Had to remove counter( containerPath ) += objectSize; as could not get counter to work - just a DEBUG?

DAQ/DAQUtils/src/CombineRawBankViewsToRawEvent.cpp

• Combines vector of RawBank::View and returns new RawEvent. For sprucing this is the new DstData, new Sprucing HltDecReports and any raw banks from previous passes that are requested to be persisted. For passthrough a new DstData bank is not created and the one from the HLT2 step is persisted.

GaudiConf/python/GaudiConf/reading.py

Complete overhaul using process which is

• 'Turbo' (HLT2 that has been through Sprucing passthrough),
• 'Spruce'
• 'Hlt2' - at some point 'Hlt2' will not be required as an option downstream as all streams will go through Sprucing so it would be 'Turbo'.

Driving the reading based on process removes complications of raw event locations for analysts. The stream argument must also be handed to these functions as 'Turbo' and 'Spruce' RawEvents are located at "Event/{stream}".

• Added unpack_rawevent function which calls LHCb__UnpackRawEvent to unpack RawEvent to Views. Location of RawEvent is handled by process argument. By default only "DstData" and "HltDecReports" banks are unpacked which is sufficient for quick file reading
• unpackers is now driven by process argument which removes complications of TES ROOT for analysts
• Although the DstData view is hardcoded now in HltPackedDataDecoder.cpp (which is temporary), HltPackedDataDecoder.cpp still needs access to the HltDecReports in the RawEvent. For Sprucing/passthrough this is "Event/{stream}" but for HLT2 remains as "DAQ/RawEvent". This is handled behind the scenes through the process argument in decoder
• Added a hlt2_decisions function driven by process argument
• Added a spruce_decisions function that handles the fact that the sprucing HltDecReports is now in "Event/{stream}"
• mc_unpackers will throw an error if process!=Hlt2 as not ready for sprucing yet.

Hlt/HltDAQ/src/component/HltDecReportsWriter.cpp

• Now gives both a "OutputRawEvent" and a new "OutputView"
• Note the check "// delete any previously inserted dec reports" has been removed as a result

DAQ/DAQUtils/src/BackwardsCompatibleMergeViewIntoRawEvent.cpp

• Needed to copy vector of RawBank::View into existing RawEvent ie. "DAQ/RawEvent" (see comment on reading.py) Temporary measure needed if Writers only output Views

cc @graven, @sesen

One can use the following to inspect a DST now utilising the improved functions in reading.py

Click to expand

##Usage, inside Moore/
#./run python -i DSTexplore.py -input xxx.[dst, mdf] -tck yyy.tck.json -process process -stream stream

import sys, GaudiPython as GP
from GaudiConf import IOExtension
from GaudiConf.reading import (decoder, unpackers, unpack_rawevent,
                               hlt2_decisions, spruce_decisions)
from Configurables import ApplicationMgr, LHCbApp
LHCb = GP.gbl.LHCb
import argparse

#Argument parser
parser = argparse.ArgumentParser(
    usage="./run python -i %(prog)s xxx.[dst, mdf] yyy.tck.json process stream",
    description='Inspect Moore output')
parser.add_argument('-input', type=str, help='Input MDF or DST')
parser.add_argument('-tck', type=str, help='.tck.json file from the job')
parser.add_argument(
    '-process',
    type=str,
    help=
    'process can be Spruce or Turbo (or Hlt2). Note Hlt2 option will be removed as all streams will go through Sprucing'
)
parser.add_argument(
    '-stream',
    type=str,
    nargs='?',
    default="default",
    help='Stream to test, default if no value given')
args = parser.parse_args()


# Helper function to select events with given line decision
def advance_HLT(decision):
    """Return event with given line decision."""
    while True:
        appMgr.run(1)
        if not evt['/Event']:
            sys.exit("Did not find positive {0} decision".format(decision))
        if args.process == 'Hlt2':
            loc = '/Event/Hlt/DecReports'
        else:
            loc = '/Event/' + args.stream
        reports = evt[loc]
        report = reports.decReport('{0}Decision'.format(decision))
        if report.decision() == 1:
            break

    return


##Helper functions for returning routing bits
def routing_bits():
    """Return a list with the 96 routing bit values."""

    if args.process == 'Hlt2':
        loc = '/Event/DAQ/RawEvent'
    else:
        loc = '/Event/' + args.stream
    rawevent = evt[loc]
    rbbanks = rawevent.banks(LHCb.RawBank.HltRoutingBits)
    on_bits = []
    for bank in range(0, len(rbbanks)):
        d = rbbanks[bank].data()
        bits = "{:032b}{:032b}{:032b}".format(d[2], d[1], d[0])
        ordered = list(map(int, reversed(bits)))
        for i in range(0, len(ordered)):
            if ordered[i] == 1:
                on_bits.append(str(i))

    return on_bits


# Change the tags if required
LHCbApp(
    DataType="Upgrade",
    Simulation=True,
    DDDBtag="dddb-20201211",
    CondDBtag="sim-20201218-vc-md100",
)

assert args.process == "Spruce" or args.process == "Turbo" or args.process == "Hlt2", 'process is Turbo or Spruce (or Hlt2). Note Hlt2 option will be removed as all streams will go through Sprucing'

# Setup HltANNSvc
if args.process == 'Hlt2':
    from Moore.tcks import load_hlt2_configuration as load_tck
else:
    from Moore.tcks import load_sprucing_configuration as load_tck

ann = load_tck(args.tck, annsvc_name="HltANNSvc")

inputFiles = [args.input]
IOExtension().inputFiles((inputFiles), clear=True)

# Unpack the raw event to give RawBank::Views.
# Unpacks 'DstData' and 'HltDecReports' to RawBank::Views by default
# which are input to `decoder` (which calls `HltPackedDataDecoder`)
# and `HltDecReportsDecoder`
algs = [
    unpack_rawevent(process=args.process, stream=args.stream, OutputLevel=4)
]

# Decoder uses HltANNSvc to convert integers back to the packed TES locations according to tck
algs += [decoder(process=args.process, stream=args.stream, OutputLevel=4)]

# HltDecReports decoders uses HltANNSvc to decode integers back to the line decision names according to tck
algs += [
    hlt2_decisions(process=args.process, stream=args.stream, OutputLevel=4)
]

if args.process == 'Spruce' or args.process == 'Turbo':
    algs += [
        spruce_decisions(
            process=args.process, stream=args.stream, OutputLevel=4)
    ]

# Unpack TES locations
algs += unpackers(process=args.process, OutputLevel=4)

ApplicationMgr(TopAlg=algs)

appMgr = GP.AppMgr()
evt = appMgr.evtsvc()

appMgr.run(1)
evt.dump()