Use rawbank views for sprucing
-
Review changes -
-
Download -
Patches
-
Plain diff
To go with DaVinci!618 (merged), Moore!1137 (merged), MooreAnalysis!65 (merged)
Make Sprucing use RawBank::View to access persistreco objects rawbanks from previous passes
Motivation
This is a move towards the extensive use of "views" which will allow the Sprucing AND HLT2 to customise which rawbanks get persisted in a NEW RawEvent without mutilating the input RawEvent by using eg. BankKillers
Sprucing will now have a new RawEvent "Event/{stream}"
where "stream"=default if none given and will not rely on the "DAQ/RawEvent"
This functionality does not exist for HLT2 yet either, customising which raw banks are output only appears to work when you use Brunel MC input with input_raw_format = 4.3
. Running HLT1->HLT2 where input_raw_format = 0.3
means everything is in /DAQ/RawEvent
and you cannot customise which (detector for instance) rawbanks are persisted
https://gitlab.cern.ch/lhcb-dpa/project/-/issues/124
Whats changed
Hlt/HltDAQ/src/component/HltPackedDataDecoder.cpp
- Hardcoded to take View from "/Event/DAQ/RawBanks/DstData" achieved through
LHCb__UnpackRawEvent
- this is temporary!
Hlt/HltDAQ/src/component/HltPackedDataWriter.cpp
- Now uses
MergingTransformer
such that a "OutputRawEvent" and "OutputView" are accessible - Had to change the way 2 errors are reported
- Had to remove
counter( containerPath ) += objectSize;
as could not getcounter
to work - just a DEBUG?
DAQ/DAQUtils/src/CombineRawBankViewsToRawEvent.cpp
- Combines vector of RawBank::View and returns new RawEvent. For sprucing this is the new
DstData
, new SprucingHltDecReports
and any raw banks from previous passes that are requested to be persisted. For passthrough a newDstData
bank is not created and the one from the HLT2 step is persisted.
GaudiConf/python/GaudiConf/reading.py
Complete overhaul using process
which is
- 'Turbo' (HLT2 that has been through Sprucing passthrough),
- 'Spruce'
- 'Hlt2' - at some point 'Hlt2' will not be required as an option downstream as all streams will go through Sprucing so it would be 'Turbo'.
Driving the reading based on process
removes complications of raw event locations for analysts. The stream
argument must also be handed to these functions as 'Turbo' and 'Spruce' RawEvents are located at "Event/{stream}"
.
- Added
unpack_rawevent
function which callsLHCb__UnpackRawEvent
to unpack RawEvent to Views. Location of RawEvent is handled byprocess
argument. By default only "DstData" and "HltDecReports" banks are unpacked which is sufficient for quick file reading -
unpackers
is now driven byprocess
argument which removes complications of TES ROOT for analysts - Although the
DstData
view is hardcoded now inHltPackedDataDecoder.cpp
(which is temporary),HltPackedDataDecoder.cpp
still needs access to theHltDecReports
in the RawEvent. For Sprucing/passthrough this is"Event/{stream}"
but for HLT2 remains as"DAQ/RawEvent"
. This is handled behind the scenes through theprocess
argument indecoder
- Added a
hlt2_decisions
function driven byprocess
argument - Added a
spruce_decisions
function that handles the fact that the sprucingHltDecReports
is now in"Event/{stream}"
-
mc_unpackers
will throw an error ifprocess!=Hlt2
as not ready for sprucing yet.
Hlt/HltDAQ/src/component/HltDecReportsWriter.cpp
- Now gives both a "OutputRawEvent" and a new "OutputView"
- Note the check "// delete any previously inserted dec reports" has been removed as a result
DAQ/DAQUtils/src/BackwardsCompatibleMergeViewIntoRawEvent.cpp
- Needed to copy vector of RawBank::View into existing RawEvent ie.
"DAQ/RawEvent"
(see comment onreading.py
) Temporary measure needed if Writers only output Views
One can use the following to inspect a DST now utilising the improved functions in reading.py
Click to expand
##Usage, inside Moore/
#./run python -i DSTexplore.py -input xxx.[dst, mdf] -tck yyy.tck.json -process process -stream stream
import sys, GaudiPython as GP
from GaudiConf import IOExtension
from GaudiConf.reading import (decoder, unpackers, unpack_rawevent,
hlt2_decisions, spruce_decisions)
from Configurables import ApplicationMgr, LHCbApp
LHCb = GP.gbl.LHCb
import argparse
#Argument parser
parser = argparse.ArgumentParser(
usage="./run python -i %(prog)s xxx.[dst, mdf] yyy.tck.json process stream",
description='Inspect Moore output')
parser.add_argument('-input', type=str, help='Input MDF or DST')
parser.add_argument('-tck', type=str, help='.tck.json file from the job')
parser.add_argument(
'-process',
type=str,
help=
'process can be Spruce or Turbo (or Hlt2). Note Hlt2 option will be removed as all streams will go through Sprucing'
)
parser.add_argument(
'-stream',
type=str,
nargs='?',
default="default",
help='Stream to test, default if no value given')
args = parser.parse_args()
# Helper function to select events with given line decision
def advance_HLT(decision):
"""Return event with given line decision."""
while True:
appMgr.run(1)
if not evt['/Event']:
sys.exit("Did not find positive {0} decision".format(decision))
if args.process == 'Hlt2':
loc = '/Event/Hlt/DecReports'
else:
loc = '/Event/' + args.stream
reports = evt[loc]
report = reports.decReport('{0}Decision'.format(decision))
if report.decision() == 1:
break
return
##Helper functions for returning routing bits
def routing_bits():
"""Return a list with the 96 routing bit values."""
if args.process == 'Hlt2':
loc = '/Event/DAQ/RawEvent'
else:
loc = '/Event/' + args.stream
rawevent = evt[loc]
rbbanks = rawevent.banks(LHCb.RawBank.HltRoutingBits)
on_bits = []
for bank in range(0, len(rbbanks)):
d = rbbanks[bank].data()
bits = "{:032b}{:032b}{:032b}".format(d[2], d[1], d[0])
ordered = list(map(int, reversed(bits)))
for i in range(0, len(ordered)):
if ordered[i] == 1:
on_bits.append(str(i))
return on_bits
# Change the tags if required
LHCbApp(
DataType="Upgrade",
Simulation=True,
DDDBtag="dddb-20201211",
CondDBtag="sim-20201218-vc-md100",
)
assert args.process == "Spruce" or args.process == "Turbo" or args.process == "Hlt2", 'process is Turbo or Spruce (or Hlt2). Note Hlt2 option will be removed as all streams will go through Sprucing'
# Setup HltANNSvc
if args.process == 'Hlt2':
from Moore.tcks import load_hlt2_configuration as load_tck
else:
from Moore.tcks import load_sprucing_configuration as load_tck
ann = load_tck(args.tck, annsvc_name="HltANNSvc")
inputFiles = [args.input]
IOExtension().inputFiles((inputFiles), clear=True)
# Unpack the raw event to give RawBank::Views.
# Unpacks 'DstData' and 'HltDecReports' to RawBank::Views by default
# which are input to `decoder` (which calls `HltPackedDataDecoder`)
# and `HltDecReportsDecoder`
algs = [
unpack_rawevent(process=args.process, stream=args.stream, OutputLevel=4)
]
# Decoder uses HltANNSvc to convert integers back to the packed TES locations according to tck
algs += [decoder(process=args.process, stream=args.stream, OutputLevel=4)]
# HltDecReports decoders uses HltANNSvc to decode integers back to the line decision names according to tck
algs += [
hlt2_decisions(process=args.process, stream=args.stream, OutputLevel=4)
]
if args.process == 'Spruce' or args.process == 'Turbo':
algs += [
spruce_decisions(
process=args.process, stream=args.stream, OutputLevel=4)
]
# Unpack TES locations
algs += unpackers(process=args.process, OutputLevel=4)
ApplicationMgr(TopAlg=algs)
appMgr = GP.AppMgr()
evt = appMgr.evtsvc()
appMgr.run(1)
evt.dump()
Merge request reports
- version 32d8a3c38e
- version 317c047532
- version 30f23b8b39
- version 29329e065e
- version 28f04326f5
- version 27c43c4cb7
- version 26d80b08be
- version 252285a7c5
- version 245a32c643
- version 23adf4d25a
- version 2289c47583
- version 214313a8c5
- version 200b4c4d7c
- version 199724f63f
- version 184cef7338
- version 17a3e5a14f
- version 167e071b45
- version 152eccf2f1
- version 140814cb4c
- version 13c8b89b0a
- version 12321d0bb3
- version 11508a6dea
- version 109714af99
- version 969159d67
- version 873afd26b
- version 7c71c5644
- version 62a3ec577
- version 55756de2c
- version 4809f785c
- version 3160a8759
- version 27f564bdd
- version 1946741e3
- master (base)
- latest versionee07c4f830 commits,
- version 32d8a3c38e29 commits,
- version 317c04753228 commits,
- version 30f23b8b3927 commits,
- version 29329e065e26 commits,
- version 28f04326f525 commits,
- version 27c43c4cb724 commits,
- version 26d80b08be23 commits,
- version 252285a7c522 commits,
- version 245a32c64321 commits,
- version 23adf4d25a20 commits,
- version 2289c4758319 commits,
- version 214313a8c518 commits,
- version 200b4c4d7c17 commits,
- version 199724f63f17 commits,
- version 184cef733816 commits,
- version 17a3e5a14f16 commits,
- version 167e071b4515 commits,
- version 152eccf2f114 commits,
- version 140814cb4c13 commits,
- version 13c8b89b0a12 commits,
- version 12321d0bb311 commits,
- version 11508a6dea10 commits,
- version 109714af999 commits,
- version 969159d678 commits,
- version 873afd26b7 commits,
- version 7c71c56447 commits,
- version 62a3ec5776 commits,
- version 55756de2c5 commits,
- version 4809f785c4 commits,
- version 3160a87593 commits,
- version 27f564bdd2 commits,
- version 1946741e31 commit,
- Side-by-side
- Inline