Extending definition of data quality in the bookkeeping
This is in follow up to the discussion we had on Friday, hopefully this matches with what everyone had in mind. Once everybody agrees, I'll assign this to @azhelezo to start work on it. This is only for the data quality part, for the SMOG2 part see #23 (closed).
cc @samarian @fpolci @mstahl @clemenci @sklaver
Requirements:
- We will continue to use
DQFlag=OK
as the way of declaring if data is "good for physics". - If a run is
DQFlag=BAD
this means the run is only of use for detector debugging and the bookkeeping doesn't need to worry about enhanced metadata.- Subsystem experts can look at specific run ranges as needed for their own studies based on the contents of the Problem DB outside of DIRAC.
- For each year we decide what
DQFlag=OK
means so we have a consistent definition.- This will almost always mean all subsystems except
PLUME
andSMOG2
are okay. - As a special case in 2024, we will likely decide
DQFlag=OK
doesn't mean theUT
is okay. - This needs to be documented somewhere. @fpolci this shouldn't be LHCbDIRAC's responsibility.
- Choosing to add new values will require this to be communicated to the LHCbDIRAC team a couple of weeks in advance.
- This will almost always mean all subsystems except
- We will add an
ExtendedDQOK
flag in addition toDQFlag
which can contain a list of systems which need to be okay in addition to the default definition ofDQFlag=OK
. - When running central productions (e.g. sprucing)
ExtendedDQOK
will be ignored. - By default users will get data from a bookkeeping path regardless of ExtendedDQOK state, i.e.
PLUME
,SMOG2
and, in 2024,UT
could be eitherOK
orBAD
. - Users interested in subdetectors in addition to the default set can refine their bookkeeping queries with something like
ExtendedDQOK=UT
orExtendedDQOK=SMOG2,UT
in a similar way to which users can currently specifyRunNumber
andDQFlag
.
Technical details
-
Add a table ( ExtendedDQOK
) to the bookkeeping database with the column(ID,RunNumber,SystemName)
- The primary key can NOT be
RunNumber
as there may be multiple values per run.
- The primary key can NOT be
-
Add a method to the bookkeeping client to get the list of runs for a given state getExtendedDQOK(runs=[], systems=[])
-
Adapt the monet to collect this information and register it in the bookkeeping when marking a run as DQFlag=OK
- There should be a value in the CS which limits the list
-
Adapt the BookkeepingWatchAgent
to respect theExtendedDQOK
in the input data query -
Adapt the Production Request YAML schema to accept ExtendedDQOK
-
Adapt the Analysis Productions YAML schema to accept ExtendedDQOK