[Discussion] Config handling API development
This snippet provides a possible implementation of FE config administration on LocalDB, with the following features:
- identification of a "branch" of config revision by
serialNumber
,stage
andbranch
name. - Each
config
document points to the latestconfig_revision
document, which contains the actual config (exceptPixelConfig
).
{
_id: ObjectId("6434ecce31379fc0ad231b26"),
serialNumber: '20UPGXF0000013',
stage: 'MODULE/INITIAL_WARM',
branch: 'default',
current_revision_id: ObjectId("6434ecce31379fc0ad231b2d")
}
- Each
config_revision
object contains:- the actual FE config object,
- "diff" from the previous (parent) revision,
- pointer to
PixelConfig
- user-specified tag list
- message on the commit
- (timestamp)
{
_id: ObjectId("6434ef0fb49d2269a6e3549a"),
parent_revision_id: null,
config_data: {
RD53B: {
GlobalConfig: {
AiRegionRow: 0,
AuroraActiveLanes: 1,
...
VrefIn: 1,
VrefRsensBot: 0,
VrefRsensTop: 0
},
Parameter: {
ADCcalPar: [ 5.894350051879883, 0.1920430064201355, 4990 ],
ChipId: 15,
EnforceNameIdCheck: true,
...
VcalPar: [ 0.46000000834465027, 0.20069999992847443 ]
}
}
},
diff: {
RD53B: {
GlobalConfig: {
AiRegionRow: 0,
...
}
}
},
pix_config: ObjectId("6434ef0fb49d2269a6e35494"),
message: 'commit with ITkPixV1.1Q13_Chip4.json',
tags: []
-
PixelConfig
is not documented asMongoDB
object, but it is stored inGridFS
. Here I'd propose to use pythonpickle
and save data as apython
dictionary object, so that one can easily check identity of the config byhashlib md5
which is a default parameter in eachGridFS
object.
List of API methods (may add more)
Quite analogous to git
commands
-
__init__(hostname, port)
: connect to MongoDB server and setup the client -
create_config(serial_number, stage, branch = 'default')
: create a new config instance. Serial number and stage are mandatory. -
get_info( config_id )
: get the contents ofconfig
object -
info( config_id )
: print the contents ofconfig
object:
config id = 6434b841e9f445e668522512
- Serial Number: 20UPGXF0000013
- Stage: MODULE/INITIAL_WARM
- Branch: default
- HEAD: 6434b842e9f445e668522517
-
copy_config(original_id, serial_number, stage, branch = 'default')
: from anoriginal
config, create a copy of it with a different set of identifiers of(serial_number, stage, branch)
. The difference of this method from thebranch()
method below is that forbranch()
caseserial_number
andstage
are fixed and onlybranch
parameter can be different, so more constrained. -
checkout(serial_number, stage, branch = 'default')
: returns the ID of theconfig
specified by the identifiers. -
branch(parent_id, new_branch, revision_id="HEAD")
: create a copy ofconfig
of the sameserial_number
andstage
with a differentbranch
name. The revision history up to the moment of copying is shared. Returns the ID of the created branch. -
get_config(config_id, revision=None, add_pixel_cfg = False)
: returns the FE config of theconfig
at the specifiedrevision
ID. Whenadd_pixel_cfg=True
, the correspondingPixelConfig
object is inserted into theconfig
fromGridFS
. -
commit(config_id, fe_cfg, message="")
: revise the config with the specifiedfe_cfg
.message
can be put to book-keep the revision history.
new commit: 6434b842e9f445e668522517 --> 20UPGXF0000013 | MODULE/INITIAL_WARM | default
-
_get_prev_commit( revision_id )
: an API-internal method to reconstruct the revision history list. -
get_revision_id(config_id, tag="HEAD")
: method to get the revision ID by a tag, e.g.HEAD
,HEAD^2
or any user-defined tags. -
get_revision_history( config_id )
: get the full revision history as a list of revision IDs, in the descending order of the commit (latest commit is the first). -
get_log( config_id, depth = 10 )
: show revision history of the branch down to the specified depth.
config id = 6434b841e9f445e668522512
- Serial Number: 20UPGXF0000013
- Stage: MODULE/INITIAL_WARM
- Branch: default
- HEAD: 6434b842e9f445e668522517
--------------------------------------
commit 6434b842e9f445e668522517: (HEAD) revision4 on branch default
commit 6434b842e9f445e668522515: another revision
commit 6434b841e9f445e668522514: revision2
commit 6434b841e9f445e668522513: this is the first revision.
--------------------------------------
Supposed use-case
- The YARR scan's "before" config can be queried to MongoDB using this API. Most of the cases it should be just fetching the latest revision of the branch. The API operation can be integrated in
scanConsole
as pre/post processes. - Supporting parallel branches for {cold, warm, LP} and its evolution by scans.
- Revision of configs after YARR scan finishes, or by
module-qc-tools
- Creation of new branches for user's R&D works
- Tools for the downloading/uploading config with production DB is not present in this API (yet).
- Readout
connectivity
is not administrated by this API (yet).
Misc
- Pixel config data size by
python pickle
serialization is around 1.2 MB.
Open points
- At which repository should this API live? Possible clients are
YARR
,module-qc-analysis-tools
andlocaldb-tools
. - Some of YARR's
dbAccessor
features should be deprecated if this API is commissioned.