Store register values in the configuration database

Description

This adds tables to store register values in the configuration database, and "plumbing" commands to import and export whole configurations. Imported values can optionally be deduplicated. The file format is expected to be compatible with the one used by read_configuration_file.

When importing, each "component" is attributed a name based on the file name. The names need to be unique within a configuration.

This MR builds on top of !224 (merged) (database migrations in particular need to be applied in order, I don't want to deal with merging them).

Related Issue

I guess #121

How Has This Been Tested?

⚠ Following these steps will modify your database ⚠

ℹ The following commands assume that you're working from a poetry shell.

As usual, start by migrating your database:

gem-configdb alembic upgrade head

I'm sure you'll want to test this MR with actual files for read_configuration_file that I don't have access to, but first start with the data I used so you have an idea what to expect if it works. So generate some:

gem-configdb generate-test-data test_data_0
gem-configdb generate-test-data test_data_1
gem-configdb generate-test-data test_data_2

Each folder will contain one endcap worth of VFAT data, plus some additional random values. You can inspect the files or generate more yourself, the loader doesn't know what it loads (apart that it loads all .cfg files in a folder).

Importing is easy, if a bit slow (about 20s per set, most of it spent reading files and mashing values in Python):

$ gem-configdb import-data test_data_0
42
$ gem-configdb import-data test_data_1
43
$ gem-configdb import-data test_data_2
44

The numbers are the ids of the newly created configurations.

To export the data, do:

$ gem-configdb export-data 42 empty-folder

Of course, you'll get an error if empty-folder isn't actually empty. You'll also get an error if it exists. So make sure this isn't the case. Oh, and by the way, 42 comes from the loading of test_data_0, you'll get a different number.

I didn't write a tool to compare empty-folder to test_data_0 because it's a pain to do with all those text files. Instead, we can leverage the database by importing empty-folder back, enabling deduplication:

$ gem-configdb import-data empty-folder -r 42
WARNING:root:No difference with respect to the reference configuration
42

As you can see, the loader detected that it could reuse the good old configuration 42 instead of creating a new one, so it did just that. But what if we try to load something else?

$ gem-configdb import-data test_data_1 -r 42
45

You can't see it, but this time most of the data from configuration 42 was reused (about 80%, because each set of generated data contains 10% of random configurations -- this can be configured at generation time with --random-fraction).

Let's finally check that the dedup'ed configuration is identical to the non-dedup'ed one:

$ gem-configdb export-data test_data_1 -r 45
WARNING:root:No difference with respect to the reference configuration
45

To revert the schema to the tip of !224 (merged), do:

gem-configdb alembic downgrade f4adeca56f06

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• My code follows the code style of this project.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I have read the CONTRIBUTING document.
• I have added tests to cover my changes.
• All new and existing tests passed.