Skip to content

Uniform treatment of Algorithm properties in PyConf

In PyConf.components we inspect all properties an Algorithm is configured with. If we determine the property value is DataHandle-like, we assume the property defines input, and if the property value is Tool-like the property defines tool handles. With this information, we can then instantiate the set of underlying Configurable instances to ensure that the algorithm's data dependencies will be met (by creating the producing algorithms) and the tools will be available (by creating the tools).

There are many workarounds in PyConf.components for the cases where a property value is a list of DataHandle or Tool. We would need to add similar workarounds to support other containing structures, such as dict.

In Moore#51 (closed), the issue of 'input-providing properties' came up. These are properties which are not DataHandle directly, but somehow 'contain' data which is implicit input to an algorithm. In a sense, these properties are just like any other container (list, dict), in that the Algorithm must make sure to extract any 'special' object from within them. These special objects are:

  1. DataHandle objects, in which case we need to configure their producers.
  2. Tool objects, in which case we need to configure the tool.
    • More generally this means 'anything with an equivalent Configurable', such as services.

We should unify how we treat property values. Each property value should have all data and configurable objects extracted from it, and the Algorithm should then store these in a uniform way (e.g. a flattened list).


One way to think about this is to not consider Tool as something special. It can:

  1. Inject data dependencies to an Algorithm.
  2. Require an underlying Configurable to be instantiated.

We can see how these requirement affect the behaviour of Algorithm, and use that to define the interfaces all other things can adopt (e.g. functors). Point 1 could mean moving property 'inputs' into direct algorithm inputs, and point 2 could mean defining a .configuration() interface which returns a dataflow_config object.

The downside of unification is that sometimes one does want to treat things specially, e.g. to retrieve the list of all Tool objects held by an Algorithm.

All that said, this issue represents a low-priority refactoring task. It has the potential to make PyConf more understandable with more clearly defined boundaries, but we can always add a few more hacks to support additional property types.

/cc @rmatev

To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information