Skip to content
GitLab
Projects Groups Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
  • Sign in
  • L LHCb
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
    • Locked Files
  • Issues 197
    • Issues 197
    • List
    • Boards
    • Service Desk
    • Milestones
    • Iterations
    • Requirements
  • Jira
    • Jira
  • Merge requests 50
    • Merge requests 50
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
    • Test Cases
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Code review
    • Insights
    • Issue
    • Repository
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • LHCbLHCb
  • LHCb
  • Issues
  • #112
Closed
Open
Issue created Jan 28, 2021 by Alex Pearce@apearce🌈Contributor

Uniform treatment of Algorithm properties in PyConf

In PyConf.components we inspect all properties an Algorithm is configured with. If we determine the property value is DataHandle-like, we assume the property defines input, and if the property value is Tool-like the property defines tool handles. With this information, we can then instantiate the set of underlying Configurable instances to ensure that the algorithm's data dependencies will be met (by creating the producing algorithms) and the tools will be available (by creating the tools).

There are many workarounds in PyConf.components for the cases where a property value is a list of DataHandle or Tool. We would need to add similar workarounds to support other containing structures, such as dict.

In Moore#51 (closed), the issue of 'input-providing properties' came up. These are properties which are not DataHandle directly, but somehow 'contain' data which is implicit input to an algorithm. In a sense, these properties are just like any other container (list, dict), in that the Algorithm must make sure to extract any 'special' object from within them. These special objects are:

  1. DataHandle objects, in which case we need to configure their producers.
  2. Tool objects, in which case we need to configure the tool.
    • More generally this means 'anything with an equivalent Configurable', such as services.

We should unify how we treat property values. Each property value should have all data and configurable objects extracted from it, and the Algorithm should then store these in a uniform way (e.g. a flattened list).


One way to think about this is to not consider Tool as something special. It can:

  1. Inject data dependencies to an Algorithm.
  2. Require an underlying Configurable to be instantiated.

We can see how these requirement affect the behaviour of Algorithm, and use that to define the interfaces all other things can adopt (e.g. functors). Point 1 could mean moving property 'inputs' into direct algorithm inputs, and point 2 could mean defining a .configuration() interface which returns a dataflow_config object.

The downside of unification is that sometimes one does want to treat things specially, e.g. to retrieve the list of all Tool objects held by an Algorithm.

All that said, this issue represents a low-priority refactoring task. It has the potential to make PyConf more understandable with more clearly defined boundaries, but we can always add a few more hacks to support additional property types.

/cc @rmatev

To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information
Assignee
Assign to
Time tracking