Introduce runners to generalize setup of software for tasks

Runners

Runners generalize the execution of a task. They can perform general setup procedures for tasks that are not necessarily the same between different environments. An example is running each task in a shifter environment.

A runner inherits from the Runner class. It implements two functions: run and terminate. The run function executes the task (passed as an argument). The terminate function stops the currently running task.

The chosen runner can be specified at command line using the -r option. The default is default, which corresponds to a BasicRunner.

Available Runners

BasicRunner

The BasicRunner is the same as the current behaviour. It uses subprocess.Popen to execute the task as a process in the same environment as PyTaskFarmer.

ShifterRunner

The ShifterRunner executes each task inside shifter. This can be preferable over starting PyTaskFarmer inside shifter as it does not require a recent version of Python in the image.

The shifter itself is started using subprocess module with the following command.

shifter --image image -- /bin/bash -c "setup_code && task"

The setup_code is user-configurable set of commands to setup the environment (ie: source ALRB) in shifter.

Defining Runners

The BasicRunner is always available under the name default. Other runners, that require user-specific configuration, should be defined inside the ~/.pytaskfarmer/runners.d directory. All files ending in .ini are loaded and expected to be in the INI format with the following scheme.

[runnername]
Runner = runner.python.class
Arg0 = value0
Arg1 = value1

The runnername is the name of the runner and the Runner value is the name of the python class (along with package and module) containing the implementation. The remaining key:value pairs are passed to the runner.python.class constructor.

There can be multiple runners per file.

Examples

ShifterRunner for Muon Collider Framework

Runner definition:

[mcc]
Runner: taskfarmer.runners.ShifterRunner
image: docker:gitlab-registry.cern.ch/berkeleylab/muoncollider/muoncollider-docker/mucoll-ilc-framework:1.5.1-centos8
setup: source ${HOME}/.bashrc.muc && source ${HOME}/MuonCollider/LBLMuCWorkspace/setup.sh ${CFS}/atlas/kkrizka/MCC/build/

Tasklist contents:

Marlin /global/cfs/cdirs/atlas/kkrizka/MCC/build/packages/ACTSTracking/example/actsseed_steer.xml --global.LCIOInputFiles=muonGun_sim_MuColl_v1_slice0.000.slcio --MyLCParquet.OutputDir=file:///global/cfs/cdirs/atlas/kkrizka/MCC/tracking/muonGun_sim_MuColl_v1_BIB/data_actsseed_bib --MyLCParquet.SampleName=muonGun_sim_MuColl_v1_slice0.000
Marlin /global/cfs/cdirs/atlas/kkrizka/MCC/build/packages/ACTSTracking/example/actsseed_steer.xml --global.LCIOInputFiles=muonGun_sim_MuColl_v1_slice0.001.slcio --MyLCParquet.OutputDir=file:///global/cfs/cdirs/atlas/kkrizka/MCC/tracking/muonGun_sim_MuColl_v1_BIB/data_actsseed_bib --MyLCParquet.SampleName=muonGun_sim_MuColl_v1_slice0.001

Execution:

pytaskfarmer.py tasklist -r mcc

MuonColliderRunner

This runner does not exist (yet), but it demonstrates the flexibility of the runner abstraction.

[mysteeringfile]
Runner: mcc.MuonColliderRunner
steering: /global/cfs/cdirs/atlas/kkrizka/MCC/build/packages/ACTSTracking/example/actsseed_steer.xml

Tasklist (aka filelist) contents:

muonGun_sim_MuColl_v1_slice0.000.slcio
muonGun_sim_MuColl_v1_slice0.001.slcio

Execution:

pytaskfarmer.py filelist -r mysteeringfile

Repository Organization

Added a taskfarmer package that contains all the reusable modules. Currently there is tasks containing the Tasklist class and runners with all modules. This is to help with code organization.

The downside is that now the package has to be installed into PYTHONPATH (ie: using pip install --user) before using.