Introduce runners to generalize setup of software for tasks
Runners
Runners generalize the execution of a task. They can perform general setup procedures for tasks that are not necessarily the same between different environments. An example is running each task in a shifter environment.
A runner inherits from the Runner
class. It implements two functions: run
and terminate
. The run
function executes the task (passed as an argument). The terminate
function stops the currently running task.
The chosen runner can be specified at command line using the -r
option. The default is default
, which corresponds to a BasicRunner
.
Available Runners
BasicRunner
The BasicRunner
is the same as the current behaviour. It uses subprocess.Popen
to execute the task as a process in the same environment as PyTaskFarmer.
ShifterRunner
The ShifterRunner
executes each task inside shifter. This can be preferable over starting PyTaskFarmer inside shifter as it does not require a recent version of Python in the image.
The shifter itself is started using subprocess
module with the following command.
shifter --image image -- /bin/bash -c "setup_code && task"
The setup_code
is user-configurable set of commands to setup the environment (ie: source ALRB) in shifter.
Defining Runners
The BasicRunner
is always available under the name default
. Other runners, that require user-specific configuration, should be defined inside the ~/.pytaskfarmer/runners.d
directory. All files ending in .ini
are loaded and expected to be in the INI format with the following scheme.
[runnername]
Runner = runner.python.class
Arg0 = value0
Arg1 = value1
The runnername
is the name of the runner and the Runner
value is the name of the python class (along with package and module) containing the implementation. The remaining key:value pairs are passed to the runner.python.class
constructor.
There can be multiple runners per file.
Examples
ShifterRunner for Muon Collider Framework
Runner definition:
[mcc]
Runner: taskfarmer.runners.ShifterRunner
image: docker:gitlab-registry.cern.ch/berkeleylab/muoncollider/muoncollider-docker/mucoll-ilc-framework:1.5.1-centos8
setup: source ${HOME}/.bashrc.muc && source ${HOME}/MuonCollider/LBLMuCWorkspace/setup.sh ${CFS}/atlas/kkrizka/MCC/build/
Tasklist contents:
Marlin /global/cfs/cdirs/atlas/kkrizka/MCC/build/packages/ACTSTracking/example/actsseed_steer.xml --global.LCIOInputFiles=muonGun_sim_MuColl_v1_slice0.000.slcio --MyLCParquet.OutputDir=file:///global/cfs/cdirs/atlas/kkrizka/MCC/tracking/muonGun_sim_MuColl_v1_BIB/data_actsseed_bib --MyLCParquet.SampleName=muonGun_sim_MuColl_v1_slice0.000
Marlin /global/cfs/cdirs/atlas/kkrizka/MCC/build/packages/ACTSTracking/example/actsseed_steer.xml --global.LCIOInputFiles=muonGun_sim_MuColl_v1_slice0.001.slcio --MyLCParquet.OutputDir=file:///global/cfs/cdirs/atlas/kkrizka/MCC/tracking/muonGun_sim_MuColl_v1_BIB/data_actsseed_bib --MyLCParquet.SampleName=muonGun_sim_MuColl_v1_slice0.001
Execution:
pytaskfarmer.py tasklist -r mcc
MuonColliderRunner
This runner does not exist (yet), but it demonstrates the flexibility of the runner abstraction.
[mysteeringfile]
Runner: mcc.MuonColliderRunner
steering: /global/cfs/cdirs/atlas/kkrizka/MCC/build/packages/ACTSTracking/example/actsseed_steer.xml
Tasklist (aka filelist) contents:
muonGun_sim_MuColl_v1_slice0.000.slcio
muonGun_sim_MuColl_v1_slice0.001.slcio
Execution:
pytaskfarmer.py filelist -r mysteeringfile
Repository Organization
Added a taskfarmer
package that contains all the reusable modules. Currently there is tasks
containing the Tasklist
class and runners
with all modules. This is to help with code organization.
The downside is that now the package has to be installed into PYTHONPATH
(ie: using pip install --user
) before using.