Skip to content

Draft: `chain` job manager

Patrick Louis S Connor requested to merge experimental into master

Goal: have a simple command line to submit several commands in a row for several data sets on HTCondor:

chain mergeNtuples applyMClumi applyPUcleaning applyJEScorrections -c config.info -i Pythia.csv

The config.info is as usual. The new thing is the CSV file, which is a table with rows corresponding to samples and columns to additional information, e.g.

name,location,applyMClumi
PythiaFlat,/pnfs/desy.de/.../QCD_Pt-15to7000_TuneCP5_Flat2018_13TeV_pythia8/cd1475d24c5fc_darwin21_RunIISummer20UL18MiniAODv2-106X_upgrade2018_realistic_v16_L1v1-v2/,1373000000

This creates and submit a DAGMan (logs are in the hidden directory .dag).

  • basic implementation
  • forks (e.g. chain applyX applyY+getA+getB applyZ+getA+getB ...)
  • option auto-removal of intermediate steps with SCRIPT POST
  • try each job before submitting with SCRIPT PRE
  • copy executables on the fly with SCRIPT PRE
  • skip first few jobs whose output already exists if applicable
Edited by Patrick Louis S Connor

Merge request reports