Skip to content

Draft: Grid submit cleanup

Dan Guestrequested to merge
dguest/training-dataset-dumper:gridup into main

I thought using bulk grid submission (see #147) would be a good excuse to clean up some of the uglier bits of the grid submit code. The idea is to pass off most of the input and output dataset naming stuff to a python script that generates a json file, which we then use with prun --inOutDsJson. Turns out this grid submit script has grown a bit complicated over the years though, so it has a few loose ends.

Main objectives:

  • Move the logic that names the output dataset into a python script
  • Support the -n option by building a list of files in each dataset, and teaching the python script to use that to build the input dataset
  • Support the --no-meta flag by putting the dataset blacklist into the job configuration file and passing the job the input dataset name

Merge request reports