Skip to content

PrunDriver: enable adding additionnal files to the TarBall sent to the grid thanks new option `optGridPrunShipAdditionalFilesOrDirs`

Romain Bouquet requested to merge (removed):bouquet_21.2_PrunDriverCopyAdditionalFilesOrDir into 21.2

Hi @krumnack, @akraszna

To include additional files to be sent to grid (e.g. some custom calibration files) with the prun command, the desired files (or directories) needs to be in the job submission directory (and cannot be somewhere else).
For PrunDriver the submission occurs in jobELGDir = data.submitDir + "/elg"

Hence to be able to send additional files or directories to grid they need to be in the jobELGDir.

This MR enables to do so by creating symbolic links of files (or directories) that the user wants to include. It is done thanks to the use of a new flag
EL::Job::optGridPrunShipAdditionalFilesOrDirs
which should contain the list of path to be included separated with comma

NB: for heavy files one needs also to ask for prun not to ignore them by using EL::Job::optUserFiles and providing only the name of files (not the path), also comma separated.

E.g. for our group we want to send a custom CDI file (stored on EOS) we need to call in our submission script

// for symbolic link creation in the submission directory 
driver->options()->setString(
  EL::Job::optGridPrunShipAdditionalFilesOrDirs, 
  "/eos/atlas/atlascerngroupdisk/phys-higgs/HSG5/Run2/VH/bTaggingCDI/2020-21-13TeV-MC16-CDI_smooth_VHLegacy-07Oct-v02.root") ;

// for prun to not skip that heavy CDI file in the submission directory  
driver->options()->setString(
  EL::Job::optUserFiles,
  "2020-21-13TeV-MC16-CDI_smooth_VHLegacy-07Oct-v02.root");

I tested that the CDI file is correctly copied in the tar.gz file sent to the grid.

Tagging for information @hanar, @eschopf and @yama

Edited by Romain Bouquet

Merge request reports