Skip to content

[master] Add new possibility to throttle tasks in TS

Philippe Charpentier requested to merge charpent/LHCbDIRAC:220110-PhC-NewTransformationParameter into master

BEGINRELEASENOTES

  • TS

NEW: When running large productions or large DMS transformations, The production or data manager has to continuously monitor the status of the transformations and increment e.g. the end run of the data range in order to keep a reasonable number of Waiting jobs or running replication requests. This new feature allows with a new TransformationPlugin parameter ThrottlePendingTasks to limit the number of pending tasks targeting a given storage element (i.e. as source for jobs or as destination for replications or removals).In order to not overload the TS, one should set the Period parameter to a reasonable value. By default if ThrottlePendingTasks is set, there is a minimum period of 6 hours between each running of the plugin but this can be altered using the script below...

  • TS

CHANGE: factorize some of the checks that were done in TransformationPlugin to move them to PluginUtilities to improve readability / maintainability(?)

NEW: new script dirac-transformation-add-parameter allows to add a parameter to an existing transformation

FIX: in several scripts use the getTransformations() method for parsing (list) of transformations passed as arguments

  • DMS

FIX: fix dirac-dms-add-file as no registration error was reported

  • All

FIX: remove many from __future__ import in scripts (not all!)

ENDRELEASENOTES

Edited by Philippe Charpentier

Merge request reports