Skip to content

Optimize TTL for productions that set "TTLOptimizationType" to "TTLStatistics"

This PR introduces the TTL optimization based on the job history execution saved in processes.TTLPredictionHistorySite.

In order for this PR to work, two new fields should be added to processes.JOBAGENT:

`productionId` int DEFAULT NULL,
`TTLOptimizationType` varchar(255) COLLATE latin1_general_ci DEFAULT NULL,

This PR:

  • parses "LPMJobTypeID" and "TTLOptimizationType" from JDL. If both of them exist, then the processes.JOBAGENT entry for these kinds of jobs will be set. Otherwise, both of them will be null. Only jobs that have these two fields will be optimized.
  • modifies the job brokering logic from JobBroker.getMatchJob:
    • if no processes.JOBAGENT entry matches the job request from the worker node, then tries to optimize the TTL. If nothing matches this request, try finding jobs disregarding the Site constraint. If nothing matches this request, try optimizing the TTL disregarding the Site constraint.
    • if the request from JobBrokering.getMatchJobTTL returns entries, then each one is parsed. Based on TTLOptimizationType, the correspondent TTL optimization method is applied (using TaskQueueUtils.getEstimatedTTL) for each entry. If the returned TTL is less than the TTL on the worker node, then we have a processes.JOBAGENT entry id that can be used.
  • modifies the job brokering logic from JobBroker.getNumberWaitingForSite:
    • if the request contains limit, then the query will return the first limit entries. The default value is 1.
    • if the request contains OptimizeTTL, then a new constraint (productionId is not null and TTLOptimizationType is not null) is added to the query
    • if the query returns more than one entry (i.e., limit > 1), then all the entries are returned in an array named entries.
  • modifies TaskQueueUtils by adding a getEstimatedTTL function
  • modifies JobHistoryConsumer to force each consumer to implement a getEstimatedTTL method that returns the estimated TTL.
  • modifies JobHistoryFactory to implement getEstimatedTTL which applies the consumer's optimization method based on the TTLOptimizationType flag.
  • modifies JobHistoryStatistics to implement getEstimatedTTL based on the statistics method (using the data accumulated in processes.TTLPredictionHistorySite)

Signed-off-by: Elena Mihailescu maria.mihailescu@upb.ro

Merge request reports

Loading