Optimize TTL for productions that set "TTLOptimizationType" to "TTLStatistics"
This PR introduces the TTL optimization based on the job history execution saved in processes.TTLPredictionHistorySite
.
In order for this PR to work, two new fields should be added to processes.JOBAGENT
:
`productionId` int DEFAULT NULL,
`TTLOptimizationType` varchar(255) COLLATE latin1_general_ci DEFAULT NULL,
This PR:
- parses
"LPMJobTypeID"
and"TTLOptimizationType"
from JDL. If both of them exist, then theprocesses.JOBAGENT
entry for these kinds of jobs will be set. Otherwise, both of them will be null. Only jobs that have these two fields will be optimized. - modifies the job brokering logic from
JobBroker.getMatchJob
:- if no
processes.JOBAGENT
entry matches the job request from the worker node, then tries to optimize the TTL. If nothing matches this request, try finding jobs disregarding theSite
constraint. If nothing matches this request, try optimizing the TTL disregarding theSite
constraint. - if the request from
JobBrokering.getMatchJobTTL
returns entries, then each one is parsed. Based onTTLOptimizationType
, the correspondent TTL optimization method is applied (usingTaskQueueUtils.getEstimatedTTL
) for each entry. If the returned TTL is less than the TTL on the worker node, then we have aprocesses.JOBAGENT
entry id that can be used.
- if no
- modifies the job brokering logic from
JobBroker.getNumberWaitingForSite
:- if the request contains
limit
, then the query will return the firstlimit
entries. The default value is 1. - if the request contains
OptimizeTTL
, then a new constraint (productionId is not null and TTLOptimizationType is not null
) is added to the query - if the query returns more than one entry (i.e.,
limit > 1
), then all the entries are returned in an array namedentries
.
- if the request contains
- modifies
TaskQueueUtils
by adding agetEstimatedTTL
function - modifies
JobHistoryConsumer
to force each consumer to implement agetEstimatedTTL
method that returns the estimated TTL. - modifies
JobHistoryFactory
to implementgetEstimatedTTL
which applies the consumer's optimization method based on theTTLOptimizationType
flag. - modifies
JobHistoryStatistics
to implementgetEstimatedTTL
based on the statistics method (using the data accumulated inprocesses.TTLPredictionHistorySite
)
Signed-off-by: Elena Mihailescu maria.mihailescu@upb.ro