ai.djl.serving.wlm

Class WorkLoadManager

java.lang.Object

ai.djl.serving.wlm.WorkLoadManager

public class WorkLoadManager
extends java.lang.Object

WorkLoadManager is responsible to manage the work load of worker thread. the manage scales up/down the required amount of worker threads per model.

Constructor Summary

Constructors

Constructor and Description
WorkLoadManager() Constructs a WorkLoadManager instance.

Method Summary

Modifier and Type	Method and Description
int	getNumRunningWorkers(ModelInfo modelInfo) Returns the number of running workers of a model.
int	getQueueLength(ModelInfo modelInfo) Returns the current number of request in the queue.
java.util.List<WorkerThread>	getWorkers(ModelInfo modelInfo) Returns the workers for the specific model.
void	modelChanged(ModelInfo modelInfo) Triggers a model change event.
java.util.concurrent.CompletableFuture<ai.djl.modality.Output>	runJob(Job job) Adds an inference job to the job queue of the next free worker.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

WorkLoadManager

public WorkLoadManager()

Constructs a WorkLoadManager instance.

Method Detail

getWorkers

public java.util.List<WorkerThread> getWorkers(ModelInfo modelInfo)

Returns the workers for the specific model.

Parameters:

modelInfo - the name of the model we are looking for.

Returns:

the list of workers responsible to handle predictions for this model.

runJob

public java.util.concurrent.CompletableFuture<ai.djl.modality.Output> runJob(Job job)

Adds an inference job to the job queue of the next free worker. scales up worker if necessary.

Parameters:

job - an inference job to be executed.

Returns:

true if submit success, false otherwise.

getNumRunningWorkers

public int getNumRunningWorkers(ModelInfo modelInfo)

Returns the number of running workers of a model. running workers are workers which are not stopped, in error or scheduled to scale down.

Parameters:

modelInfo - the model we are interested in.

Returns:

number of running workers.

modelChanged

public void modelChanged(ModelInfo modelInfo)

Triggers a model change event. scales up and down workers to match minWorkers/maxWorkers.

Parameters:

modelInfo - the changed model.

getQueueLength

public int getQueueLength(ModelInfo modelInfo)

Returns the current number of request in the queue.

Parameters:

modelInfo - the model

Returns:

the current number of request in the queue