Package ai.djl.serving.wlm

Class WorkLoadManager

  • All Implemented Interfaces:
    java.lang.AutoCloseable
    public class WorkLoadManager
extends java.lang.Object
implements java.lang.AutoCloseable
    WorkLoadManager is responsible to manage the work load of worker thread. the manage scales up/down the required amount of worker threads per model.

    • Constructor Detail

      • WorkLoadManager

        public WorkLoadManager()
        Constructs a WorkLoadManager instance.

    • Method Detail

      • getWorkers

        public java.util.List<WorkerThread> getWorkers​(ModelInfo modelInfo)
        Returns the workers for the specific model.
        Parameters:
        modelInfo - the name of the model we are looking for.
        Returns:
        the list of workers responsible to handle predictions for this model.

      • unregisterModel

        public void unregisterModel​(ModelInfo model)
        Removes a model from management.
        Parameters:
        model - the model to remove

      • runJob

        public java.util.concurrent.CompletableFuture<ai.djl.modality.Output> runJob​(Job job)
        Adds an inference job to the job queue of the next free worker. scales up worker if necessary.
        Parameters:
        job - an inference job to be executed.
        Returns:
        true if submit success, false otherwise.

      • getNumRunningWorkers

        public int getNumRunningWorkers​(ModelInfo modelInfo)
        Returns the number of running workers of a model. running workers are workers which are not stopped, in error or scheduled to scale down.
        Parameters:
        modelInfo - the model we are interested in.
        Returns:
        number of running workers.

      • getQueueLength

        public int getQueueLength​(ModelInfo modelInfo)
        Returns the current number of request in the queue.
        Parameters:
        modelInfo - the model
        Returns:
        the current number of request in the queue

      • close

        public void close()
        Specified by:
        close in interface java.lang.AutoCloseable