Package ai.djl.serving.wlm

Class WorkLoadManager

java.lang.Object
ai.djl.serving.wlm.WorkLoadManager
public class WorkLoadManager extends Object
WorkLoadManager is responsible to manage the work load of worker thread. the manage scales up/down the required amount of worker threads per wpc.

  • Constructor Details

    • WorkLoadManager

      public WorkLoadManager()
      Constructs a WorkLoadManager instance.

  • Method Details

    • registerWorkerPool

      public <I, O> WorkerPool<I,O> registerWorkerPool(WorkerPoolConfig<I,O> wpc)
      Registers a WorkerPool (model).

      This operation is idempotent and will return the existing workerpool if the wpc was already registered.

      Type Parameters:
      I - the wpc input class
      O - the wpc output class
      Parameters:
      wpc - the wpc to create the worker pool for
      Returns:
      the WorkerPool

    • unregisterWorkerPool

      public void unregisterWorkerPool(WorkerPoolConfig<?,?> wpc)
      Removes a worker pool from management.
      Parameters:
      wpc - the wpc to remove

    • runJob

      public <I, O> CompletableFuture<O> runJob(Job<I,O> job)
      Adds an inference job to the job queue of the next free worker. scales up worker if necessary.
      Type Parameters:
      I - the wpc input class
      O - the wpc output class
      Parameters:
      job - an inference job to be executed.
      Returns:
      true if submit success, false otherwise.

    • getNumRunningWorkers

      public int getNumRunningWorkers(WorkerPoolConfig<?,?> wpc)
      Returns the number of running workers of a wpc. running workers are workers which are not stopped, in error or scheduled to scale down.
      Parameters:
      wpc - the wpc we are interested in.
      Returns:
      number of running workers.

    • getWorkerPoolById

      public <I, O> WorkerPool<I,O> getWorkerPoolById(String id)
      Returns the WorkerPool for a wpc.
      Type Parameters:
      I - the wpc class
      O - the wpc class
      Parameters:
      id - the wpc id
      Returns:
      the WorkerPool

    • getWorkerPool

      public <I, O> WorkerPool<I,O> getWorkerPool(WorkerPoolConfig<I,O> wpc)
      Returns the WorkerPool for a model.
      Type Parameters:
      I - the wpc input class
      O - the wpc output class
      Parameters:
      wpc - the worker type to get the worker pool for
      Returns:
      the WorkerPool

    • close

      public void close()
      Close all wpcs related to the WorkloadManager.