Package ai.djl.serving.wlm

Class WorkerPool<I,O>

java.lang.Object
ai.djl.serving.wlm.WorkerPool<I,O>
public class WorkerPool<I,O> extends Object
Manages the work load for a single model.

  • Method Details

    • increaseRef

      public void increaseRef()
      Increases the reference count.

    • decreaseRef

      public int decreaseRef()
      Decrease the reference count and return the current count.
      Returns:
      the current count

    • getWpc

      public WorkerPoolConfig<I,O> getWpc()
      Returns the model of the worker pool.
      Returns:
      the model of the worker pool

    • getWorkerGroups

      public Map<ai.djl.Device,WorkerGroup<I,O>> getWorkerGroups()
      Returns a map of WorkerGroup.
      Returns:
      a map of WorkerGroup

    • getWorkers

      public List<WorkerThread<I,O>> getWorkers()
      Returns a list of worker thread.
      Returns:
      the workers

    • getJobQueue

      public LinkedBlockingDeque<WorkerJob<I,O>> getJobQueue()
      Returns the JobQueue for this model.
      Returns:
      the jobQueue

    • getMaxWorkers

      public int getMaxWorkers()
      Returns the maximum number of workers for a model across all devices.
      Returns:
      the maximum number of workers for a model across all devices

    • isAllWorkerDied

      public boolean isAllWorkerDied()
      Return if all workers died.
      Returns:
      true if all workers died

    • isAllWorkerBusy

      public boolean isAllWorkerBusy()
      Returns true if all workers are busy.
      Returns:
      true if all workers are busy

    • isFullyScaled

      public boolean isFullyScaled()
      Returns if the worker groups is fully scaled.
      Returns:
      true if the worker groups is fully scaled

    • initWorkers

      public void initWorkers(String deviceName)
      Initializes new worker capacities for this model.
      Parameters:
      deviceName - the device for the model, null for default devices

    • initWorkers

      public void initWorkers(String deviceName, int minWorkers, int maxWorkers)
      Initializes new worker capacities for this model.
      Parameters:
      deviceName - the device for the model, null for default devices
      minWorkers - minimum amount of workers (-1 for model default).
      maxWorkers - maximum amount of workers (-1 for model default).

    • scaleWorkers

      public void scaleWorkers(String deviceName, int minWorkers, int maxWorkers)
      Sets new worker capacities for this model.
      Parameters:
      deviceName - the device for the model, null for all loaded devices
      minWorkers - minimum amount of workers.
      maxWorkers - maximum amount of workers.

    • shutdownWorkers

      public void shutdownWorkers()
      Shutdown all works.

    • cleanup

      public void cleanup()
      removes all stopped workers and workers in state error from the pool.

    • shutdown

      public void shutdown()
      Shuts down all the worker threads in the work pool.