Package ai.djl.serving.wlm

Class WorkerPool<I,​O>

  • public class WorkerPool<I,​O>
extends java.lang.Object
    Manages the work load for a single model.

    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      void cleanup()
      removes all stopped workers and workers in state error from the pool.
      java.util.concurrent.LinkedBlockingDeque<WorkerJob<I,​O>> getJobQueue()
      Returns the JobQueue for this model.
      int getMaxWorkers()
      Returns the maximum number of workers for a model across all devices.
      java.util.Map<ai.djl.Device,​WorkerGroup<I,​O>> getWorkerGroups()
      Returns a map of WorkerGroup.
      java.util.List<WorkerThread<I,​O>> getWorkers()
      Returns a list of worker thread.
      void initWorkers​(java.lang.String deviceName, int minWorkers, int maxWorkers)
      Initializes new worker capacities for this model.
      boolean isAllWorkerBusy()
      Returns true if all workers are busy.
      boolean isFullyScaled()
      Returns if the worker groups is fully scaled.
      void scaleWorkers​(java.lang.String deviceName, int minWorkers, int maxWorkers)
      Sets new worker capacities for this model.
      void shutdown()
      Shuts down all the worker threads in the work pool.
      void shutdownWorkers()
      Shutdown all works.

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    • Method Detail

      • getWorkerGroups

        public java.util.Map<ai.djl.Device,​WorkerGroup<I,​O>> getWorkerGroups()
        Returns a map of WorkerGroup.
        Returns:
        a map of WorkerGroup

      • getWorkers

        public java.util.List<WorkerThread<I,​O>> getWorkers()
        Returns a list of worker thread.
        Returns:
        the workers

      • getJobQueue

        public java.util.concurrent.LinkedBlockingDeque<WorkerJob<I,​O>> getJobQueue()
        Returns the JobQueue for this model.
        Returns:
        the jobQueue

      • getMaxWorkers

        public int getMaxWorkers()
        Returns the maximum number of workers for a model across all devices.
        Returns:
        the maximum number of workers for a model across all devices

      • isAllWorkerBusy

        public boolean isAllWorkerBusy()
        Returns true if all workers are busy.
        Returns:
        true if all workers are busy

      • isFullyScaled

        public boolean isFullyScaled()
        Returns if the worker groups is fully scaled.
        Returns:
        true if the worker groups is fully scaled

      • initWorkers

        public void initWorkers​(java.lang.String deviceName,
                        int minWorkers,
                        int maxWorkers)
        Initializes new worker capacities for this model.
        Parameters:
        deviceName - the device for the model, null for default devices
        minWorkers - minimum amount of workers.
        maxWorkers - maximum amount of workers.

      • scaleWorkers

        public void scaleWorkers​(java.lang.String deviceName,
                         int minWorkers,
                         int maxWorkers)
        Sets new worker capacities for this model.
        Parameters:
        deviceName - the device for the model, null for all loaded devices
        minWorkers - minimum amount of workers.
        maxWorkers - maximum amount of workers.

      • shutdownWorkers

        public void shutdownWorkers()
        Shutdown all works.

      • cleanup

        public void cleanup()
        removes all stopped workers and workers in state error from the pool.

      • shutdown

        public void shutdown()
        Shuts down all the worker threads in the work pool.