Package ai.djl.serving.wlm

Class WorkerPool<I,​O>

  • public class WorkerPool<I,​O>
extends java.lang.Object
    Manages the work load for a single model.

    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      void cleanup()
      removes all stopped workers and workers in state error from the pool.
      java.util.concurrent.LinkedBlockingDeque<WorkerJob<I,​O>> getJobQueue()
      Returns the JobQueue for this model.
      int getMaxWorkers()
      Returns the maximum number of workers for a model across all devices.
      java.util.Map<ai.djl.Device,​WorkerGroup<I,​O>> getWorkerGroups()
      Returns a map of WorkerGroup.
      java.util.List<WorkerThread<I,​O>> getWorkers()
      Returns a list of worker thread.
      void initWorkers​(java.lang.String deviceName, int minWorkers, int maxWorkers)
      Initializes new worker capacities for this model.
      boolean isFullyScaled()
      Returns if the worker groups is fully scaled.
      void scaleWorkers​(java.lang.String deviceName, int minWorkers, int maxWorkers)
      Sets new worker capacities for this model.
      void shutdown()
      Shuts down all the worker threads in the work pool.
      void shutdownWorkers()
      Shutdown all works.

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    • Method Detail

      • getWorkerGroups

        public java.util.Map<ai.djl.Device,​WorkerGroup<I,​O>> getWorkerGroups()
        Returns a map of WorkerGroup.
        Returns:
        a map of WorkerGroup

      • getWorkers

        public java.util.List<WorkerThread<I,​O>> getWorkers()
        Returns a list of worker thread.
        Returns:
        the workers

      • getJobQueue

        public java.util.concurrent.LinkedBlockingDeque<WorkerJob<I,​O>> getJobQueue()
        Returns the JobQueue for this model.
        Returns:
        the jobQueue

      • getMaxWorkers

        public int getMaxWorkers()
        Returns the maximum number of workers for a model across all devices.
        Returns:
        the maximum number of workers for a model across all devices

      • isFullyScaled

        public boolean isFullyScaled()
        Returns if the worker groups is fully scaled.
        Returns:
        true if the worker groups is fully scaled

      • initWorkers

        public void initWorkers​(java.lang.String deviceName,
                        int minWorkers,
                        int maxWorkers)
        Initializes new worker capacities for this model.
        Parameters:
        deviceName - the device for the model, null for default devices
        minWorkers - minimum amount of workers.
        maxWorkers - maximum amount of workers.

      • scaleWorkers

        public void scaleWorkers​(java.lang.String deviceName,
                         int minWorkers,
                         int maxWorkers)
        Sets new worker capacities for this model.
        Parameters:
        deviceName - the device for the model, null for all loaded devices
        minWorkers - minimum amount of workers.
        maxWorkers - maximum amount of workers.

      • shutdownWorkers

        public void shutdownWorkers()
        Shutdown all works.

      • cleanup

        public void cleanup()
        removes all stopped workers and workers in state error from the pool.

      • shutdown

        public void shutdown()
        Shuts down all the worker threads in the work pool.