Package ai.djl.serving.wlm

Class WorkerPoolConfig<I,O>

java.lang.Object
ai.djl.serving.wlm.WorkerPoolConfig<I,O>
Type Parameters:
I - the input type
O - the output type
Direct Known Subclasses:
ModelInfo
public abstract class WorkerPoolConfig<I,O> extends Object
A WorkerPoolConfig represents a task that could be run in the WorkLoadManager.

Each WorkerThread (also WorkerPool and WorkerGroup) focuses on executing a single worker type. They contain the configuration for the thread, any persistent data, and the code to run on the thread.

  • Field Details

    • id

      protected transient String id

    • uid

      protected transient String uid

    • version

      protected String version

    • modelUrl

      protected String modelUrl

    • queueSize

      protected int queueSize

    • batchSize

      protected int batchSize

    • maxBatchDelayMillis

      protected int maxBatchDelayMillis

    • maxIdleSeconds

      protected int maxIdleSeconds

    • minWorkers

      protected Integer minWorkers

    • maxWorkers

      protected Integer maxWorkers

  • Constructor Details

    • WorkerPoolConfig

      protected WorkerPoolConfig()

  • Method Details

    • load

      public abstract void load(ai.djl.Device device) throws ai.djl.ModelException, IOException
      Loads the worker type to the specified device.
      Parameters:
      device - the device to load worker type on
      Throws:
      IOException - if failed to read worker type file
      ai.djl.ModelException - if failed to load the specified model

    • newThread

      public abstract WorkerPoolConfig.ThreadConfig<I,O> newThread(ai.djl.Device device)
      Starts a new WorkerThread for this WorkerPoolConfig.
      Parameters:
      device - the device to run on
      Returns:
      the new WorkerPoolConfig.ThreadConfig

    • initialize

      public abstract void initialize() throws IOException, ai.djl.ModelException
      Initialize the worker.
      Throws:
      IOException - if failed to download worker
      ai.djl.repository.zoo.ModelNotFoundException - if model not found
      ai.djl.ModelException

    • close

      public abstract void close()
      Close all loaded workers.

    • withDefaultDevice

      public ai.djl.Device withDefaultDevice(String deviceName)
      Returns the default device for this model if device is null.
      Parameters:
      deviceName - the device to use if it is not null
      Returns:
      a non-null device

    • getStatus

      public abstract WorkerPoolConfig.Status getStatus()
      Returns the worker type loading status.
      Returns:
      the worker type loading status

    • isParallelLoading

      public abstract boolean isParallelLoading()
      Returns if the worker type can be load parallel on multiple devices.
      Returns:
      if the worker type can be load parallel on multiple devices

    • getLoadOnDevices

      public abstract String[] getLoadOnDevices()
      Returns the devices the worker type will be loaded on at startup.
      Returns:
      the devices the worker type will be loaded on at startup

    • setId

      public void setId(String id)
      Sets the worker configs ID.
      Parameters:
      id - the worker configs ID

    • getUid

      public String getUid()
      Returns the worker configs unique ID.
      Returns:
      the worker configs unique ID

    • getId

      public String getId()
      Returns the worker configs ID.
      Returns:
      the worker configs ID

    • getVersion

      public String getVersion()
      Returns the worker type version.
      Returns:
      the worker type version

    • getModelUrl

      public String getModelUrl()
      Returns the worker type url.
      Returns:
      the worker type url

    • setMaxIdleSeconds

      public void setMaxIdleSeconds(int maxIdleSeconds)
      Sets the configured max idle time in seconds of workers.
      Parameters:
      maxIdleSeconds - the configured max idle time in seconds of workers

    • getMaxIdleSeconds

      public int getMaxIdleSeconds()
      Returns the configured max idle time in seconds of workers.
      Returns:
      the max idle time in seconds

    • setBatchSize

      public void setBatchSize(int batchSize)
      Sets the configured batch size.
      Parameters:
      batchSize - the configured batch size

    • getBatchSize

      public int getBatchSize()
      Returns the configured batch size.
      Returns:
      the configured batch size

    • setMaxBatchDelayMillis

      public void setMaxBatchDelayMillis(int maxBatchDelayMillis)
      Sets the maximum delay in milliseconds to aggregate a batch.
      Parameters:
      maxBatchDelayMillis - the maximum delay in milliseconds to aggregate a batch

    • getMaxBatchDelayMillis

      public int getMaxBatchDelayMillis()
      Returns the maximum delay in milliseconds to aggregate a batch.
      Returns:
      the maximum delay in milliseconds to aggregate a batch

    • setQueueSize

      public void setQueueSize(int queueSize)
      Sets the configured size of the workers queue.
      Parameters:
      queueSize - the configured size of the workers queue

    • getQueueSize

      public int getQueueSize()
      Returns the configured size of the workers queue.
      Returns:
      requested size of the workers queue.

    • setMinWorkers

      public void setMinWorkers(int minWorkers)
      Sets the starting number of min workers.
      Parameters:
      minWorkers - Sets the starting number of min workers

    • getMinWorkers

      public int getMinWorkers(ai.djl.Device device)
      Returns the minimum number of workers.
      Parameters:
      device - the device to get the min workers for
      Returns:
      the minimum number of workers

    • setMaxWorkers

      public void setMaxWorkers(int maxWorkers)
      Sets the starting number of max workers.
      Parameters:
      maxWorkers - Sets the starting number of max workers

    • getMaxWorkers

      public int getMaxWorkers(ai.djl.Device device)
      Returns the maximum number of workers.
      Parameters:
      device - the device to get the max workers for
      Returns:
      the maximum number of workers

    • setMinMaxWorkers

      public void setMinMaxWorkers(int minWorkers, int maxWorkers)
      Sets the starting minimum and maximum number of workers.
      Parameters:
      minWorkers - the new minimum number of workers
      maxWorkers - the new maximum number of workers

    • equals

      public boolean equals(Object o)
      Overrides:
      equals in class Object

    • hashCode

      public int hashCode()
      Overrides:
      hashCode in class Object

    • toString

      public String toString()
      Overrides:
      toString in class Object