Package ai.djl.serving.wlm

Class WorkLoadManager

  • public class WorkLoadManager
extends java.lang.Object
    WorkLoadManager is responsible to manage the work load of worker thread. the manage scales up/down the required amount of worker threads per wpc.

    • Constructor Detail

      • WorkLoadManager

        public WorkLoadManager()
        Constructs a WorkLoadManager instance.

    • Method Detail

      • registerWorkerPool

        public <I,​O> WorkerPool<I,​O> registerWorkerPool​(WorkerPoolConfig<I,​O> wpc)
        Registers a WorkerPool (model).

        This operation is idempotent and will return the existing workerpool if the wpc was already registered.

        Type Parameters:
        I - the wpc input class
        O - the wpc output class
        Parameters:
        wpc - the wpc to create the worker pool for
        Returns:
        the WorkerPool

      • unregisterWorkerPool

        public void unregisterWorkerPool​(WorkerPoolConfig<?,​?> wpc)
        Removes a worker pool from management.
        Parameters:
        wpc - the wpc to remove

      • runJob

        public <I,​O> java.util.concurrent.CompletableFuture<O> runJob​(Job<I,​O> job)
        Adds an inference job to the job queue of the next free worker. scales up worker if necessary.
        Type Parameters:
        I - the wpc input class
        O - the wpc output class
        Parameters:
        job - an inference job to be executed.
        Returns:
        true if submit success, false otherwise.

      • getNumRunningWorkers

        public int getNumRunningWorkers​(WorkerPoolConfig<?,​?> wpc)
        Returns the number of running workers of a wpc. running workers are workers which are not stopped, in error or scheduled to scale down.
        Parameters:
        wpc - the wpc we are interested in.
        Returns:
        number of running workers.

      • getWorkerPoolById

        public <I,​O> WorkerPool<I,​O> getWorkerPoolById​(java.lang.String id)
        Returns the WorkerPool for a wpc.
        Type Parameters:
        I - the wpc class
        O - the wpc class
        Parameters:
        id - the wpc id
        Returns:
        the WorkerPool

      • getWorkerPool

        public <I,​O> WorkerPool<I,​O> getWorkerPool​(WorkerPoolConfig<I,​O> wpc)
        Returns the WorkerPool for a model.
        Type Parameters:
        I - the wpc input class
        O - the wpc output class
        Parameters:
        wpc - the worker type to get the worker pool for
        Returns:
        the WorkerPool

      • close

        public void close()
        Close all wpcs related to the WorkloadManager.