Package ai.djl.serving.wlm

Class PermanentBatchAggregator<I,O>

java.lang.Object
ai.djl.serving.wlm.PermanentBatchAggregator<I,O>
public class PermanentBatchAggregator<I,O> extends Object
a batch aggregator that never terminates by itself. the surrounding thread has to be interrupted by sending an interrupt signal.

  • Field Details

  • Constructor Details

    • PermanentBatchAggregator

      public PermanentBatchAggregator(WorkerPoolConfig<I,O> wpc, LinkedBlockingDeque<WorkerJob<I,O>> jobQueue)
      Constructs a PermanentBatchAggregator instance.
      Parameters:
      wpc - the workerPoolConfig to use.
      jobQueue - the job queue for polling data from.

  • Method Details

    • pollBatch

      protected List<WorkerJob<I,O>> pollBatch() throws InterruptedException
      Fills in the list with a batch of jobs.
      Returns:
      a list of jobs read by this batch interation.
      Throws:
      InterruptedException - if interrupted

    • isFinished

      public boolean isFinished()
      Checks if this BatchAggregator and the thread can be shutdown or if this aggregator waits for more data.
      Returns:
      true if we can shutdown the thread. for example when max idle time exceeded in temporary batch aggregator.

    • getRequest

      public List<Job<I,O>> getRequest() throws InterruptedException
      Poll the queue and return a list of Input Objects for the model.
      Returns:
      list of input objects to pass to the model.
      Throws:
      InterruptedException - if thread gets interrupted while waiting for new data in the queue.

    • sendResponse

      public void sendResponse()
      Sends to response to all waiting clients.

    • sendError

      public void sendError(Throwable error)
      Completes the job with an error.
      Parameters:
      error - the exception

    • drainTo

      protected void drainTo(List<WorkerJob<I,O>> list, long maxDelay) throws InterruptedException
      Throws:
      InterruptedException