Package ai.djl.serving.wlm

Class PermanentBatchAggregator

  • public class PermanentBatchAggregator
extends java.lang.Object
    a batch aggregator that never terminates by itself. the surrounding thread has to be interrupted by sending an interrupt signal.

    • Constructor Summary

      Constructors 
      Constructor Description
      PermanentBatchAggregator​(ModelInfo model, java.util.concurrent.LinkedBlockingDeque<WorkerJob> jobQueue)
      Constructs a PermanentBatchAggregator instance.

    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      protected void drainTo​(java.util.List<WorkerJob> list, int maxDelay)  
      java.util.List<ai.djl.modality.Input> getRequest()
      Poll the queue and return a list of Input Objects for the model.
      boolean isFinished()
      Checks if this BatchAggregator and the thread can be shutdown or if this aggregator waits for more data.
      protected java.util.List<WorkerJob> pollBatch()
      Fills in the list with a batch of jobs.
      void sendError​(java.lang.Throwable error)
      Completes the job with an error.
      void sendResponse​(java.util.List<ai.djl.modality.Output> outputs)
      Sends to response to all waiting clients.

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    • Field Detail

      • batchSize

        protected int batchSize

      • maxBatchDelay

        protected int maxBatchDelay

      • wjs

        protected java.util.List<WorkerJob> wjs

      • jobQueue

        protected java.util.concurrent.LinkedBlockingDeque<WorkerJob> jobQueue

    • Constructor Detail

      • PermanentBatchAggregator

        public PermanentBatchAggregator​(ModelInfo model,
                                java.util.concurrent.LinkedBlockingDeque<WorkerJob> jobQueue)
        Constructs a PermanentBatchAggregator instance.
        Parameters:
        model - the model to use.
        jobQueue - the job queue for polling data from.

    • Method Detail

      • pollBatch

        protected java.util.List<WorkerJob> pollBatch()
                                       throws java.lang.InterruptedException
        Fills in the list with a batch of jobs.
        Returns:
        a list of jobs read by this batch interation.
        Throws:
        java.lang.InterruptedException - if interrupted

      • isFinished

        public boolean isFinished()
        Checks if this BatchAggregator and the thread can be shutdown or if this aggregator waits for more data.
        Returns:
        true if we can shutdown the thread. for example when max idle time exceeded in temporary batch aggregator.

      • getRequest

        public java.util.List<ai.djl.modality.Input> getRequest()
                                                 throws java.lang.InterruptedException
        Poll the queue and return a list of Input Objects for the model.
        Returns:
        list of input objects to pass to the model.
        Throws:
        java.lang.InterruptedException - if thread gets interrupted while waiting for new data in the queue.

      • sendResponse

        public void sendResponse​(java.util.List<ai.djl.modality.Output> outputs)
        Sends to response to all waiting clients.
        Parameters:
        outputs - list of model-outputs in same order as the input objects.

      • sendError

        public void sendError​(java.lang.Throwable error)
        Completes the job with an error.
        Parameters:
        error - the exception

      • drainTo

        protected void drainTo​(java.util.List<WorkerJob> list,
                       int maxDelay)
                throws java.lang.InterruptedException
        Throws:
        java.lang.InterruptedException