Package ai.djl.serving.wlm

Class WorkLoadManager.WorkerPool<I,O>

java.lang.Object

ai.djl.serving.wlm.WorkLoadManager.WorkerPool<I,O>

All Implemented Interfaces:

java.lang.AutoCloseable

Enclosing class:

WorkLoadManager

public final class WorkLoadManager.WorkerPool<I,O>
extends java.lang.Object
implements java.lang.AutoCloseable

Manages the work load for a single model.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
class	WorkLoadManager.WorkerPool.WorkerPoolDevice	The WorkLoadManager.WorkerPool.WorkerPoolDevice manages the WorkLoadManager.WorkerPool for a particular Device.

Constructor Summary

Constructors

Constructor	Description
WorkerPool(ModelInfo<I,O> model)	Construct and initial data structure.

Method Summary

Modifier and Type	Method	Description
void	cleanup()	removes all stopped workers and workers in state error from the pool.
void	close()
WorkLoadManager.WorkerPool.WorkerPoolDevice	forDevice(ai.djl.Device device)	Returns the WorkLoadManager.WorkerPool.WorkerPoolDevice for a particular Device.
java.util.concurrent.LinkedBlockingDeque<WorkerJob<I,O>>	getJobQueue()	Returns the JobQueue for this model.
int	getMaxWorkers()	Returns the maximum number of workers for a model across all devices.
int	getMinWorkers()	Returns the minimum number of workers for a model across all devices.
java.util.List<WorkerThread<I,O>>	getWorkers()	Returns a list of worker thread.
void	log()	Logs the current state of this WorkerPool when level "Debug" is enabled.
WorkLoadManager.WorkerPool<I,O>	scaleWorkers(ai.djl.Device device, int newMinWorkers, int newMaxWorkers)	Sets new worker capcities for this model.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

WorkerPool

public WorkerPool(ModelInfo<I,O> model)

Construct and initial data structure.

Parameters:

model - the model this WorkerPool belongs to.

Method Detail

getWorkers

public java.util.List<WorkerThread<I,O>> getWorkers()

Returns a list of worker thread.

Returns:

the workers

getJobQueue

public java.util.concurrent.LinkedBlockingDeque<WorkerJob<I,O>> getJobQueue()

Returns the JobQueue for this model.

Returns:

the jobQueue

getMinWorkers

public int getMinWorkers()

Returns the minimum number of workers for a model across all devices.

Returns:

the minimum number of workers for a model across all devices

getMaxWorkers

public int getMaxWorkers()

Returns the maximum number of workers for a model across all devices.

Returns:

the maximum number of workers for a model across all devices

scaleWorkers

public WorkLoadManager.WorkerPool<I,O> scaleWorkers(ai.djl.Device device,
                                                          int newMinWorkers,
                                                          int newMaxWorkers)

Sets new worker capcities for this model.

Parameters:

device - the device for the model or cpu if null

newMinWorkers - minimum amount of workers.

newMaxWorkers - maximum amount of workers.

Returns:

this ModelInfo

forDevice

public WorkLoadManager.WorkerPool.WorkerPoolDevice forDevice(ai.djl.Device device)

Returns the WorkLoadManager.WorkerPool.WorkerPoolDevice for a particular Device.

Parameters:

device - the device for a worker pool

Returns:

the WorkLoadManager.WorkerPool.WorkerPoolDevice or null if device is not used

log

public void log()

Logs the current state of this WorkerPool when level "Debug" is enabled.

Logs all thread-ids in the pool.

cleanup

public void cleanup()

removes all stopped workers and workers in state error from the pool.

close

public void close()

Specified by:

close in interface java.lang.AutoCloseable