-
public final class TaskRouterCompletely task-agnostic. For each request it finds (or lazy-loads) the adapter for the model, runs it, times it, and reports — for any inference kind that has a registered engine.
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description public classTaskRouter.Companion
-
Field Summary
Fields Modifier and Type Field Description private volatile Function0<Unit>onOutOfMemory
-
Constructor Summary
Constructors Constructor Description TaskRouter(Map<String, InferenceAdapter> adapters, Map<String, ModelConfig> configs, AdapterLoader loader, EventReporter reporter)
-
Method Summary
Modifier and Type Method Description final Function0<Unit>getOnOutOfMemory()Invoked when an adapter throws OutOfMemoryError — the owner runs the emergency shutdown. final UnitsetOnOutOfMemory(Function0<Unit> onOutOfMemory)Invoked when an adapter throws OutOfMemoryError — the owner runs the emergency shutdown. final Unitgate(String reason)final Unitungate()final List<TaskResult>handle(TaskMessage batch)-
-
Constructor Detail
-
TaskRouter
TaskRouter(Map<String, InferenceAdapter> adapters, Map<String, ModelConfig> configs, AdapterLoader loader, EventReporter reporter)
-
-
Method Detail
-
getOnOutOfMemory
final Function0<Unit> getOnOutOfMemory()
Invoked when an adapter throws OutOfMemoryError — the owner runs the emergency shutdown.
-
setOnOutOfMemory
final Unit setOnOutOfMemory(Function0<Unit> onOutOfMemory)
Invoked when an adapter throws OutOfMemoryError — the owner runs the emergency shutdown.
-
handle
final List<TaskResult> handle(TaskMessage batch)
-
-
-
-