All Classes and Interfaces
Class
Description
Resolves the test account for a
RunPipelineCrawl run.Tracks the
crawl_ids of pipeline-crawl runs that currently have an
open observation stream (an in-flight OrchestrationDriver run).Chunker scaffold: the raw chunker input (echo output) carries no vector-set
directives, so the chunker no-ops ("nothing to chunk").
Streams court-1000 PipeDoc fixtures off the classpath.
Stage selector for the court-1000 corpus.
Seeds the courtlistener_test database on startup for JDBC crawl E2E tests.
Ninja cleanup for pipeline-crawl runs: by default a run deletes every
resource it provisioned, so an hourly health-check e2e leaves the
platform exactly as it found it.
Everything one run needs torn down.
Creates a datasource for a
RunPipelineCrawl run and returns its
id + api key directly.Credentials for a freshly-created datasource.
Builds the chunker module's
custom_config for a pipeline-crawl
run.Shared helper for pipeline-crawl graph building: applies per-run tuning
overrides to graph nodes.
Idempotent registration of the chunker / embedder / VectorSet fixtures
the chunker→embedder→sink Pipeline tab shape needs before its IndexPlan
can provision KNN fields.
gRPC client wrapper for engine pipeline-graph queries used by the pipeline-crawl
orchestrator.
Outcome of
EngineGraphClient.getOrCreateActiveGraph(PipelineGraph, String, String, String): the active graph's id and
version, plus whether this call created it.Wraps the gRPC response observer with the boilerplate common to every
progress event: serialization on the observer, Timestamp build, pretty
step name lookup.
Global exception mapper that captures unhandled errors, records them in
LastErrorTracker, and returns a structured JSON 500 response.Binds a datasource to an active graph at its entry node, returning the
datasource-instance id directly.
Tiny
StreamObserver-to-CompletableFuture adapter for unary
gRPC calls.Bounded retry wrapper around a blocking gRPC stub call.
Retry budget for a single op.
OpenSearch index administration REST API for the Quinoa UI.
Polls repository-service's indexing ledger for per-crawl receipt
progress.
Receipt counts for one crawl, partitioned the way the run report
wants them.
Drives court-1000 PipeDocs at the opensearch-sink (OSS) production indexing
path:
PipeStepProcessorService.processData (unary).Snapshot of a push job's progress.
One open bidi stream to connector-intake's
uploadPipeDocStream —
the SAME protocol the JDBC connector speaks (IntakeCrawlStream is
the reference implementation): StreamContext handshake first
(datasource + api key + crawl_id), then one PipeDocItem per doc
with per-doc acks; backpressure is HTTP/2 native.Per-doc intake outcome.
Drives the JDBC source for a
RunPipelineCrawl run:
Create the crawl definition via gRPC.Result of
JdbcCrawlManager.setupAndTrigger(String, String, String, JdbcSourceConfig): the connector's triggerId (also
the crawl_id on every dispatched PipeStream) and the crawlName
the connector now has registered.Subscribes to the JDBC connector's
streamCrawlStatus server-streaming RPC and hydrates a RunLedger
from the per-event payload.The active subscription.
In-memory tracker for the most recent 500-level error.
Immutable snapshot of a single recorded error.
The bounded supply of work the
MockWorkEngineService hands out.Control + observability for the
MockWorkEngineService.A mock
ModuleWorkService engine for exercising a module's demand-pull
work loop in isolation — no real engine, Kafka, Redis or S3.Accumulates what the module-under-test did with the served work: how many units
it acked, by
ProcessingStatus (SUCCESS / RETRYABLE / PERMANENT) and by
result disposition (UPDATED_PAYLOAD / NO_OP / REJECT), plus a wall-clock
throughput between the first served unit and the last ack.Engine-path perf (Test 2): the sidecar acts as its OWN connector and pushes
corpus docs through the production front door — connector-intake's
uploadPipeDocStream — so the engine does everything it does for a
real crawl: claim-check staging, fan-out to the subscribed graph, work
dispensing over the bidi work stream, and the module's own worker loop
self-ramps against the backlog.Live, interactive load generator for one module's
process() interface,
driven through the dev process-once door.Static, config-driven map of module name → base URL for the Module Test
door.
REST resource for the module-testing-sidecar harness.
Forwards a serialized
PipeStream to a module's dev test door
(POST /module-runtime/v1/process-once) and returns the door's raw
JSON outcome.The door's HTTP status plus its raw response body (JSON on success).
Module Test — invoke a single module's processor directly via its dev test
door (
POST /module-runtime/v1/process-once), bypassing the engine
pull-loop and queue (roadmap #3).REST resource backing the testing-sidecar's "OpenSearch Index Tests" tab.
Polls opensearch-manager for per-crawl indexed-doc counts across the
whole index family (base +
--chunk--* + --vs--*).One tick's view of the crawl's index family.
opensearch-sink scaffold, e2e-faithful: provisions exactly what a pipeline
crawl provisions before the sink sees a doc —
the full 2×2 vector-set registration (sentence-10-3 + sentences_internal
× minilm + paraphrase-minilm) via
EmbedderFixturesRegistrar,
matching what the captured corpus docs actually carry;
an IndexPlan with a PINNED id and the caller-chosen
IndexingStrategy (NESTED / SEPARATE_INDICES / CHUNK_COMBINED —
the same three the e2e exercises; default CHUNK_COMBINED, the e2e
default), gated on READY like the crawl gates dispatch;
the semantic centroid configs assigned to the index, so the sink can
write the document/section/paragraph centroid SPRs the corpus carries
("KNN field not provisioned at bind time" otherwise).
The plan id + instance ride StreamMetadata.context_params — what
OpenSearchSinkOptionsParser reads.Blocking, non-Mutiny equivalent of
E2EPipelineTestService's
registerConfigs(...) chain — registers the minimum ChunkerConfigs,
EmbeddingConfigs, VectorSet recipes, and IndexPlan the opensearch-sink needs
to resolve a plan and publish indexing work (inline Kafka event below the
size threshold, S3 claim-check + pointer above it; OSM's
KafkaIndexingConsumer indexes — no redis on this path).Result of a single setup run.
Result of a sink-perf setup: the plan the sink resolves plus what to
tear down afterwards.
The actual pipeline-crawl orchestration body (preferred harness path — blocking gRPC, no Mutiny).
Hard wall-clock deadline exceeded.
Wait loop interrupted (sidecar shutdown, etc).
The OS-drain phase found docs the platform terminally failed to
index.
Dispatch-stream subscription terminated with a real failure: either
the connector emitted a malformed event our handler refused to accept,
or the gRPC stream errored with a non-CANCELLED status.
Parser scaffold: the parser has no captured
<step>-input-court dataset
— its input is the raw source file, which exists BEFORE any capture tap (and
the court corpus is JDBC, which skips the parser entirely).No-setup scaffold: the corpus doc already carries everything the step needs.
Renders a final
PipeDoc into a readable Struct for the UI's
single-doc output panel.gRPC entry point for
RunPipelineCrawl.REST/SSE bridge from the FE's Pipeline tab to the gRPC
PipelineCrawlerImpl.FE-facing payload.
JDBC source payload.
S3 source payload.
SINGLE_DOC source payload.
Live observation feed for the Pipeline-tab panels.
Builds a
PipelineGraph from a RunPipelineCrawlRequest's
source / steps / observe / transport selection.Result of
PipelineGraphFactory.build(String, String, PipelineSource, Set, Set, Set, List, int, Struct, Struct, Struct, Struct, Struct, String) — graph + names callers need to dispatch into it.Produces a protobuf
TypeRegistry populated from generated Java classes.Streaming HTTP client to connector-intake's raw upload endpoint
(
POST /uploads/raw) — the S3-style path that streams a document of
any length straight to :intake/S3 without ever buffering it whole.Per-
(key, step) rich observation collector for the Pipeline-tab
panels, fed live by PipelineEventsConsumer from the engine's
pipeline-events audit stream.Snapshot returned by
RichObservationCollector.snapshot(String, String); immutable.Single source of truth for one pipeline-crawl run's progress.
Drops a run's per-graph redis streams once the pipeline-crawl finishes.
Owns the lifetime of pipeline-crawl runs INDEPENDENTLY of any transport
connection.
One run: its thread, journal, and subscribers.
Summary row for one run.
Run state for the runs listing.
Drives the S3 source for a
RunPipelineCrawl run — the analogue of
JdbcCrawlManager for buckets.Result of
S3CrawlManager.hydrateAndTrigger(String, String, String, S3SourceConfig): the connector's request id (the
crawl's idempotency key) and how many docs were uploaded during
hydration (0 when hydration was skipped).Subscribes to the s3-connector's
StreamCrawlStatus for one crawl
and mirrors its state into a S3DispatchSubscriber.S3DispatchState the orchestrator's
wait loop reads each tick — the S3 counterpart of
JdbcDispatchSubscriber, but count-based: the connector reports
how many objects it dispatched (and, at COMPLETED, the final total)
rather than per-doc ids.Live mirror of one crawl's connector-side state.
Resolves the set of documents to upload into S3 before an S3 pipeline-crawl
run, when
hydrate_sample_data is set.One document to upload: its object key (path within the corpus), bytes, mime type.
One sample-data fixture jar's self-description, read from its
META-INF/pipestream-sample-data.json resource.Reads the per-doc
PipeDocs out of a registered sample-data set.Discovers every sample-data fixture jar on the classpath by scanning for its
META-INF/pipestream-sample-data.json descriptor, and exposes them as a
queryable registry — keyed by artifactId and by the step each dataset feeds.Read-only view of the
SampleDataRegistry: which sample-data fixture
jars are on the classpath, and what each one is.Curated set of well-known sample documents bundled into the sidecar JAR
for quick testing without needing a repository or manual upload.
Reads in-process tap counters and rich-observation snapshots for the
orchestrator's poll loop.
Drives the SINGLE_DOC source for a
RunPipelineCrawl run.Result of
SingleDocDispatcher.uploadAndDispatch(String, String, String, SingleDocSourceConfig, String): the doc_id the upload produced.One stage's receive-side view in a
RunLedger: which doc ids
landed at this stage's tap, and how many we expect.Per-step integration scaffolding for module perf / isolation tests.
Factory for per-step
StepScaffolds — forStep("embedder") etc.Captures the most recent
PipeDoc that reached the terminal sidecar
node, keyed by account.Batch cleanup of leftover test graphs.
A single failed graph delete.
Outcome of a batch cleanup run.
Demand-pull processor for the module-testing-sidecar service.
Helper class for building test requests to the TestProcessor.
Processing modes supported by the test processor.
Builder for creating test documents with various configurations.
Builder for creating test requests with various configurations.
Per-account, per-step document counter for transport + e2e validation.
Per-step totals for REST / worker status (copy of live counters).