Class S3CrawlManager

java.lang.Object

ai.pipestream.module.pipelineprobe.pipelinecrawl.S3CrawlManager

@ApplicationScoped
public class S3CrawlManager
extends Object

Drives the S3 source for a RunPipelineCrawl run — the analogue of JdbcCrawlManager for buckets. Given the resolved (datasourceId, apiKey) from DatasourceManager and the S3SourceConfig from the request, this:

1. optionally hydrates the bucket by uploading sample docs via the s3-connector's UploadTestFile RPC (when hydrate_sample_data is set), and
2. triggers the crawl via StartCrawl, returning the connector's request_id.

Request building mirrors the legacy S3CrawlE2ETestService (hydrateData / startCrawl) verbatim, but on a plain blocking gRPC stub (no Mutiny) — same shape as JdbcCrawlManager.

Unlike JDBC, the s3-connector does NOT emit a dispatch-ledger stream and StartCrawl does not return a doc count, so there is no expectedRowCount/per-doc-id ledger to hydrate. The orchestrator verifies the S3 run with a stability-based indexed-count poll instead of the shared ledger close (see OrchestrationDriver.runS3Dispatch).

Wiring: quarkus.grpc.clients.s3-connector.* + quarkus.stork."s3-connector".* for the gRPC stub.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static final record	S3CrawlManager.S3Dispatch	Result of hydrateAndTrigger(String, String, String, S3SourceConfig): the connector's request id (the crawl's idempotency key) and how many docs were uploaded during hydration (0 when hydration was skipped).

Constructor Summary

Constructors

Constructor	Description
S3CrawlManager()	Creates the S3 crawl manager.

Method Summary

Modifier and Type	Method	Description
S3CrawlManager.S3Dispatch	hydrateAndTrigger(String accountId, String datasourceId, String apiKey, ai.pipestream.testing.harness.v1.S3SourceConfig source)	Optionally hydrate the bucket, then trigger the crawl.

Methods inherited from class Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

S3CrawlManager

public S3CrawlManager()

Creates the S3 crawl manager. Managed by CDI.

Method Details

hydrateAndTrigger

public S3CrawlManager.S3Dispatch hydrateAndTrigger(String accountId,
 String datasourceId,
 String apiKey,
 ai.pipestream.testing.harness.v1.S3SourceConfig source)

Optionally hydrate the bucket, then trigger the crawl. Returns the connector request id and the hydrated-doc count.

Parameters:

accountId - the run's account id (logging/correlation context)

datasourceId - the datasource the crawl is bound to

apiKey - api key attached as x-api-key call metadata on the start-crawl call

source - the S3 source config (connection, bucket, prefix, hydrate flag)

Returns:

the S3CrawlManager.S3Dispatch carrying the request id and the hydrated-doc count