Class S3SampleDocLoader

java.lang.Object
ai.pipestream.module.pipelineprobe.pipelinecrawl.S3SampleDocLoader
@ApplicationScoped public class S3SampleDocLoader extends Object
Resolves the set of documents to upload into S3 before an S3 pipeline-crawl run, when hydrate_sample_data is set.

Primary source: the published sample-doc-types JAR on the classpath — the mixed-format raw corpus (pdf/office/epub/html/email/image/ …) maintained in the sample-documents project. The loader locates that jar via a category-dir anchor, opens it as a FileSystem, walks every regular file, and uploads each (object key = path within the jar, e.g. pdf/simple.pdf). This mirrors how module-parser's TestDocumentLoader consumes the same corpus, and keeps the docs in the jar — no filesystem dependency at deploy time. Bounded by module.testing.sidecar.s3-crawl.max-sample-docs.

Optional dev override: set module.testing.sidecar.s3-crawl.sample-docs-dir to a directory to read raw docs off disk instead (handy when iterating on the corpus locally). Empty by default — the jar is the source of truth.

Graceful fallback: if neither the override dir nor the jar yields files (e.g. the dependency is missing), the loader returns the bundled 9 SampleDocument entries so the path still works.

  • Constructor Details

    • S3SampleDocLoader

      public S3SampleDocLoader()
      Creates the sample-doc loader. Managed by CDI.

  • Method Details

    • resolveSamples

      public List<S3SampleDocLoader.Sample> resolveSamples()
      Resolve the documents to hydrate: optional disk override → sample-doc-types jar → bundled fallback. Never returns more than maxSampleDocs.
      Returns:
      the resolved samples to upload, capped at maxSampleDocs; the bundled fallback set when no override dir or jar yields files