Class EmbedderFixturesRegistrar

java.lang.Object
ai.pipestream.module.pipelineprobe.pipelinecrawl.EmbedderFixturesRegistrar
@ApplicationScoped public class EmbedderFixturesRegistrar extends Object
Idempotent registration of the chunker / embedder / VectorSet fixtures the chunker→embedder→sink Pipeline tab shape needs before its IndexPlan can provision KNN fields.

Without this, the embedder runs and produces vectors but the manager rejects the indexing call with "KNN field not provisioned at bind time: index=...--vs--sentence-10-3--minilm field=vector dim=384" — the plan was created with empty vector_set_ids[] so the per-VS provisioner never ran.

What's registered (matches the names DefaultChunkerConfig hardcodes into the chunker's custom_config so the chunker emits chunks stamped with the same chunk_config_id the manager expects):

  • sentence-10-3 chunker config — the chunker's primary chunking algorithm for body.
  • sentences_internal chunker config — the chunker's always-emit Stage-2 SPRs (per §21.9).
  • minilm embedding config — the only embedder DefaultChunkerConfig wires today.
  • One VectorSet per (chunker, minilm) pair, returned to the caller for inclusion in the IndexPlan's vector_set_ids.

All calls are idempotent: ALREADY_EXISTS responses recover by name lookup. Reruns of the same Pipeline-tab configuration reuse the same rows in the manager DB instead of accumulating per-run duplicates.

Mirrors the same fixture set the legacy E2EPipelineTestService registers — narrowed from the cartesian 2x2x2 to just the one chunker width × one embedder the Pipeline tab actually uses today.

  • Field Details

    • SENTENCE_CHUNKER

      public static final String SENTENCE_CHUNKER
      Chunker config_id for the body-text sentence chunker.
      See Also:

    • SENTENCES_INTERNAL_CHUNKER

      public static final String SENTENCES_INTERNAL_CHUNKER
      Chunker config_id for the always-emitted Stage-2 sentences SPR.
      See Also:

    • MINILM

      public static final String MINILM
      Embedding config_id wired by DefaultChunkerConfig.
      See Also:

    • MINILM_MODEL_ID

      public static final String MINILM_MODEL_ID
      Underlying model identifier for the minilm embedding config.
      See Also:

    • PARAPHRASE_MINILM_L3

      public static final String PARAPHRASE_MINILM_L3
      Second embedder. This is the embedder module's config key (embedder.models.paraphrase-minilm, serving-name paraphrase_MiniLM_L3_v2). It MUST equal the directive's config_id (DirectiveEmbedderResolver does a strict config-id match, no serving-name fallback) AND the manager EmbeddingConfig name, so the produced vectors route to the em_paraphrase-minilm KNN field — exactly the way minilm aligns end-to-end. Lets the e2e exercise the 2-chunker × 2-embedder = 4 vector-set cartesian.
      See Also:

    • PARAPHRASE_MINILM_L3_MODEL_ID

      public static final String PARAPHRASE_MINILM_L3_MODEL_ID
      Underlying model identifier for the paraphrase-minilm embedding config.
      See Also:

    • EMBEDDING_DIM

      public static final int EMBEDDING_DIM
      Vector dimensions. Both minilm and paraphrase_MiniLM_L3_v2 are 384-dim MiniLM-family models; verified live via POST /predictions/<name> against the DJL serving instance on localhost:8090.
      See Also:

  • Constructor Details

    • EmbedderFixturesRegistrar

      public EmbedderFixturesRegistrar()
      Creates the embedder fixtures registrar. Instances are managed by CDI.

  • Method Details

    • registerForEmbedderPipeline

      public List<String> registerForEmbedderPipeline()
      Idempotently registers chunker + embedder + VectorSet fixtures for the chunker→embedder→sink Pipeline shape.
      Returns:
      VectorSet ids to drop into the IndexPlan's vector_set_ids field — one per (chunker, minilm) pair the chunker emits.

    • registerSemanticConfigsForIndex

      public void registerSemanticConfigsForIndex(String indexName, ai.pipestream.opensearch.v1.IndexingStrategy strategy)
      Registers the semantic-graph SemanticConfigs and eagerly provisions their centroid side-indices on indexName, so the sink can write the document/section/paragraph centroid SPRs semantic-graph emits. One config per embedder ("minilm", "paraphrase-minilm"); each AssignSemanticConfigToIndex provisions that embedder's em_<embedder> field on every centroid index (CHUNK_COMBINED), so together they cover the 2×2 of centroid fields. Without this the sink fails with "KNN field not provisioned at bind time … Call AssignSemanticConfigToIndex first".

      Idempotent: a pre-existing SemanticConfig is reused; assign re-runs provisioning (itself idempotent).

      Parameters:
      indexName - base index (idx-<graphId>) the centroids hang off
      strategy - the indexing strategy used when provisioning the centroid side-indices