Class CrawlCleanupService

java.lang.Object
ai.pipestream.module.pipelineprobe.pipelinecrawl.CrawlCleanupService
@ApplicationScoped public class CrawlCleanupService extends Object
Ninja cleanup for pipeline-crawl runs: by default a run deletes every resource it provisioned, so an hourly health-check e2e leaves the platform exactly as it found it. Two entry points:
  • cleanupRun(CrawlCleanupService.RunResources) — per-run teardown, called from the orchestrator's finally (success AND failure paths) unless the request set skip_cleanup. retain_index keeps the OS indices + their IndexPlan for inspection; everything else still goes.
  • sweepLeakedRuns(boolean) — admin sweep over EVERYTHING matching the pipeline-crawl- naming convention: accounts, their drives, datasources, graphs, index plans + indices. Cleans up the backlog left by runs that predate per-run cleanup (or whose sidecar died mid-run).

Every step is best-effort with a per-step audit line — a partial earlier failure never blocks the remaining steps, mirroring OpenSearchTestSetupService.teardown. Deletion order is leaf-first: plan/indices and drives before datasource before account, so a re-run after a partial failure still finds the parents.

  • Field Details

    • CRAWL_PREFIX

      public static final String CRAWL_PREFIX
      Naming-convention prefix every disposable crawl resource carries.
      See Also:

  • Constructor Details

    • CrawlCleanupService

      public CrawlCleanupService()
      Creates the crawl cleanup service. The gRPC stubs and the active-crawl registry are injected by the container; instances are managed by CDI.

  • Method Details

    • cleanupRun

      public List<String> cleanupRun(CrawlCleanupService.RunResources r)
      Per-run ninja teardown. Returns a human audit of what was destroyed/kept.
      Parameters:
      r - the run's provisioned resources to tear down
      Returns:
      an ordered, human-readable audit line per cleanup step (each step best-effort; failures are recorded rather than thrown)

    • sweepLeakedRuns

      public List<String> sweepLeakedRuns(boolean dryRun)
      Admin sweep over every leaked pipeline-crawl-* resource. Accounts of currently-active crawls are skipped. dryRun reports what WOULD be deleted without touching anything.
      Parameters:
      dryRun - when true, report what would be deleted without deleting anything; when false, perform the deletions
      Returns:
      an ordered, human-readable audit; if any crawl is currently active on this sidecar the sweep is refused and the audit says so