Package ai.platon.pulsar.skeleton.crawl

Interface CrawlEventHandlers

  • All Implemented Interfaces:
    
public interface CrawlEventHandlers

    Event handlers during the crawl phase of the webpage lifecycle.

    Crawl event handlers are triggered at the beginning and end of the crawl iteration, wrapping around the load and browse phases. They are ideal for:

    • Filtering or modifying URLs before processing

    • Handling crawl results after page loading

    • Implementing crawl-level logic (e.g., rate limiting, deduplication)

    crawl.onWillLoad → [Load Phase] → [Browse Phase] → crawl.onLoaded
    crawlEventHandlers.apply {
    onWillLoad.addLast { url: UrlAware ->
        // Filter or modify URL before loading
        if (url.url.contains("skip-this")) null else url
    }
    onLoaded.addLast { url, page ->
        // Process results after loading
        println("Loaded: ${url.url}, success: ${page != null}")
    }
}

    • Nested Class Summary

      Nested Classes 
      Modifier and Type Class Description

    • Field Summary

      Fields 
      Modifier and Type Field Description

    • Constructor Summary

      Constructors 
      Constructor Description

    • Enum Constant Summary

      Enum Constants 
      Enum Constant Description

    • Constructor Detail

    • Method Detail

      • chain

         abstract CrawlEventHandlers chain(CrawlEventHandlers other)

        Chains another crawl event handler to the tail of this one.

        Chained handlers execute in order: this handler's callbacks run first, then the other handler's callbacks.

        Parameters:
        other - The crawl event handlers to chain
        Returns:

        This handler instance for fluent chaining

      • getOnWillLoad

         abstract UrlAwareEventHandler getOnWillLoad()

        Fires when the URL is about to be loaded in the crawl loop.

        This is the first event in the page lifecycle. Use this handler to:

        • Filter URLs (return null to skip)

        • Modify URL properties before loading

        • Log or track crawl progress

        (UrlAware) -> UrlAware?

        onWillLoad.addLast { url: UrlAware ->
    if (isBlacklisted(url.url)) null else url
}
        Returns:

        The URL to continue processing, or null to skip this URL

      • getOnLoaded

         abstract UrlAwareWebPageEventHandler getOnLoaded()

        Fires when the URL has been loaded in the crawl loop.

        This is the last event in the page lifecycle. Use this handler to:

        • Process loaded page content

        • Handle load failures (page will be null)

        • Collect statistics or metrics

        (UrlAware, WebPage?) -> Any?

        onLoaded.addLast { url, page ->
    if (page != null) {
        saveToDatabase(page)
    } else {
        logFailure(url)
    }
}