Package ai.platon.pulsar.skeleton.crawl

Interface LoadEventHandlers

  • All Implemented Interfaces:
    
public interface LoadEventHandlers

    Event handlers during the loading phase of the webpage lifecycle.

    Load event handlers manage URL normalization, fetching, and parsing operations. They are triggered in sequence during the page loading process.

    onNormalize → onWillLoad → onWillFetch → [Browser Phase] → onFetched →
onWillParse → onWillParseHTMLDocument → onHTMLDocumentParsed → onParsed → onLoaded
    loadEventHandlers.apply {
    onNormalize.addLast { url -> url.replace(Regex("\\?utm_.*"), "") }
    onHTMLDocumentParsed.addLast { page, doc ->
        val title = doc.selectFirst("h1")?.text()
        println("Title: $title")
    }
    onLoaded.addLast { page -> println("Loaded: ${page.url}") }
}

    • Constructor Detail

    • Method Detail

      • chain

         abstract LoadEventHandlers chain(LoadEventHandlers other)

        Chains another load event handler to the tail of this one.

        Parameters:
        other - The load event handlers to chain
        Returns:

        This handler instance for fluent chaining

      • getOnNormalize

         abstract UrlFilterEventHandler getOnNormalize()

        Fires when the URL is about to be normalized.

        URL normalization transforms URLs into a canonical form, such as:

        • Removing fragments (e.g., #section)

        • Removing tracking parameters (e.g., ?utm_source=...)

        • Converting to lowercase

        (String) -> String?

        onNormalize.addLast { url ->
    url.replace(Regex("\\?utm_.*"), "")  // Remove UTM parameters
}
        Returns:

        The normalized URL, or null to reject the URL

      • getOnWillLoad

         abstract UrlEventHandler getOnWillLoad()

        Fires when the URL is about to be loaded.

        This event occurs after normalization but before fetching. Use it to:

        • Log loading attempts

        • Perform final URL validation

        • Add request metadata

        (String) -> String?

        onWillLoad.addLast { url ->
    println("Loading: $url")
    url  // Return URL to continue, or null to skip
}

      • getOnWillFetch

         abstract WebPageEventHandler getOnWillFetch()

        Fires when the URL is about to be fetched.

        This event occurs just before the actual network request or browser navigation. The WebPage object is available but content is not yet loaded.

        (WebPage) -> Any?

        onWillFetch.addLast { page ->
    println("Fetching: ${page.url}, attempt: ${page.fetchCount}")
}

      • getOnFetched

         abstract WebPageEventHandler getOnFetched()

        Fires when the URL has been fetched.

        This event occurs after the page content has been retrieved from the network or browser. The WebPage now contains the raw content.

        (WebPage) -> Any?

        onFetched.addLast { page ->
    println("Fetched: ${page.url}, size: ${page.contentLength} bytes")
}

      • getOnWillParse

         abstract WebPageEventHandler getOnWillParse()

        Fires when the webpage is about to be parsed.

        This event marks the beginning of the parsing phase. The page content is available but not yet converted to a DOM document.

        (WebPage) -> Any?

      • getOnWillParseHTMLDocument

         abstract WebPageEventHandler getOnWillParseHTMLDocument()

        Fires when the HTML document is about to be parsed.

        This event occurs just before HTML content is parsed into a DOM structure.

        (WebPage) -> Any?

      • getOnHTMLDocumentParsed

         abstract WebPageHTMLDocumentEventHandler getOnHTMLDocumentParsed()

        Fires when the HTML document has been parsed.

        This is the primary event for data extraction. The FeaturedDocument provides CSS selector and DOM manipulation capabilities.

        (WebPage, FeaturedDocument) -> Any?

        onHTMLDocumentParsed.addLast { page: WebPage, document: FeaturedDocument ->
    val title = document.selectFirst("h1")?.text()
    val price = document.selectFirst(".price")?.text()
    saveProduct(title, price)
}

      • getOnParsed

         abstract WebPageEventHandler getOnParsed()

        Fires when the webpage parsing is complete.

        This event occurs after all parsing operations have finished.

        (WebPage) -> Any?

      • getOnLoaded

         abstract WebPageEventHandler getOnLoaded()

        Fires when the webpage is fully loaded.

        This is the final load event, occurring after all fetching and parsing is complete. Ideal for cleanup or final processing.

        (WebPage) -> Any?

        onLoaded.addLast { page ->
    println("Completed: ${page.url}, status: ${page.protocolStatus}")
}