Constructor Detail

• AbstractLoadEventHandlers

  AbstractLoadEventHandlers(UrlFilterEventHandler onNormalize, UrlEventHandler onWillLoad, WebPageEventHandler onWillFetch, WebPageEventHandler onFetched, WebPageEventHandler onWillParse, WebPageEventHandler onWillParseHTMLDocument, WebPageHTMLDocumentEventHandler onHTMLDocumentParsed, WebPageEventHandler onParsed, WebPageEventHandler onLoaded)

Method Detail

• getOnNormalize

   UrlFilterEventHandler getOnNormalize()

  Fires when the URL is about to be normalized.

  URL normalization transforms URLs into a canonical form, such as:

  • Removing fragments (e.g., #section)

  • Removing tracking parameters (e.g., ?utm_source=...)

  • Converting to lowercase

  (String) -> String?

  onNormalize.addLast { url ->
      url.replace(Regex("\\?utm_.*"), "")  // Remove UTM parameters
  }

  Returns:

  The normalized URL, or null to reject the URL

• getOnWillLoad

   UrlEventHandler getOnWillLoad()

  Fires when the URL is about to be loaded.

  This event occurs after normalization but before fetching. Use it to:

  • Log loading attempts

  • Perform final URL validation

  • Add request metadata

  (String) -> String?

  onWillLoad.addLast { url ->
      println("Loading: $url")
      url  // Return URL to continue, or null to skip
  }

• getOnWillFetch

   WebPageEventHandler getOnWillFetch()

  Fires when the URL is about to be fetched.

  This event occurs just before the actual network request or browser navigation. The WebPage object is available but content is not yet loaded.

  (WebPage) -> Any?

  onWillFetch.addLast { page ->
      println("Fetching: ${page.url}, attempt: ${page.fetchCount}")
  }

• getOnFetched

   WebPageEventHandler getOnFetched()

  Fires when the URL has been fetched.

  This event occurs after the page content has been retrieved from the network or browser. The WebPage now contains the raw content.

  (WebPage) -> Any?

  onFetched.addLast { page ->
      println("Fetched: ${page.url}, size: ${page.contentLength} bytes")
  }

• getOnWillParse

   WebPageEventHandler getOnWillParse()

  Fires when the webpage is about to be parsed.

  This event marks the beginning of the parsing phase. The page content is available but not yet converted to a DOM document.

  (WebPage) -> Any?

• getOnWillParseHTMLDocument

   WebPageEventHandler getOnWillParseHTMLDocument()

  Fires when the HTML document is about to be parsed.

  This event occurs just before HTML content is parsed into a DOM structure.

  (WebPage) -> Any?

• getOnHTMLDocumentParsed

   WebPageHTMLDocumentEventHandler getOnHTMLDocumentParsed()

  Fires when the HTML document has been parsed.

  This is the primary event for data extraction. The FeaturedDocument provides CSS selector and DOM manipulation capabilities.

  (WebPage, FeaturedDocument) -> Any?

  onHTMLDocumentParsed.addLast { page: WebPage, document: FeaturedDocument ->
      val title = document.selectFirst("h1")?.text()
      val price = document.selectFirst(".price")?.text()
      saveProduct(title, price)
  }

• getOnParsed

   WebPageEventHandler getOnParsed()

  Fires when the webpage parsing is complete.

  This event occurs after all parsing operations have finished.

  (WebPage) -> Any?

• getOnLoaded

   WebPageEventHandler getOnLoaded()

  Fires when the webpage is fully loaded.

  This is the final load event, occurring after all fetching and parsing is complete. Ideal for cleanup or final processing.

  (WebPage) -> Any?

  onLoaded.addLast { page ->
      println("Completed: ${page.url}, status: ${page.protocolStatus}")
  }

• chain

   AbstractLoadEventHandlers chain(LoadEventHandlers other)

  Chains another load event handler to the tail of this one.

  Parameters:

  other - The load event handlers to chain

  Returns:

  This handler instance for fluent chaining