Package ai.platon.pulsar.skeleton.crawl.fetch.driver

Class AbstractWebDriver

    • Method Detail

      • isRecyclable

         final Boolean isRecyclable()

        If a driver is recyclable, it should be closed after the task is done, and add to a standby pool, so it can be reused for other tasks.

        If a driver is not recyclable, it should be kept open and not be closed, and not be used by any other tasks.

      • setRecyclable

         final Unit setRecyclable(Boolean isRecyclable)

        If a driver is recyclable, it should be closed after the task is done, and add to a standby pool, so it can be reused for other tasks.

        If a driver is not recyclable, it should be kept open and not be closed, and not be used by any other tasks.

      • getIgnoreDOMFeatures

         final Boolean getIgnoreDOMFeatures()

        Whether the driver should calculate the DOM features.

        If true, the DOM features should be calculated using pulsar_utils.compute().

      • setIgnoreDOMFeatures

         final Unit setIgnoreDOMFeatures(Boolean ignoreDOMFeatures)

        Whether the driver should calculate the DOM features.

        If true, the DOM features should be calculated using pulsar_utils.compute().

      • getBrowser

         AbstractBrowser getBrowser()

        The browser of the driver. The browser defines methods and events to manipulate a real browser.

      • free

         final Unit free()

        Mark the driver as free, so it can be used to fetch a new page.

      • startWork

         final Unit startWork()

        Mark the driver as working, so it can not be used to do any another tasks.

      • retire

         final Unit retire()

        Mark the driver as retired, so it can not be used to fetch any page, and should be quit as soon as possible.

      • cancel

         final Unit cancel()

        Mark the driver as canceled, so the fetch process should return as soon as possible, and the fetch result should be dropped.

      • jvm

         JvmWebDriver jvm()

        Returns a JvmWebDriver to support other JVM languages, such as java, clojure, scala, and so on, the other JVM languages might have difficulty to handle kotlin suspend methods.

      • addInitScript

         Unit addInitScript(String script)

        Adds a script which would be evaluated in one of the following scenarios:

        • Whenever the page is navigated.

        The script is evaluated after the document was created but before any of its scripts were run. This is useful to amend the JavaScript environment, e.g. to seed Math.random.

        This function should be invoked before a navigation, usually in an onWillNavigate event handler.

        Parameters:
        script - Javascript source code to add.

      • navigateTo

         Unit navigateTo(String url)

        Navigates current page to the given URL.

        driver.navigateTo("https://www.example.com")
driver.waitForNavigation()
        Parameters:
        url - URL to navigate page to.

      • currentUrl

         String currentUrl()

        Returns a string representing the current URL that the browser is looking at. The current url is always the main frame's document.documentURI if the browser succeed to return it, and is displayed in the browser's address bar.

        If the browser failed to return a proper url, returns the passed in url to navigate, just like a real user enter a url in the address bar but the browser failed to load the page.

        NOTE: the url can be non-standard, for example:

        about:blank chrome://newtab chrome://settings

        Returns:

        A string containing the URL of the document, or the passed in url to navigate.

      • url

         String url()

        The URL read-only property of the Document interface returns the document location as a string.

        This property equals to javascript document.URL. The document.URL property returns the same value. The document.documentURI property can be used on any document types, while the document.URL property can only be used on HTML documents.

        Returns:

        A string containing the URL of the document.

      • documentURI

         String documentURI()

        Returns the document location as a string.

        This property equals to javascript document.documentURI.

        The document.URL property which returns the same value. The document.documentURI property can be used on any document types, while the document.URL property can only be used on HTML documents.

        Returns:

        The document's documentURI.

      • baseURI

         String baseURI()

        Returns the document's baseURI.

        The baseURI is a property of Node, it's the absolute base URL of the document containing the node. A baseURI is used to resolve relative URLs.

        The base URL is determined as follows:

        • By default, the base URL is the location of the document (as determined by window.location).

        • If the document has an <base> element, its href attribute is used.

        Returns:

        The document's baseURI.

      • referrer

         String referrer()

        The referrer property returns the URI of the page that linked to this page.

        The value is an empty string if the user navigated to the page directly (not through a link, but, for example, by using a bookmark).

        Inside an <iframe>, the referrer will initially be set to the same value as the href of the parent window's Window.location.

        Returns:

        The document's referrer.

      • chat

         ModelResponse chat(String prompt, String selector)

        Chat with the AI model about the specified element.

        Parameters:
        prompt - The prompt to chat with
        selector - The selector to find the element
        Returns:

        The response from the model

      • instruct

         InstructionResult instruct(String prompt)

        Instructs the webdriver to perform a series of actions based on the given prompt. This function converts the prompt into a sequence of webdriver actions, which are then executed.

        Parameters:
        prompt - The textual prompt that describes the actions to be performed by the webdriver.
        Returns:

        The response from the model, though in this implementation, the return value is not explicitly used.

      • evaluate

         <T extends Any> T evaluate(String expression, T defaultValue)

        Executes JavaScript in the context of the currently selected frame or window. If the result is not Javascript object, it is not returned.

        val title = driver.evaluate("document.title", "Untitled")
        Parameters:
        expression - Javascript expression to evaluate
        Returns:

        Remote object value in case of primitive values or null.

      • evaluateValue

         <T extends Any> T evaluateValue(String expression, T defaultValue)

        Executes JavaScript in the context of the currently selected frame or window. If the result is a Javascript object, it is returned as a JSON object.

        val title = driver.evaluate("document.title", "Untitled")
        Parameters:
        expression - Javascript expression to evaluate
        Returns:

        Remote object value in case of primitive values or JSON values (if it was requested).

      • isVisible

         Boolean isVisible(String selector)

        Returns whether the element is visible.

        driver.isVisible("input[name='q']")
        Parameters:
        selector -

        • The selector of the element to check.

        Returns:

        Whether the element is visible.

      • isChecked

         Boolean isChecked(String selector)

        Returns whether the element is checked.

        driver.isChecked("input[name='agree']")
        Parameters:
        selector -

        • The selector of the element to check.

        Returns:

        Whether the element is checked.

      • scrollDown

         Unit scrollDown(Integer count)

        The current page frame scrolls down for count times.

        driver.scrollDown(3)
        Parameters:
        count - The times to scroll down.

      • scrollUp

         Unit scrollUp(Integer count)

        The current page frame scrolls up for count times.

        driver.scrollUp(3)
        Parameters:
        count - The times to scroll up.

      • scrollToTop

         Unit scrollToTop()

        The current page frame scrolls to the top.

        driver.scrollToTop()

      • scrollToBottom

         Unit scrollToBottom()

        The current page frame scrolls to the bottom.

        driver.scrollToBottom()

      • scrollToMiddle

         Unit scrollToMiddle(Double ratio)

        The current page frame scrolls to the middle.

        driver.scrollToMiddle(0.2)
driver.scrollToMiddle(0.5)
driver.scrollToMiddle(0.8)
        Parameters:
        ratio - The ratio of the page to scroll to, 0.0 means the top, 1.0 means the bottom.

      • scrollToScreen

         Unit scrollToScreen(Double screenNumber)

        The current page frame scrolls to the middle.

        driver.scrollToScreen(0.0)
driver.scrollToScreen(0.5)
driver.scrollToScreen(1.5)
driver.scrollToScreen(3.0)
        Parameters:
        screenNumber - The screen number of the page to scroll to (0-based).

      • clickNthAnchor

         String clickNthAnchor(Integer n, String rootSelector)

        Clicks the nth anchor element in the DOM.

        This function searches for all anchor (<a>) elements within the specified root element, and clicks the nth anchor element (0-based index). If the anchor element exists, it returns the href attribute of the clicked anchor element. If the element does not exist, it returns null.

        driver.clickNthAnchor(100, "body")
        Parameters:
        n - The index of the anchor element to click (0-based).
        rootSelector - The CSS selector of the root element to search within (default is "body").
        Returns:

        The href attribute of the clicked anchor element, or null if the element does not exist.

      • outerHTML

         String outerHTML()

        Returns the document's HTML markup.

        val html = driver.outerHTML()

        If the document does not exist, returns null.

        Returns:

        The HTML markup of the document.

      • outerHTML

         String outerHTML(String selector)

        Returns the node's HTML markup, the node is located by selector.

        If the node does not exist, returns null.

        val html = driver.outerHTML("h2.title")
        Parameters:
        selector - The selector to locate the node.
        Returns:

        The HTML markup of the node.

      • selectFirstTextOrNull

         String selectFirstTextOrNull(String selector)

        Returns the node's text content, the node is located by selector.

        If the node does not exist, returns null.

        val text = driver.selectFirstTextOrNull("h2.title")
        Parameters:
        selector - The selector to locate the node.
        Returns:

        The text content of the node.

      • selectTextAll

         List<String> selectTextAll(String selector)

        Returns a list of text contents of all the elements matching the specified selector within the page.

        If no elements match the selector, returns an empty list.

        val texts = driver.selectTextAll("h2")
        Parameters:
        selector - The selector to locate the nodes.
        Returns:

        The text contents of the nodes.

      • selectFirstAttributeOrNull

         String selectFirstAttributeOrNull(String selector, String attrName)

        Returns the node's attribute value, the node is located by selector, the attribute is attrName.

        If the node does not exist, or the attribute does not exist, returns null.

        val classes = driver.selectFirstAttributeOrNull("h2.title", "class")
        Parameters:
        selector - The selector to locate the node.
        attrName - The attribute name to retrieve.
        Returns:

        The attribute value of the node.

      • selectAttributes

         Map<String, String> selectAttributes(String selector)

        Returns the node's attribute values, the node is located by selector.

        If the node do not exist, or the attribute does not exist, returns an empty list.

        val classes = driver.selectAttributes("h2.title", "class")
        Parameters:
        selector - The selector to locate the nodes.
        Returns:

        The attribute pairs of the nodes.

      • selectAttributeAll

         List<String> selectAttributeAll(String selector, String attrName, Integer start, Integer limit)

        Returns the nodes' attribute values, the nodes are located by selector, the attribute is attrName.

        If the nodes do not exist, or the attribute does not exist, returns an empty list.

        val classes = driver.selectAttributeAll("h2.title", "class")
        Parameters:
        selector - The selector to locate the nodes.
        attrName - The attribute name to retrieve.
        start - The offset of the first node to select.
        limit - The maximum number of nodes to select.
        Returns:

        The attribute values of the nodes.

      • setAttribute

         Unit setAttribute(String selector, String attrName, String attrValue)

        Set the attribute of an element located by selector.

        driver.setAttribute("h2.title", "class", "header")
        Parameters:
        selector - The CSS query to select an element.
        attrName - The attribute name to set.
        attrValue - The attribute value to set.

      • setAttributeAll

         Unit setAttributeAll(String selector, String attrName, String attrValue)

        Set the attribute of all elements matching the CSS query.

        driver.setAttributeAll("h2.title", "class", "header")
        Parameters:
        selector - The CSS query to select elements.
        attrName - The attribute name to set.
        attrValue - The attribute value to set.

      • selectFirstPropertyValueOrNull

         String selectFirstPropertyValueOrNull(String selector, String propName)

        Returns the node's property value, the node is located by selector, the property is attrName.

        If the node does not exist, or the property does not exist, returns null.

        val classes = driver.selectFirstPropertyOrNull("input#input", "value")
        Parameters:
        selector - The selector to locate the node.
        propName - The property name to retrieve.
        Returns:

        The property value of the node.

      • selectPropertyValueAll

         List<String> selectPropertyValueAll(String selector, String propName, Integer start, Integer limit)

        Returns the nodes' property values, the nodes are located by selector, the property is propName.

        If the nodes do not exist, or the property does not exist, returns an empty list.

        val classes = driver.selectPropertyAll("input#input", "value")
        Parameters:
        selector - The selector to locate the nodes.
        propName - The property name to retrieve.
        start - The offset of the first node to select.
        limit - The maximum number of nodes to select.
        Returns:

        The property values of the nodes.

      • setProperty

         Unit setProperty(String selector, String propName, String propValue)

        Set the property of an element located by selector.

        driver.setProperty("input#input", "value")
        Parameters:
        selector - The CSS query to select an element.
        propName - The property name to set.
        propValue - The property value to set.

      • setPropertyAll

         Unit setPropertyAll(String selector, String propName, String propValue)

        Set the property of all elements matching the CSS query.

        driver.setPropertyAll("input#input", "value")
        Parameters:
        selector - The CSS query to select elements.
        propName - The property name to set.
        propValue - The property value to set.

      • selectHyperlinks

         List<Hyperlink> selectHyperlinks(String selector, Integer offset, Integer limit)

        Find hyperlinks in elements matching the CSS query.

        Parameters:
        selector - The CSS query to select elements.
        offset - The offset of the first element to select.
        limit - The maximum number of elements to select.

      • selectAnchors

         List<GeoAnchor> selectAnchors(String selector, Integer offset, Integer limit)

        Find image elements matching the CSS query.

        Parameters:
        selector - The CSS query to select elements.
        offset - The offset of the first element to select.
        limit - The maximum number of elements to select.

      • selectImages

         List<String> selectImages(String selector, Integer offset, Integer limit)

        Find image elements matching the CSS query.

        Parameters:
        selector - The CSS query to select elements.
        offset - The offset of the first element to select.
        limit - The maximum number of elements to select.

      • clickTextMatches

         Unit clickTextMatches(String selector, String pattern, Integer count)

        This method clicks an element with selector whose text content matches pattern, and then focuses it. If there's no element matching selector, or the element's text content doesn't match pattern, nothing to do.

        driver.clickTextMatches("button", "submit")
        Parameters:
        selector -

        • A selector of an element to focus. If there are multiple elements satisfying the selector, the first will be focused.

        pattern - The pattern to match the text content.
        count - The number of times to click.

      • clickMatches

         Unit clickMatches(String selector, String attrName, String pattern, Integer count)

        This method clicks an element with selector whose attribute name is attrName and value matches pattern, and then focuses it. If there's no element matching selector, or the element has no attribute attrName, or the element's attribute value doesn't match pattern, nothing to do.

        driver.clickAttributeMatches("button", "type", "submit")
        Parameters:
        selector -

        • A selector of an element to focus. If there are multiple elements satisfying the selector, the first will be focused.

        attrName - The attribute name to match.
        pattern - The pattern to match the text content.
        count - The number of times to click.

      • check

         Unit check(String selector)

        This method check an element with selector. If there's no element matching selector, nothing to do.

        Parameters:
        selector -

        • A selector of an element to check. If there are multiple elements satisfying the selector, the first will be checked.

      • uncheck

         Unit uncheck(String selector)

        This method uncheck an element with selector. If there's no element matching selector, nothing to do.

        Parameters:
        selector -

        • A selector of an element to uncheck. If there are multiple elements satisfying the selector, the first will be focused.

      • waitForSelector

         Duration waitForSelector(String selector)

        Wait until the element identified by the selector becomes present in the DOM or timeout.

        val remainingTime = driver.waitForSelector("h2.title")
        Parameters:
        selector - The selector of the element to wait for.
        Returns:

        The remaining time until timeout when the element becomes present.

      • waitForSelector

         Duration waitForSelector(String selector, SuspendFunction0<Unit> action)

        Wait for the element identified by the selector to become present in the DOM, or until timeout. This method periodically checks for the existence of the element. If the element is not found during a check, the action will be executed, such as scrolling the page down.

        val remainingTime = driver.waitForSelector("h2.title") {
 driver.scrollDown()
}
        Parameters:
        action - The action to execute when the element is not found.

      • waitForNavigation

         Duration waitForNavigation(String oldUrl)

        Wait until the current url changes or timeout.

        val url = "https://www.example.com"
driver.navigateTo(url)
var remainingTime = driver.waitForNavigation()
if (remainingTime 0) {
  driver.click("a[href='/next']")
  remainingTime = driver.waitForNavigation(url)
}

      • waitForNavigation

         Duration waitForNavigation(String oldUrl, Duration timeout)

        Wait until the current url changes or timeout.

        val timeout = Duration.ofSeconds(30)
val url = "https://www.example.com"
driver.navigateTo(url)
var remainingTime = driver.waitForNavigation(timeout)
if (remainingTime 0) {
  driver.click("a[href='/next']")
  remainingTime = driver.waitForNavigation(url, timeout)
}
        Parameters:
        timeout - The maximum time to wait for the url to change.

      • waitUntil

         Duration waitUntil(SuspendFunction0<Boolean> predicate)

        Wait until the predicate returns true.

        val remainingTime = driver.waitUntil {
   driver.exists("h2.title")
}
        Parameters:
        predicate - The predicate to check.
        Returns:

        The remaining time until timeout when the predicate returns true.

      • waitUntil

         Duration waitUntil(Duration timeout, SuspendFunction0<Boolean> predicate)

        Wait until the predicate returns true.

        val remainingTime = driver.waitUntil(Duration.ofSeconds(10)) {
   driver.exists("h2.title")
}
        Parameters:
        timeout - The maximum time to wait for the predicate to return true.
        predicate - The predicate to check.
        Returns:

        The remaining time until timeout when the predicate returns true.

      • newJsoupSession

         Connection newJsoupSession()

        Create a new Jsoup session with the last page's context, which means, the same headers and cookies.

        Returns:

        The Jsoup session.

      • loadJsoupResource

         Connection.Response loadJsoupResource(String url)

        Load the url as a resource with Jsoup rather than browser rendering, with the last page's context, which means, the same headers and cookies.

        Parameters:
        url - The URL to load.
        Returns:

        The Jsoup response.

      • loadResource

         NetworkResourceResponse loadResource(String url)

        Load the url as a resource without browser rendering, with the last page's context, which means, the same headers and cookies.

        Parameters:
        url - The URL to load.
        Returns:

        The network resource response.

      • terminate

         Unit terminate()

        Force the page stop all navigations and RELEASES all resources. If a web driver is terminated, it should not be used any more and should be quit as soon as possible.

      • randomDelayMillis

         final Long randomDelayMillis(String action, IntRange fallback)

        Generate a random delay in milliseconds.

        The generated delay time is an int random value uniformly distributed in a specified range.

        The delay range should be in 1, 10000, and the default range is 500, 1000.