Package ai.platon.pulsar.crawl.fetch.driver

Interface WebDriver

  • All Implemented Interfaces:
    java.io.Closeable , java.lang.AutoCloseable 
    
public interface WebDriver
 implements Closeable

    WebDriver defines a concise interface to visit and interact with web pages, all actions and behaviors are optimized to mimic real people as closely as possible, such as scrolling, clicking, typing text, dragging and dropping, etc.

    The methods in this interface fall into three categories:

    • Control of the browser itself

    • Selection of textContent and attributes of Elements

    • Interact with the webpage

    Key methods:

    • navigateTo: open a page.

    • scrollDown: scroll down on a web page to fully load the page, most modern webpages support lazy loading using ajax tech, where the page content only starts to load when it is scrolled into view.

    • pageSource: retrieve the source code of a webpage.

    • Constructor Detail

    • Method Detail

      • jvm

         abstract JvmWebDriver jvm()

        Returns a JvmWebDriver to support other JVM languages, such as java, clojure, scala, and so on, the other JVM languages might have difficulty to handle kotlin suspend methods.

      • addInitScript

         abstract Unit addInitScript(String script)

        Adds a script which would be evaluated in one of the following scenarios:

        • Whenever the page is navigated.

        The script is evaluated after the document was created but before any of its scripts were run. This is useful to amend the JavaScript environment, e.g. to seed Math.random.

        This function should be invoked before a navigation, usually in an onWillNavigate event handler.

        Parameters:
        script - Javascript source code to add.

      • addBlockedURLs

         abstract Unit addBlockedURLs(List<String> urls)

        Blocks resource URLs from loading.

        Parameters:
        urls - URL patterns to block.

      • addProbabilityBlockedURLs

         abstract Unit addProbabilityBlockedURLs(List<String> urlPatterns)

        Block resource URL loading with a certain probability.

        Parameters:
        urlPatterns - Regular expressions of URLs to block.

      • navigateTo

         abstract Unit navigateTo(String url)

        Returns the main resource response. In case of multiple redirects, the navigation will resolve with the first non-redirect response.

        Parameters:
        url - URL to navigate page to.

      • navigateTo

         abstract Unit navigateTo(NavigateEntry entry)

        Returns the response of the main resource. In case of multiple redirects, the navigation will resolve with the first non-redirect response.

        Parameters:
        entry - NavigateEntry to navigate page to.

      • setTimeouts

         abstract Unit setTimeouts(BrowserSettings browserSettings)

        TODO: NOT IMPLEMENTED

      • currentUrl

         abstract String currentUrl()

        Returns a string representing the current URL that the browser is looking at.

        If the browser failed to return a proper url, return the passed in url to navigate.

        Returns:

        The document's URL without fragment, or the passed in url to navigate.

      • location

         abstract String location()

        Returns the document's location evaluated by javascript.

        In javascript, the window.location, or document.location, is a read-only property returns a Location object, which contains information about the URL of the document and provides methods for changing that URL and loading another URL.

        To retrieve just the URL as a string, the read-only document.URL property can also be used.

      • baseURI

         abstract String baseURI()

        Returns the document's baseURI evaluated by javascript.

        In javascript, the baseURI is a property of Node, it's the absolute base URL of the document containing the node. A baseURI is used to resolve relative URLs.

        The base URL is determined as follows:

        • By default, the base URL is the location of the document (as determined by window.location).

        • If the document has an <base> element, its href attribute is used.

      • pageSource

         abstract String pageSource()

        Returns the source of the last loaded page. If the page has been modified after loading (for example, by Javascript) there is no guarantee that the returned text is that of the modified page.

        TODO: distinguish pageSource and outerHTML

        Returns:

        The source of the current page

      • deleteCookies

         abstract Unit deleteCookies(String name)

        Deletes browser cookies with matching name.

        Parameters:
        name - Name of the cookies to remove.

      • deleteCookies

         abstract Unit deleteCookies(String name, String url, String domain, String path)

        Deletes browser cookies with matching name and url or domain/path pair.

        Parameters:
        name - Name of the cookies to remove.
        url - If specified, deletes all the cookies with the given name where domain and path match provided URL.
        domain - If specified, deletes only cookies with the exact domain.
        path - If specified, deletes only cookies with the exact path.

      • waitForSelector

         abstract Long waitForSelector(String selector)

        Returns when element specified by selector satisfies {@code state} option.

      • waitForSelector

         abstract Long waitForSelector(String selector, Long timeoutMillis)

        Returns when element specified by selector satisfies {@code state} option. Returns the time remaining until timeout.

      • bringToFront

         abstract Unit bringToFront()

        Brings the browser window to the front.

      • focus

         abstract Unit focus(String selector)

        This method fetches an element with selector and focuses it. If there's no element matching selector, nothing to do.

        Parameters:
        selector -

        • A {@link https://developer.mozilla.org/en-US/docs/Web/CSS/CSS_Selectors | selector } of an element to focus. If there are multiple elements satisfying the selector, the first will be focused.

      • type

         abstract Unit type(String selector, String text)

        This method emulates inserting text that doesn't come from a key press.

        Parameters:
        selector -

        • A {@link https://developer.mozilla.org/en-US/docs/Web/CSS/CSS_Selectors | selector } of an element to focus. If there are multiple elements satisfying the selector, the first will be focused.

        text - The text to insert.

      • click

         abstract Unit click(String selector, Integer count)

        This method clicks an element with selector and focuses it. If there's no element matching selector, nothing to do.

        Parameters:
        selector -

        • A {@link https://developer.mozilla.org/en-US/docs/Web/CSS/CSS_Selectors | selector } of an element to focus. If there are multiple elements satisfying the selector, the first will be focused.

      • clickTextMatches

         abstract Unit clickTextMatches(String selector, String pattern, Integer count)

        This method clicks an element with selector whose text content matches pattern, and then focuses it. If there's no element matching selector, or the element's text content doesn't match pattern, nothing to do.

        Parameters:
        selector -

        • A {@link https://developer.mozilla.org/en-US/docs/Web/CSS/CSS_Selectors | selector } of an element to focus. If there are multiple elements satisfying the selector, the first will be focused.

      • check

         abstract Unit check(String selector)

        This method check an element with selector. If there's no element matching selector, nothing to do.

        Parameters:
        selector -

        • A {@link https://developer.mozilla.org/en-US/docs/Web/CSS/CSS_Selectors | selector } of an element to check. If there are multiple elements satisfying the selector, the first will be focused.

      • uncheck

         abstract Unit uncheck(String selector)

        This method uncheck an element with selector. If there's no element matching selector, nothing to do.

        Parameters:
        selector -

        • A {@link https://developer.mozilla.org/en-US/docs/Web/CSS/CSS_Selectors | selector } of an element to uncheck. If there are multiple elements satisfying the selector, the first will be focused.

      • scrollTo

         abstract Unit scrollTo(String selector)

        This method fetches an element with selector, scrolls it into view if needed. If there's no element matching selector, the method does nothing.

        Parameters:
        selector -

        • A selector to search for element to scroll to. If there are multiple elements satisfying the selector, the first will be selected.

      • scrollDown

         abstract Unit scrollDown(Integer count)

        The current page frame scrolls down for count times.

        Parameters:
        count - The times to scroll down.

      • scrollUp

         abstract Unit scrollUp(Integer count)

        The current page frame scrolls up for count times.

        Parameters:
        count - The times to scroll up.

      • scrollToTop

         abstract Unit scrollToTop()

        The current page frame scrolls to the top.

      • scrollToBottom

         abstract Unit scrollToBottom()

        The current page frame scrolls to the bottom.

      • scrollToMiddle

         abstract Unit scrollToMiddle(Float ratio)

        The current page frame scrolls to the middle.

        Parameters:
        ratio - The ratio of the page to scroll to, 0.0 means the top, 1.0 means the bottom.

      • mouseWheelDown

         abstract Unit mouseWheelDown(Integer count, Double deltaX, Double deltaY, Long delayMillis)

        The mouse wheels down for count times.

        Parameters:
        count - The times to wheel down.
        deltaX - The distance to wheel horizontally.
        deltaY - The distance to wheel vertically.
        delayMillis - The delay time in milliseconds.

      • mouseWheelUp

         abstract Unit mouseWheelUp(Integer count, Double deltaX, Double deltaY, Long delayMillis)

        The mouse wheels up for count times.

        Parameters:
        count - The times to wheel up.
        deltaX - The distance to wheel horizontally.
        deltaY - The distance to wheel vertically.
        delayMillis - The delay time in milliseconds.

      • moveMouseTo

         abstract Unit moveMouseTo(Double x, Double y)

        The mouse moves to the position specified by x and y.

        Parameters:
        x - The x coordinate to move to.
        y - The y coordinate to move to.

      • moveMouseTo

         abstract Unit moveMouseTo(String selector, Integer deltaX, Integer deltaY)

        The mouse moves to the element with selector.

        Parameters:
        deltaX - The distance to the left of the element.
        deltaY - The distance to the top of the element.

      • dragAndDrop

         abstract Unit dragAndDrop(String selector, Integer deltaX, Integer deltaY)

        Performs a drag, dragenter, dragover, and drop in sequence.

        Parameters:
        selector -

        • selector of the element to drag from.

        deltaX - The distance to drag horizontally.
        deltaY - The distance to drag vertically.

      • outerHTML

         abstract String outerHTML()

        Returns the document's HTML markup.

        If the document does not exist, returns null.

        Returns:

        The HTML markup of the document.

      • outerHTML

         abstract String outerHTML(String selector)

        Returns the node's HTML markup, the node is located by selector.

        If the node does not exist, returns null.

        Parameters:
        selector - The selector to locate the node.
        Returns:

        The HTML markup of the node.

      • firstText

        @Deprecated(message = "Inappropriate name", replaceWith = @ReplaceWith(imports = {}, expression = "selectFirstTextOrNull(selector)")) String firstText(String selector)

        Returns the node's text content.

      • selectFirstTextOrNull

         abstract String selectFirstTextOrNull(String selector)

        Returns the node's text content, the node is located by selector.

        If the node does not exist, returns null.

        Parameters:
        selector - The selector to locate the node.
        Returns:

        The text content of the node.

      • selectTexts

         abstract List<String> selectTexts(String selector)

        Returns the nodes' text contents.

        If the nodes do not exist, returns an empty list.

        Parameters:
        selector - The selector to locate the nodes.
        Returns:

        The text contents of the nodes.

      • selectFirstAttributeOrNull

         abstract String selectFirstAttributeOrNull(String selector, String attrName)

        Returns the node's attribute value, the node is located by selector, the attribute is attrName.

        If the node does not exist, or the attribute does not exist, returns null.

        Parameters:
        selector - The selector to locate the node.
        attrName - The attribute name to retrieve.
        Returns:

        The attribute value of the node.

      • selectAttributes

         abstract List<String> selectAttributes(String selector, String attrName)

        Returns the nodes' attribute values, the nodes are located by selector, the attribute is attrName.

        If the nodes do not exist, or the attribute does not exist, returns an empty list.

        Parameters:
        selector - The selector to locate the nodes.
        attrName - The attribute name to retrieve.
        Returns:

        The attribute values of the nodes.

      • evaluate

         abstract Object evaluate(String expression)

        Executes JavaScript in the context of the currently selected frame or window. The script fragment provided will be executed as the body of an anonymous function.

        Parameters:
        expression - Javascript expression to evaluate
        Returns:

        Remote object value in case of primitive values or JSON values (if it was requested).

      • evaluateDetail

        @Beta() abstract JsEvaluation evaluateDetail(String expression)

        Executes JavaScript in the context of the currently selected frame or window. The script fragment provided will be executed as the body of an anonymous function.

        Parameters:
        expression - Javascript expression to evaluate
        Returns:

        expression result

      • evaluateSilently

         abstract Object evaluateSilently(String expression)

        Executes JavaScript in the context of the currently selected frame or window. The script fragment provided will be executed as the body of an anonymous function.

        All possible exceptions are suppressed and do not throw.

        Parameters:
        expression - Javascript expression to evaluate
        Returns:

        Remote object value in case of primitive values or JSON values (if it was requested).

      • captureScreenshot

         abstract String captureScreenshot(String selector)

        This method scrolls element into view if needed, and then ake a screenshot of the element.

      • captureScreenshot

         abstract String captureScreenshot(RectD rect)

        This method scrolls element into view if needed, and then ake a screenshot of the element.

      • clickablePoint

         abstract PointD clickablePoint(String selector)

        Calculate the clickable point of an element located by selector. If the element does not exist, or is not clickable, returns null.

      • boundingBox

         abstract RectD boundingBox(String selector)

        Return the bounding box of an element located by selector. If the element does not exist, returns null.

      • newJsoupSession

         abstract Connection newJsoupSession()

        Create a new Jsoup session with the last page's context, which means, the same headers and cookies.

      • loadJsoupResource

         abstract Connection.Response loadJsoupResource(String url)

        Load the url as a resource with Jsoup rather than browser rendering, with the last page's context, which means, the same headers and cookies.

      • pause

         abstract Unit pause()

        Force the page pauses all navigations and PENDING resource fetches. If the page loading stops, the user can still interact with the page, and therefore resources can continue to load.

      • stop

         abstract Unit stop()

        Force the page stop all navigations and RELEASES all resources. Interaction with the stop page results in undefined behavior and the results should not be trusted.

        If a web driver stops, it can later be used to visit new pages.

      • terminate

         abstract Unit terminate()

        Force the page stop all navigations and RELEASES all resources. If a web driver is terminated, it should not be used any more and should be quit as soon as possible.

      • free

         abstract Unit free()

        Mark the driver as free, so it can be used to fetch a new page.

      • startWork

         abstract Unit startWork()

        Mark the driver as working, so it can not be used to fetch another page.

      • retire

         abstract Unit retire()

        Mark the driver as retired, so it can not be used to fetch any page, and should be quit as soon as possible.

      • cancel

         abstract Unit cancel()

        Mark the driver as canceled, so the fetch process should return as soon as possible, and the fetch result should be dropped.

      • getBrowser

         abstract Browser getBrowser()

        The browser of the driver. The browser defines methods and events to manipulate a real browser.

      • getBrowserType

         abstract BrowserType getBrowserType()

        The browser type.

      • getSupportJavascript

         abstract Boolean getSupportJavascript()

        Whether the driver supports javascript. Web drivers such as MockDriver do not support javascript.

      • isRecovered

         abstract Boolean isRecovered()

        Whether the driver is recovered from the browser's tab list.

      • setRecovered

         abstract Unit setRecovered(Boolean isRecovered)

        Whether the driver is recovered from the browser's tab list.

      • isReused

         abstract Boolean isReused()

        Whether the driver is recovered and is reused to serve new tasks.

      • setReused

         abstract Unit setReused(Boolean isReused)

        Whether the driver is recovered and is reused to serve new tasks.

      • isInit

         abstract Boolean isInit()

        Whether the driver is initialized.

      • isReady

         abstract Boolean isReady()

        Whether the driver is ready to work.

      • isRetired

         abstract Boolean isRetired()

        Whether the driver is retired and should be quit as soon as possible.

      • isCanceled

         abstract Boolean isCanceled()

        Whether the current driver task is canceled and should return as soon as possible.