All Implemented Interfaces:: ai.platon.pulsar.crawl.fetch.driver.JvmWebDriver , ai.platon.pulsar.crawl.fetch.driver.WebDriver , java.io.Closeable , java.lang.AutoCloseable , kotlin.Comparable


public abstract class AbstractWebDriver
extends AbstractJvmWebDriver implements Comparable<AbstractWebDriver>, WebDriver, JvmWebDriver

Nested Class Summary

Nested Classes
Modifier and Type	Class	Description

Field Summary

Fields
Modifier and Type	Field	Description
`private Duration`	`idleTimeout`
`private Duration`	`waitForElementTimeout`
`private final String`	`name`
`private final Function1<String, Long>`	`delayPolicy`
`private NavigateEntry`	`navigateEntry`
`private final NavigateHistory`	`navigateHistory`
`private final Boolean`	`supportJavascript`
`private final Boolean`	`isMockedPageSource`
`private Boolean`	`isRecovered`
`private Boolean`	`isReused`
`private final AtomicReference<WebDriver.State>`	`state`
`private final Map<String, Object>`	`data`
`private Instant`	`lastActiveTime`
`private final Boolean`	`isInit`
`private final Boolean`	`isReady`
`private final Boolean`	`isWorking`
`private final Boolean`	`isRetired`
`private final Boolean`	`isQuit`
`private final Boolean`	`isCanceled`
`private final Boolean`	`isCrashed`
`private final Map<String, Object>`	`mainRequestHeaders`
`private final List<Map<String, String>>`	`mainRequestCookies`
`private final Integer`	`mainResponseStatus`
`private final String`	`mainResponseStatusText`
`private final Map<String, Object>`	`mainResponseHeaders`
`private final Browser`	`browser`
`private final Integer`	`id`
`private final BrowserType`	`browserType`
`private final Boolean`	`isIdle`

Constructor Summary

Constructors
Constructor	Description
`AbstractWebDriver(Browser browser, Integer id)`

Enum Constant Summary

Enum Constants
Enum Constant	Description

Method Summary

Modifier and Type	Method	Description
`Duration`	`getIdleTimeout()`	The idle timeout.
`Unit`	`setIdleTimeout(Duration idleTimeout)`	The idle timeout.
`Duration`	`getWaitForElementTimeout()`	The timeout to wait for an element to appear.
`Unit`	`setWaitForElementTimeout(Duration waitForElementTimeout)`	The timeout to wait for an element to appear.
`String`	`getName()`	The driver name.
`Function1<String, Long>`	`getDelayPolicy()`
`NavigateEntry`	`getNavigateEntry()`	The current navigation entry.
`Unit`	`setNavigateEntry(NavigateEntry navigateEntry)`	The current navigation entry.
`NavigateHistory`	`getNavigateHistory()`	The navigation history.
`Boolean`	`getSupportJavascript()`	Whether the driver supports javascript.
`Boolean`	`isMockedPageSource()`	Whether the page source is mocked.
`Boolean`	`isRecovered()`	Whether the driver is recovered from the browser's tab list.
`Unit`	`setRecovered(Boolean isRecovered)`	Whether the driver is recovered from the browser's tab list.
`Boolean`	`isReused()`	Whether the driver is recovered and is reused to serve new tasks.
`Unit`	`setReused(Boolean isReused)`	Whether the driver is recovered and is reused to serve new tasks.
`AtomicReference<WebDriver.State>`	`getState()`	The state of the driver.
`Map<String, Object>`	`getData()`	The associated data.
`Instant`	`getLastActiveTime()`	The time of the last active action.
`Unit`	`setLastActiveTime(Instant lastActiveTime)`	The time of the last active action.
`Boolean`	`isInit()`	Whether the driver is initialized.
`Boolean`	`isReady()`	Whether the driver is ready to work.
`Boolean`	`isWorking()`	Whether the driver is working.
`Boolean`	`isRetired()`	Whether the driver is retired and should be quit as soon as possible.
`Boolean`	`isQuit()`	Whether the driver is quit.
`Boolean`	`isCanceled()`	Whether the current driver task is canceled and should return as soon as possible.
`Boolean`	`isCrashed()`	Whether the driver is crashed.
`Map<String, Object>`	`getMainRequestHeaders()`	The main request's http headers.
`List<Map<String, String>>`	`getMainRequestCookies()`	The main request's http cookies.
`Integer`	`getMainResponseStatus()`	The main response's status.
`String`	`getMainResponseStatusText()`	The main response's status text.
`Map<String, Object>`	`getMainResponseHeaders()`	The main response's http headers.
`Browser`	`getBrowser()`	The browser of the driver.
`Integer`	`getId()`	The driver id.
`Unit`	`free()`	Mark the driver as free, so it can be used to fetch a new page.
`Unit`	`startWork()`	Mark the driver as working, so it can not be used to fetch another page.
`Unit`	`retire()`	Mark the driver as retired, so it can not be used to fetch any page, and should be quit as soon as possible.
`Unit`	`cancel()`	Mark the driver as canceled, so the fetch process should return as soon as possible, and the fetch result should be dropped.
`JvmWebDriver`	`jvm()`	Returns a JvmWebDriver to support other JVM languages, such as java, clojure, scala, and so on, the other JVM languages might have difficulty to handle kotlin suspend methods.
`Unit`	`navigateTo(String url)`	Returns the main resource response.
`String`	`location()`	Returns the document's location evaluated by javascript.
`String`	`baseURI()`	Returns the document's baseURI evaluated by javascript.
`Long`	`waitForSelector(String selector, Long timeoutMillis)`	Returns when element specified by selector satisfies {@code state} option.
`Long`	`waitForSelector(String selector)`	Returns when element specified by selector satisfies {@code state} option.
`Long`	`waitForNavigation()`
`Long`	`waitForNavigation(Long timeoutMillis)`
`Object`	`evaluateSilently(String expression)`	Executes JavaScript in the context of the currently selected frame or window.
`Boolean`	`isVisible(String selector)`	Returns whether the element is visible.
`Boolean`	`isChecked(String selector)`	Returns whether the element is checked.
`Unit`	`scrollDown(Integer count)`	The current page frame scrolls down for count times.
`Unit`	`scrollUp(Integer count)`	The current page frame scrolls up for count times.
`Unit`	`scrollToTop()`	The current page frame scrolls to the top.
`Unit`	`scrollToBottom()`	The current page frame scrolls to the bottom.
`Unit`	`scrollToMiddle(Float ratio)`	The current page frame scrolls to the middle.
`String`	`clickNthAnchor(Integer n, String rootSelector)`
`String`	`outerHTML()`	Returns the document's HTML markup.
`String`	`outerHTML(String selector)`	Returns the node's HTML markup, the node is located by selector.
`String`	`selectFirstTextOrNull(String selector)`	Returns the node's text content, the node is located by selector.
`List<String>`	`selectTexts(String selector)`	Returns the nodes' text contents.
`String`	`selectFirstAttributeOrNull(String selector, String attrName)`	Returns the node's attribute value, the node is located by selector, the attribute is attrName.
`List<String>`	`selectAttributes(String selector, String attrName)`	Returns the nodes' attribute values, the nodes are located by selector, the attribute is attrName.
`Unit`	`clickTextMatches(String selector, String pattern, Integer count)`	This method clicks an element with selector whose text content matches pattern, and then focuses it.
`Unit`	`clickMatches(String selector, String attrName, String pattern, Integer count)`
`Unit`	`check(String selector)`	This method check an element with selector.
`Unit`	`uncheck(String selector)`	This method uncheck an element with selector.
`Connection`	`newJsoupSession()`	Create a new Jsoup session with the last page's context, which means, the same headers and cookies.
`Connection.Response`	`loadJsoupResource(String url)`	Load the url as a resource with Jsoup rather than browser rendering, with the last page's context, which means, the same headers and cookies.
`NetworkResourceResponse`	`loadResource(String url)`	Load url as a resource without browser rendering, with the last page's context, which means, the same headers and cookies.
`Boolean`	`equals(Object other)`
`Integer`	`hashCode()`
`Integer`	`compareTo(AbstractWebDriver other)`
`String`	`toString()`

Methods inherited from class ai.platon.pulsar.crawl.fetch.driver.WebDriver
addBlockedURLs, addInitScript, addProbabilityBlockedURLs, allTexts, awaitTermination, boundingBox, bringToFront, captureScreenshot, captureScreenshot, clearBrowserCookies, click, clickMatches, clickablePoint, currentUrl, deleteCookies, deleteCookies, dragAndDrop, evaluate, evaluateDetail, exists, firstText, focus, getBrowserType, getCookies, isHidden, isIdle, mouseWheelDown, mouseWheelUp, moveMouseTo, moveMouseTo, navigateTo, pageSource, pause, scrollTo, setTimeouts, stop, terminate, type, visible, waitForNavigation, waitForPage, waitForSelector
Methods inherited from class ai.platon.pulsar.crawl.fetch.driver.AbstractJvmWebDriver
addBlockedURLsAsync, addInitScriptAsync, boundingBoxAsync, bringToFrontAsync, captureScreenshotAsync, captureScreenshotAsync, checkAsync, clickAsync, clickMatchesAsync, clickMatchesAsync, clickNthAnchorAsync, clickablePointAsync, currentUrlAsync, dragAndDropAsync, evaluateAsync, evaluateSilentlyAsync, existsAsync, getCookiesAsync, isCheckedAsync, isHiddenAsync, isVisibleAsync, loadResourceAsync, mainRequestCookiesAsync, mainRequestHeadersAsync, mouseWheelDownAsync, mouseWheelUpAsync, moveMouseToAsync, navigateToAsync, navigateToAsync, newJsoupSessionAsync, outerHTMLAsync, pageSourceAsync, pauseAsync, scrollDownAsync, scrollToAsync, scrollToBottomAsync, scrollToMiddleAsync, scrollToTopAsync, scrollUpAsync, selectAttributesAsync, selectFirstAttributeOptionalAsync, selectFirstAttributeOrNullAsync, selectFirstTextOptionalAsync, selectFirstTextOrNullAsync, selectTextsAsync, setTimeoutsAsync, stopAsync, terminateAsync, typeAsync, uncheckAsync, visibleAsync, waitForNavigationAsync, waitForNavigationAsync, waitForNavigationAsync, waitForSelectorAsync, waitForSelectorAsync, waitForSelectorAsync
Methods inherited from class ai.platon.pulsar.crawl.fetch.driver.JvmWebDriver
clickAsync, clickMatchesAsync, clickMatchesAsync
Methods inherited from class java.io.Closeable
close
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Constructor Detail
  - AbstractWebDriver
```
AbstractWebDriver(Browser browser, Integer id)
```
- Method Detail
  - getIdleTimeout
```
 Duration getIdleTimeout()
```
    The idle timeout.
  - setIdleTimeout
```
 Unit setIdleTimeout(Duration idleTimeout)
```
    The idle timeout.
  - getWaitForElementTimeout
```
 Duration getWaitForElementTimeout()
```
    The timeout to wait for an element to appear.
  - setWaitForElementTimeout
```
 Unit setWaitForElementTimeout(Duration waitForElementTimeout)
```
    The timeout to wait for an element to appear.
  - getName
```
 String getName()
```
    The driver name.
  - getDelayPolicy
```
 Function1<String, Long> getDelayPolicy()
```
  - getNavigateEntry
```
 NavigateEntry getNavigateEntry()
```
    The current navigation entry.
  - setNavigateEntry
```
 Unit setNavigateEntry(NavigateEntry navigateEntry)
```
    The current navigation entry.
  - getNavigateHistory
```
 NavigateHistory getNavigateHistory()
```
    The navigation history.
  - getSupportJavascript
```
 Boolean getSupportJavascript()
```
    Whether the driver supports javascript. Web drivers such as MockDriver do not support javascript.
  - isMockedPageSource
```
 Boolean isMockedPageSource()
```
    Whether the page source is mocked.
  - isRecovered
```
 Boolean isRecovered()
```
    Whether the driver is recovered from the browser's tab list.
  - setRecovered
```
 Unit setRecovered(Boolean isRecovered)
```
    Whether the driver is recovered from the browser's tab list.
  - isReused
```
 Boolean isReused()
```
    Whether the driver is recovered and is reused to serve new tasks.
  - setReused
```
 Unit setReused(Boolean isReused)
```
    Whether the driver is recovered and is reused to serve new tasks.
  - getState
```
 AtomicReference<WebDriver.State> getState()
```
    The state of the driver.
    State.INIT: The driver is initialized.
    State.READY: The driver is ready to work.
    State.WORKING: The driver is working.
    State.RETIRED: The driver is retired and should be quit as soon as possible.
    State.QUIT: The driver is quit.
  - getData
```
 Map<String, Object> getData()
```
    The associated data.
  - getLastActiveTime
```
 Instant getLastActiveTime()
```
    The time of the last active action.
  - setLastActiveTime
```
 Unit setLastActiveTime(Instant lastActiveTime)
```
    The time of the last active action.
  - isInit
```
 Boolean isInit()
```
    Whether the driver is initialized.
  - isReady
```
 Boolean isReady()
```
    Whether the driver is ready to work.
  - isWorking
```
 Boolean isWorking()
```
    Whether the driver is working.
  - isRetired
```
 Boolean isRetired()
```
    Whether the driver is retired and should be quit as soon as possible.
  - isQuit
```
 Boolean isQuit()
```
    Whether the driver is quit.
  - isCanceled
```
 Boolean isCanceled()
```
    Whether the current driver task is canceled and should return as soon as possible.
  - isCrashed
```
 Boolean isCrashed()
```
    Whether the driver is crashed.
  - getMainRequestHeaders
```
 Map<String, Object> getMainRequestHeaders()
```
    The main request's http headers.
  - getMainRequestCookies
```
 List<Map<String, String>> getMainRequestCookies()
```
    The main request's http cookies.
  - getMainResponseStatus
```
 Integer getMainResponseStatus()
```
    The main response's status.
  - getMainResponseStatusText
```
 String getMainResponseStatusText()
```
    The main response's status text.
  - getMainResponseHeaders
```
 Map<String, Object> getMainResponseHeaders()
```
    The main response's http headers.
  - getBrowser
```
 Browser getBrowser()
```
    The browser of the driver. The browser defines methods and events to manipulate a real browser.
  - getId
```
 Integer getId()
```
    The driver id.
  - free
```
 Unit free()
```
    Mark the driver as free, so it can be used to fetch a new page.
  - startWork
```
 Unit startWork()
```
    Mark the driver as working, so it can not be used to fetch another page.
  - retire
```
 Unit retire()
```
    Mark the driver as retired, so it can not be used to fetch any page, and should be quit as soon as possible.
  - cancel
```
 Unit cancel()
```
    Mark the driver as canceled, so the fetch process should return as soon as possible, and the fetch result should be dropped.
  - jvm
```
 JvmWebDriver jvm()
```
    Returns a JvmWebDriver to support other JVM languages, such as java, clojure, scala, and so on, the other JVM languages might have difficulty to handle kotlin suspend methods.
  - navigateTo
```
 Unit navigateTo(String url)
```
    Returns the main resource response. In case of multiple redirects, the navigation will resolve with the first non-redirect response.
    
    Parameters:
    
    url - URL to navigate page to.
  - location
```
 String location()
```
    Returns the document's location evaluated by javascript.
    In javascript, the window.location, or document.location, is a read-only property returns a Location object, which contains information about the URL of the document and provides methods for changing that URL and loading another URL.
    To retrieve just the URL as a string, the read-only document.URL property can also be used.
  - baseURI
```
 String baseURI()
```
    Returns the document's baseURI evaluated by javascript.
    In javascript, the baseURI is a property of Node, it's the absolute base URL of the document containing the node. A baseURI is used to resolve relative URLs.
    The base URL is determined as follows:
    By default, the base URL is the location of the document (as determined by window.location).
    If the document has an <base> element, its href attribute is used.
  - waitForSelector
```
 Long waitForSelector(String selector, Long timeoutMillis)
```
    Returns when element specified by selector satisfies {@code state} option. Returns the time remaining until timeout.
  - waitForSelector
```
 Long waitForSelector(String selector)
```
    Returns when element specified by selector satisfies {@code state} option.
  - waitForNavigation
```
 Long waitForNavigation()
```
  - waitForNavigation
```
 Long waitForNavigation(Long timeoutMillis)
```
  - evaluateSilently
```
 Object evaluateSilently(String expression)
```
    Executes JavaScript in the context of the currently selected frame or window. The script fragment provided will be executed as the body of an anonymous function.
    All possible exceptions are suppressed and do not throw.
    
    Parameters:
    
    expression - Javascript expression to evaluate
    
    Returns:
    
    Remote object value in case of primitive values or JSON values (if it was requested).
  - isVisible
```
 Boolean isVisible(String selector)
```
    Returns whether the element is visible.
  - isChecked
```
 Boolean isChecked(String selector)
```
    Returns whether the element is checked.
  - scrollDown
```
 Unit scrollDown(Integer count)
```
    The current page frame scrolls down for count times.
    
    Parameters:
    
    count - The times to scroll down.
  - scrollUp
```
 Unit scrollUp(Integer count)
```
    The current page frame scrolls up for count times.
    
    Parameters:
    
    count - The times to scroll up.
  - scrollToTop
```
 Unit scrollToTop()
```
    The current page frame scrolls to the top.
  - scrollToBottom
```
 Unit scrollToBottom()
```
    The current page frame scrolls to the bottom.
  - scrollToMiddle
```
 Unit scrollToMiddle(Float ratio)
```
    The current page frame scrolls to the middle.
    
    Parameters:
    
    ratio - The ratio of the page to scroll to, 0.0 means the top, 1.0 means the bottom.
  - clickNthAnchor
```
 String clickNthAnchor(Integer n, String rootSelector)
```
  - outerHTML
```
 String outerHTML()
```
    Returns the document's HTML markup.
    If the document does not exist, returns null.
    
    Returns:
    
    The HTML markup of the document.
  - outerHTML
```
 String outerHTML(String selector)
```
    Returns the node's HTML markup, the node is located by selector.
    If the node does not exist, returns null.
    
    Parameters:
    
    selector - The selector to locate the node.
    
    Returns:
    
    The HTML markup of the node.
  - selectFirstTextOrNull
```
 String selectFirstTextOrNull(String selector)
```
    Returns the node's text content, the node is located by selector.
    If the node does not exist, returns null.
    
    Parameters:
    
    selector - The selector to locate the node.
    
    Returns:
    
    The text content of the node.
  - selectTexts
```
 List<String> selectTexts(String selector)
```
    Returns the nodes' text contents.
    If the nodes do not exist, returns an empty list.
    
    Parameters:
    
    selector - The selector to locate the nodes.
    
    Returns:
    
    The text contents of the nodes.
  - selectFirstAttributeOrNull
```
 String selectFirstAttributeOrNull(String selector, String attrName)
```
    Returns the node's attribute value, the node is located by selector, the attribute is attrName.
    If the node does not exist, or the attribute does not exist, returns null.
    
    Parameters:
    
    selector - The selector to locate the node.
    
    attrName - The attribute name to retrieve.
    
    Returns:
    
    The attribute value of the node.
  - selectAttributes
```
 List<String> selectAttributes(String selector, String attrName)
```
    Returns the nodes' attribute values, the nodes are located by selector, the attribute is attrName.
    If the nodes do not exist, or the attribute does not exist, returns an empty list.
    
    Parameters:
    
    selector - The selector to locate the nodes.
    
    attrName - The attribute name to retrieve.
    
    Returns:
    
    The attribute values of the nodes.
  - clickTextMatches
```
 Unit clickTextMatches(String selector, String pattern, Integer count)
```
    This method clicks an element with selector whose text content matches pattern, and then focuses it. If there's no element matching selector, or the element's text content doesn't match pattern, nothing to do.
    Parameters:
    
    selector -
    A {@link https://developer.mozilla.org/en-US/docs/Web/CSS/CSS_Selectors | selector } of an element to focus. If there are multiple elements satisfying the selector, the first will be focused.
  - clickMatches
```
 Unit clickMatches(String selector, String attrName, String pattern, Integer count)
```
  - check
```
 Unit check(String selector)
```
    This method check an element with selector. If there's no element matching selector, nothing to do.
    Parameters:
    
    selector -
    A {@link https://developer.mozilla.org/en-US/docs/Web/CSS/CSS_Selectors | selector } of an element to check. If there are multiple elements satisfying the selector, the first will be focused.
  - uncheck
```
 Unit uncheck(String selector)
```
    This method uncheck an element with selector. If there's no element matching selector, nothing to do.
    Parameters:
    
    selector -
    A {@link https://developer.mozilla.org/en-US/docs/Web/CSS/CSS_Selectors | selector } of an element to uncheck. If there are multiple elements satisfying the selector, the first will be focused.
  - newJsoupSession
```
 Connection newJsoupSession()
```
    Create a new Jsoup session with the last page's context, which means, the same headers and cookies.
  - loadJsoupResource
```
 Connection.Response loadJsoupResource(String url)
```
    Load the url as a resource with Jsoup rather than browser rendering, with the last page's context, which means, the same headers and cookies.
  - loadResource
```
 NetworkResourceResponse loadResource(String url)
```
    Load url as a resource without browser rendering, with the last page's context, which means, the same headers and cookies.
  - equals
```
 Boolean equals(Object other)
```
  - hashCode
```
 Integer hashCode()
```
  - compareTo
```
 Integer compareTo(AbstractWebDriver other)
```
  - toString
```
 String toString()
```

Class AbstractWebDriver

Nested Class Summary

Field Summary

Constructor Summary

Enum Constant Summary

Method Summary

Methods inherited from class ai.platon.pulsar.crawl.fetch.driver.WebDriver

Methods inherited from class ai.platon.pulsar.crawl.fetch.driver.AbstractJvmWebDriver

Methods inherited from class ai.platon.pulsar.crawl.fetch.driver.JvmWebDriver

Methods inherited from class java.io.Closeable

Methods inherited from class java.lang.Object

Constructor Detail

AbstractWebDriver

Method Detail

getIdleTimeout

setIdleTimeout

getWaitForElementTimeout

setWaitForElementTimeout

getName

getDelayPolicy

getNavigateEntry

setNavigateEntry

getNavigateHistory

getSupportJavascript

isMockedPageSource

isRecovered

setRecovered

isReused

setReused

getState

getData

getLastActiveTime

setLastActiveTime

isInit

isReady

isWorking

isRetired

isQuit

isCanceled

isCrashed

getMainRequestHeaders

getMainRequestCookies

getMainResponseStatus

getMainResponseStatusText

getMainResponseHeaders

getBrowser

getId

free

startWork

retire

cancel

jvm

navigateTo

location

baseURI

waitForSelector

waitForSelector

waitForNavigation

waitForNavigation

evaluateSilently

isVisible

isChecked

scrollDown

scrollUp

scrollToTop

scrollToBottom

scrollToMiddle

clickNthAnchor

outerHTML

outerHTML

selectFirstTextOrNull

selectTexts

selectFirstAttributeOrNull

selectAttributes

clickTextMatches

clickMatches

check

uncheck

newJsoupSession

loadJsoupResource