-
- All Implemented Interfaces:
-
ai.platon.pulsar.crawl.fetch.driver.JvmWebDriver,ai.platon.pulsar.crawl.fetch.driver.WebDriver,java.io.Closeable,java.lang.AutoCloseable
public abstract class AbstractJvmWebDriver implements JvmWebDriver, WebDriver
-
-
Field Summary
-
Constructor Summary
Constructors Constructor Description AbstractJvmWebDriver()
-
Method Summary
Modifier and Type Method Description abstract IntegergetId()The unique driver id. abstract StringgetName()The driver name. abstract BrowsergetBrowser()The browser of the driver. abstract NavigateEntrygetNavigateEntry()The current navigation entry. abstract UnitsetNavigateEntry(NavigateEntry navigateEntry)The current navigation entry. abstract NavigateHistorygetNavigateHistory()The navigation history. abstract BrowserTypegetBrowserType()The browser type. abstract BooleangetSupportJavascript()Indicates whether the driver supports javascript. abstract BooleangetIsMockedPageSource()Indicate whether the page source is mocked. abstract BooleangetIsRecovered()Whether the driver is recovered from the browser's tab list. abstract UnitsetIsRecovered(Boolean isRecovered)Whether the driver is recovered from the browser's tab list. abstract BooleangetIsReused()Whether the driver is recovered and is reused to serve new tasks. abstract UnitsetIsReused(Boolean isReused)Whether the driver is recovered and is reused to serve new tasks. abstract AtomicReference<WebDriver.State>getState()The driver status. abstract Map<String, Object>getData()The associated data. abstract InstantgetLastActiveTime()The time of the last action. abstract DurationgetIdleTimeout()The idle timeout. abstract UnitsetIdleTimeout(Duration idleTimeout)The idle timeout. BooleangetIsIdle()abstract DurationgetWaitForElementTimeout()The timeout to wait for some object. abstract UnitsetWaitForElementTimeout(Duration waitForElementTimeout)The timeout to wait for some object. abstract BooleangetIsInit()abstract BooleangetIsReady()abstract BooleangetIsFree()abstract BooleangetIsWorking()abstract BooleangetIsRetired()abstract BooleangetIsQuit()abstract BooleangetIsCanceled()abstract BooleangetIsCrashed()abstract StringgetSessionId()Function1<String, Long>getDelayPolicy()abstract Map<String, Object>getMainRequestHeaders()abstract List<Map<String, String>>getMainRequestCookies()abstract IntegergetMainResponseStatus()abstract StringgetMainResponseStatusText()abstract Map<String, Object>getMainResponseHeaders()<ERROR CLASS>addInitScriptAsync(String script)Adds a script which would be evaluated in one of the following scenarios: Whenever the page is navigated.
<ERROR CLASS>addBlockedURLsAsync(List<String> urls)Blocks URLs from loading. <ERROR CLASS>navigateToAsync(String url)Returns the main resource response. <ERROR CLASS>navigateToAsync(NavigateEntry entry)Returns the response of the main resource. <ERROR CLASS>setTimeoutsAsync(BrowserSettings browserSettings)<ERROR CLASS>currentUrlAsync()Returns a string representing the current URL that the browser is looking at. <ERROR CLASS>pageSourceAsync()Returns the source of the last loaded page. <ERROR CLASS>mainRequestHeadersAsync()<ERROR CLASS>mainRequestCookiesAsync()<ERROR CLASS>getCookiesAsync()<ERROR CLASS>bringToFrontAsync()Brings page to front (activates tab). <ERROR CLASS>waitForSelectorAsync(String selector)Returns when element specified by selector satisfies {@code state} option. <ERROR CLASS>waitForSelectorAsync(String selector, Long timeoutMillis)Returns when element specified by selector satisfies {@code state} option. <ERROR CLASS>waitForSelectorAsync(String selector, Duration timeout)<ERROR CLASS>waitForNavigationAsync()<ERROR CLASS>waitForNavigationAsync(Long timeoutMillis)<ERROR CLASS>waitForNavigationAsync(Duration timeout)<ERROR CLASS>existsAsync(String selector)<ERROR CLASS>isVisibleAsync(String selector)<ERROR CLASS>visibleAsync(String selector)<ERROR CLASS>isHiddenAsync(String selector)<ERROR CLASS>isCheckedAsync(String selector)<ERROR CLASS>typeAsync(String selector, String text)<ERROR CLASS>clickAsync(String selector, Integer count)<ERROR CLASS>clickMatchesAsync(String selector, String pattern, Integer count)<ERROR CLASS>clickMatchesAsync(String selector, String attrName, String pattern, Integer count)<ERROR CLASS>clickNthAnchorAsync(Integer n, String rootSelector)<ERROR CLASS>checkAsync(String selector)<ERROR CLASS>uncheckAsync(String selector)<ERROR CLASS>scrollToAsync(String selector)<ERROR CLASS>scrollDownAsync(Integer count)<ERROR CLASS>scrollUpAsync(Integer count)<ERROR CLASS>scrollToTopAsync()<ERROR CLASS>scrollToBottomAsync()<ERROR CLASS>scrollToMiddleAsync(Float ratio)<ERROR CLASS>mouseWheelDownAsync(Integer count, Double deltaX, Double deltaY, Long delayMillis)<ERROR CLASS>mouseWheelUpAsync(Integer count, Double deltaX, Double deltaY, Long delayMillis)<ERROR CLASS>moveMouseToAsync(Double x, Double y)<ERROR CLASS>dragAndDropAsync(String selector, Integer deltaX, Integer deltaY)<ERROR CLASS>outerHTMLAsync(String selector)<ERROR CLASS>firstTextAsync(String selector)<ERROR CLASS>allTextsAsync(String selector)<ERROR CLASS>firstAttrAsync(String selector, String attrName)<ERROR CLASS>allAttrsAsync(String selector, String attrName)<ERROR CLASS>evaluateAsync(String expression)Executes JavaScript in the context of the currently selected frame or window. <ERROR CLASS>evaluateSilentlyAsync(String expression)Executes JavaScript in the context of the currently selected frame or window. <ERROR CLASS>captureScreenshotAsync(String selector)This method scrolls element into view if needed, and then ake a screenshot of the element. <ERROR CLASS>captureScreenshotAsync(RectD rect)<ERROR CLASS>clickablePointAsync(String selector)Calculate the clickable point of an element located by selector. <ERROR CLASS>boundingBoxAsync(String selector)Return the bounding box of an element located by selector. <ERROR CLASS>newSessionAsync()Create a new Jsoup session with the last page's context, which means, the same headers and cookies. <ERROR CLASS>loadResourceAsync(String url)Load url as a resource without browser rendering, with the last page's context, which means, the same headers and cookies. <ERROR CLASS>pauseAsync()Force the page pauses all navigations and PENDING resource fetches. <ERROR CLASS>stopAsync()Force the page stop all navigations and RELEASES all resources. <ERROR CLASS>terminateAsync()Force the page stop all navigations and RELEASES all resources. -
Methods inherited from class ai.platon.pulsar.crawl.fetch.driver.AbstractJvmWebDriver
addBlockedURLs, addInitScript, addProbabilityBlockedURLs, allAttrs, allTexts, awaitTermination, baseURI, boundingBox, bringToFront, cancel, captureScreenshot, captureScreenshot, check, click, clickAsync, clickMatches, clickMatches, clickMatchesAsync, clickNthAnchor, clickTextMatches, clickablePoint, currentUrl, dragAndDrop, evaluate, evaluateDetail, evaluateSilently, exists, firstAttr, firstText, focus, free, getCookies, isChecked, isHidden, isVisible, jvm, loadResource, location, mainRequestCookies, mainRequestHeaders, mouseWheelDown, mouseWheelUp, moveMouseTo, moveMouseTo, navigateTo, navigateTo, newSession, outerHTML, pageSource, pause, quit, retire, scrollDown, scrollTo, scrollToBottom, scrollToMiddle, scrollToTop, scrollUp, setTimeouts, startWork, stop, stopLoading, terminate, type, uncheck, visible, waitForNavigation, waitForNavigation, waitForNavigation, waitForPage, waitForSelector, waitForSelector, waitForSelector -
Methods inherited from class ai.platon.pulsar.crawl.fetch.driver.WebDriver
close -
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
-
-
Method Detail
-
getBrowser
abstract Browser getBrowser()
The browser of the driver. The browser defines methods and events to manipulate a real browser.
-
getNavigateEntry
abstract NavigateEntry getNavigateEntry()
The current navigation entry.
-
setNavigateEntry
abstract Unit setNavigateEntry(NavigateEntry navigateEntry)
The current navigation entry.
-
getNavigateHistory
abstract NavigateHistory getNavigateHistory()
The navigation history.
-
getBrowserType
abstract BrowserType getBrowserType()
The browser type.
-
getSupportJavascript
abstract Boolean getSupportJavascript()
Indicates whether the driver supports javascript. Web drivers such as MockDriver do not support javascript.
-
getIsMockedPageSource
abstract Boolean getIsMockedPageSource()
Indicate whether the page source is mocked.
-
getIsRecovered
abstract Boolean getIsRecovered()
Whether the driver is recovered from the browser's tab list.
-
setIsRecovered
abstract Unit setIsRecovered(Boolean isRecovered)
Whether the driver is recovered from the browser's tab list.
-
getIsReused
abstract Boolean getIsReused()
Whether the driver is recovered and is reused to serve new tasks.
-
setIsReused
abstract Unit setIsReused(Boolean isReused)
Whether the driver is recovered and is reused to serve new tasks.
-
getState
abstract AtomicReference<WebDriver.State> getState()
The driver status.
-
getLastActiveTime
abstract Instant getLastActiveTime()
The time of the last action.
-
getIdleTimeout
abstract Duration getIdleTimeout()
The idle timeout.
-
setIdleTimeout
abstract Unit setIdleTimeout(Duration idleTimeout)
The idle timeout.
-
getWaitForElementTimeout
abstract Duration getWaitForElementTimeout()
The timeout to wait for some object.
-
setWaitForElementTimeout
abstract Unit setWaitForElementTimeout(Duration waitForElementTimeout)
The timeout to wait for some object.
-
getIsReady
abstract Boolean getIsReady()
-
getIsWorking
abstract Boolean getIsWorking()
-
getIsRetired
abstract Boolean getIsRetired()
-
getIsCanceled
abstract Boolean getIsCanceled()
-
getIsCrashed
abstract Boolean getIsCrashed()
-
getSessionId
abstract String getSessionId()
-
getDelayPolicy
Function1<String, Long> getDelayPolicy()
-
getMainRequestHeaders
abstract Map<String, Object> getMainRequestHeaders()
-
getMainRequestCookies
abstract List<Map<String, String>> getMainRequestCookies()
-
getMainResponseStatus
abstract Integer getMainResponseStatus()
-
getMainResponseStatusText
abstract String getMainResponseStatusText()
-
getMainResponseHeaders
abstract Map<String, Object> getMainResponseHeaders()
-
addInitScriptAsync
<ERROR CLASS> addInitScriptAsync(String script)
Adds a script which would be evaluated in one of the following scenarios:
Whenever the page is navigated.
The script is evaluated after the document was created but before any of its scripts were run. This is useful to amend the JavaScript environment, e.g. to seed Math.random.
- Parameters:
script- Javascript source code to add.
-
addBlockedURLsAsync
<ERROR CLASS> addBlockedURLsAsync(List<String> urls)
Blocks URLs from loading.
- Parameters:
urls- URL patterns to block.
-
navigateToAsync
<ERROR CLASS> navigateToAsync(String url)
Returns the main resource response. In case of multiple redirects, the navigation will resolve with the first non-redirect response.
- Parameters:
url- URL to navigate page to.
-
navigateToAsync
<ERROR CLASS> navigateToAsync(NavigateEntry entry)
Returns the response of the main resource. In case of multiple redirects, the navigation will resolve with the first non-redirect response.
- Parameters:
entry- NavigateEntry to navigate page to.
-
setTimeoutsAsync
<ERROR CLASS> setTimeoutsAsync(BrowserSettings browserSettings)
-
currentUrlAsync
<ERROR CLASS> currentUrlAsync()
Returns a string representing the current URL that the browser is looking at.
-
pageSourceAsync
<ERROR CLASS> pageSourceAsync()
Returns the source of the last loaded page. If the page has been modified after loading (for example, by Javascript) there is no guarantee that the returned text is that of the modified page.
-
mainRequestHeadersAsync
<ERROR CLASS> mainRequestHeadersAsync()
-
mainRequestCookiesAsync
<ERROR CLASS> mainRequestCookiesAsync()
-
getCookiesAsync
<ERROR CLASS> getCookiesAsync()
-
bringToFrontAsync
<ERROR CLASS> bringToFrontAsync()
Brings page to front (activates tab).
-
waitForSelectorAsync
<ERROR CLASS> waitForSelectorAsync(String selector)
Returns when element specified by selector satisfies {@code state} option.
-
waitForSelectorAsync
<ERROR CLASS> waitForSelectorAsync(String selector, Long timeoutMillis)
Returns when element specified by selector satisfies {@code state} option. Returns the time remaining until timeout.
-
waitForSelectorAsync
<ERROR CLASS> waitForSelectorAsync(String selector, Duration timeout)
-
waitForNavigationAsync
<ERROR CLASS> waitForNavigationAsync()
-
waitForNavigationAsync
<ERROR CLASS> waitForNavigationAsync(Long timeoutMillis)
-
waitForNavigationAsync
<ERROR CLASS> waitForNavigationAsync(Duration timeout)
-
existsAsync
<ERROR CLASS> existsAsync(String selector)
-
isVisibleAsync
<ERROR CLASS> isVisibleAsync(String selector)
-
visibleAsync
<ERROR CLASS> visibleAsync(String selector)
-
isHiddenAsync
<ERROR CLASS> isHiddenAsync(String selector)
-
isCheckedAsync
<ERROR CLASS> isCheckedAsync(String selector)
-
clickAsync
<ERROR CLASS> clickAsync(String selector, Integer count)
-
clickMatchesAsync
<ERROR CLASS> clickMatchesAsync(String selector, String pattern, Integer count)
-
clickMatchesAsync
<ERROR CLASS> clickMatchesAsync(String selector, String attrName, String pattern, Integer count)
-
clickNthAnchorAsync
<ERROR CLASS> clickNthAnchorAsync(Integer n, String rootSelector)
-
checkAsync
<ERROR CLASS> checkAsync(String selector)
-
uncheckAsync
<ERROR CLASS> uncheckAsync(String selector)
-
scrollToAsync
<ERROR CLASS> scrollToAsync(String selector)
-
scrollDownAsync
<ERROR CLASS> scrollDownAsync(Integer count)
-
scrollUpAsync
<ERROR CLASS> scrollUpAsync(Integer count)
-
scrollToTopAsync
<ERROR CLASS> scrollToTopAsync()
-
scrollToBottomAsync
<ERROR CLASS> scrollToBottomAsync()
-
scrollToMiddleAsync
<ERROR CLASS> scrollToMiddleAsync(Float ratio)
-
mouseWheelDownAsync
<ERROR CLASS> mouseWheelDownAsync(Integer count, Double deltaX, Double deltaY, Long delayMillis)
-
mouseWheelUpAsync
<ERROR CLASS> mouseWheelUpAsync(Integer count, Double deltaX, Double deltaY, Long delayMillis)
-
moveMouseToAsync
<ERROR CLASS> moveMouseToAsync(Double x, Double y)
-
dragAndDropAsync
<ERROR CLASS> dragAndDropAsync(String selector, Integer deltaX, Integer deltaY)
-
outerHTMLAsync
<ERROR CLASS> outerHTMLAsync(String selector)
-
firstTextAsync
<ERROR CLASS> firstTextAsync(String selector)
-
allTextsAsync
<ERROR CLASS> allTextsAsync(String selector)
-
firstAttrAsync
<ERROR CLASS> firstAttrAsync(String selector, String attrName)
-
allAttrsAsync
<ERROR CLASS> allAttrsAsync(String selector, String attrName)
-
evaluateAsync
<ERROR CLASS> evaluateAsync(String expression)
Executes JavaScript in the context of the currently selected frame or window. The script fragment provided will be executed as the body of an anonymous function.
- Parameters:
expression- Javascript expression to evaluate
-
evaluateSilentlyAsync
<ERROR CLASS> evaluateSilentlyAsync(String expression)
Executes JavaScript in the context of the currently selected frame or window. The script fragment provided will be executed as the body of an anonymous function.
All possible exceptions are suppressed and do not throw.
- Parameters:
expression- Javascript expression to evaluate
-
captureScreenshotAsync
<ERROR CLASS> captureScreenshotAsync(String selector)
This method scrolls element into view if needed, and then ake a screenshot of the element.
-
captureScreenshotAsync
<ERROR CLASS> captureScreenshotAsync(RectD rect)
-
clickablePointAsync
<ERROR CLASS> clickablePointAsync(String selector)
Calculate the clickable point of an element located by selector. If the element does not exist, or is not clickable, returns null.
-
boundingBoxAsync
<ERROR CLASS> boundingBoxAsync(String selector)
Return the bounding box of an element located by selector. If the element does not exist, returns null.
-
newSessionAsync
<ERROR CLASS> newSessionAsync()
Create a new Jsoup session with the last page's context, which means, the same headers and cookies.
-
loadResourceAsync
<ERROR CLASS> loadResourceAsync(String url)
Load url as a resource without browser rendering, with the last page's context, which means, the same headers and cookies.
-
pauseAsync
<ERROR CLASS> pauseAsync()
Force the page pauses all navigations and PENDING resource fetches. If the page loading stops, the user can still interact with the page, and therefore resources can continue to load.
-
stopAsync
<ERROR CLASS> stopAsync()
Force the page stop all navigations and RELEASES all resources. Interaction with the stop page results in undefined behavior and the results should not be trusted.
If a web driver stops, it can later be used to visit new pages.
-
terminateAsync
<ERROR CLASS> terminateAsync()
Force the page stop all navigations and RELEASES all resources. If a web driver is terminated, it should not be used any more and should be quit as soon as possible.
-
-
-