All Implemented Interfaces:: ai.platon.pulsar.common.config.Parameterized


public class LoadOptions
extends CommonOptions

LoadOptions can represent the controling arguments which effects how we load a webpage:

// fetch only when after 1d since last fetch
session.load('https://www.jd.com', '-expires 1d')
// fetch now ignoring errors
session.load('https://www.jd.com', '-refresh')
// do not fetch after dead time
session.load('https://www.jd.com', '-deadTime 2022-04-15T18:36:54.941Z')
// activate the parse phrase
session.load('https://www.jd.com', '-parse')
// write the page content into storage
session.load('https://www.jd.com', '-storeContent')

NOTICE: every option with name optionName has to take a Parameter name -optionName.

Nested Class Summary

Nested Classes
Modifier and Type	Class	Description
`public class`	`LoadOptions.Companion`

Field Summary

Fields
Modifier and Type	Field	Description
`private String`	`label`
`private String`	`taskId`
`private Instant`	`taskTime`
`private Instant`	`deadTime`
`private String`	`authToken`
`private Boolean`	`readonly`
`private Boolean`	`isResource`
`private Duration`	`expires`
`private Instant`	`expireAt`
`private Duration`	`fetchInterval`
`private String`	`outLinkSelector`
`private String`	`outLinkPattern`
`private String`	`clickTarget`
`private String`	`nextPageSelector`
`private Integer`	`iframe`
`private Integer`	`topLinks`
`private Integer`	`topNAnchorGroups`
`private String`	`waitNonBlank`
`private String`	`requireNotBlank`
`private Integer`	`requireSize`
`private Integer`	`requireImages`
`private Integer`	`requireAnchors`
`private FetchMode`	`fetchMode`
`private BrowserType`	`browser`
`private Integer`	`scrollCount`
`private Duration`	`scrollInterval`
`private Duration`	`scriptTimeout`
`private Duration`	`pageLoadTimeout`
`private BrowserType`	`itemBrowser`
`private Duration`	`itemExpires`
`private Instant`	`itemExpireAt`
`private Integer`	`itemScrollCount`
`private Duration`	`itemScrollInterval`
`private Duration`	`itemScriptTimeout`
`private Duration`	`itemPageLoadTimeout`
`private String`	`itemRequireNotBlank`
`private Integer`	`itemRequireSize`
`private Integer`	`itemRequireImages`
`private Integer`	`itemRequireAnchors`
`private Boolean`	`persist`
`private Boolean`	`storeContent`
`private Boolean`	`refresh`
`private Boolean`	`retryFailed`
`private Boolean`	`ignoreFailure`
`private Integer`	`nMaxRetry`
`private Integer`	`nJitRetry`
`private Boolean`	`lazyFlush`
`private Boolean`	`preferParallel`
`private Boolean`	`incognito`
`private Boolean`	`noRedirect`
`private Boolean`	`hardRedirect`
`private Boolean`	`parse`
`private Boolean`	`reparseLinks`
`private Boolean`	`ignoreUrlQuery`
`private Boolean`	`noNorm`
`private Boolean`	`noFilter`
`private Condition`	`netCondition`
`private Integer`	`test`
`private String`	`version`
`private final String`	`outLinkSelectorOrNull`
`private String`	`referrer`
`private final Params`	`modifiedParams`
`private final Map<String, Object>`	`modifiedOptions`
`private final VolatileConfig`	`conf`
`private PulsarEventHandler`	`eventHandler`
`private PulsarEventHandler`	`itemEventHandler`
`private Boolean`	`isHelp`
`private Boolean`	`expandAtSign`
`private Boolean`	`acceptUnknownOptions`
`private Boolean`	`allowParameterOverwriting`
`private final String`	`args`
`private final Array<String>`	`argv`

Constructor Summary

Constructors
Constructor	Description
`LoadOptions(Array<String> argv, VolatileConfig conf, PulsarEventHandler eventHandler, PulsarEventHandler itemEventHandler)`

Method Summary

Modifier and Type	Method	Description
`final String`	`getLabel()`	The label of this load task.
`final Unit`	`setLabel(String label)`	The label of this load task.
`final String`	`getTaskId()`	The task id.
`final Unit`	`setTaskId(String taskId)`	The task id.
`final Instant`	`getTaskTime()`	The task time, we usually use a task time to indicate the name of a batch task.
`final Unit`	`setTaskTime(Instant taskTime)`	The task time, we usually use a task time to indicate the name of a batch task.
`final Instant`	`getDeadTime()`	The dead time, if now deadTime, the task should be discarded as soon as possible.
`final Unit`	`setDeadTime(Instant deadTime)`	The dead time, if now deadTime, the task should be discarded as soon as possible.
`final String`	`getAuthToken()`	The auth token for this task.
`final Unit`	`setAuthToken(String authToken)`	The auth token for this task.
`final Boolean`	`getReadonly()`	If true, get a copy of the webpage and do not modify it.
`final Unit`	`setReadonly(Boolean readonly)`	If true, get a copy of the webpage and do not modify it.
`final Boolean`	`getIsResource()`	If true, fetch the page content as a resource without browser rendering.
`final Unit`	`setIsResource(Boolean isResource)`	If true, fetch the page content as a resource without browser rendering.
`final Duration`	`getExpires()`	The expiry time.
`final Unit`	`setExpires(Duration expires)`	The expiry time.
`final Instant`	`getExpireAt()`	The time point to expire.
`final Unit`	`setExpireAt(Instant expireAt)`	The time point to expire.
`final Duration`	`getFetchInterval()`	The fetch interval, used for periodically tasks.
`final Unit`	`setFetchInterval(Duration fetchInterval)`	The fetch interval, used for periodically tasks.
`final String`	`getOutLinkSelector()`	The CSS selector to select out links in the portal page.
`final Unit`	`setOutLinkSelector(String outLinkSelector)`	The CSS selector to select out links in the portal page.
`final String`	`getOutLinkPattern()`	The pattern to select out links in the portal page.
`final Unit`	`setOutLinkPattern(String outLinkPattern)`	The pattern to select out links in the portal page.
`final String`	`getClickTarget()`	The selector of the element to click for out links, if it's blank, no element should be clicked.
`final Unit`	`setClickTarget(String clickTarget)`	The selector of the element to click for out links, if it's blank, no element should be clicked.
`final String`	`getNextPageSelector()`	The css selector of next page anchor.
`final Unit`	`setNextPageSelector(String nextPageSelector)`	The css selector of next page anchor.
`final Integer`	`getIframe()`	The iframe id to switch to.
`final Unit`	`setIframe(Integer iframe)`	The iframe id to switch to.
`final Integer`	`getTopLinks()`	Specify how many links to select if we deal with out-pages.
`final Unit`	`setTopLinks(Integer topLinks)`	Specify how many links to select if we deal with out-pages.
`final Integer`	`getTopNAnchorGroups()`	Try the top N anchor groups.
`final Unit`	`setTopNAnchorGroups(Integer topNAnchorGroups)`	Try the top N anchor groups.
`final String`	`getWaitNonBlank()`	Wait for ajax content until the element is filled by a non-blank text.
`final Unit`	`setWaitNonBlank(String waitNonBlank)`	Wait for ajax content until the element is filled by a non-blank text.
`final String`	`getRequireNotBlank()`	Keep the pages only if the required text is not blank.
`final Unit`	`setRequireNotBlank(String requireNotBlank)`	Keep the pages only if the required text is not blank.
`final Integer`	`getRequireSize()`	Fetch pages smaller than requireSize in bytes.
`final Unit`	`setRequireSize(Integer requireSize)`	Fetch pages smaller than requireSize in bytes.
`final Integer`	`getRequireImages()`	Re-fetch a page who's images less than requireImages.
`final Unit`	`setRequireImages(Integer requireImages)`	Re-fetch a page who's images less than requireImages.
`final Integer`	`getRequireAnchors()`	Re-fetch a page who's anchors less than requireAnchors.
`final Unit`	`setRequireAnchors(Integer requireAnchors)`	Re-fetch a page who's anchors less than requireAnchors.
`final FetchMode`	`getFetchMode()`	The fetch mode.
`final Unit`	`setFetchMode(FetchMode fetchMode)`	The fetch mode.
`final BrowserType`	`getBrowser()`	Specify which browser to use, google chrome is the default.
`final Unit`	`setBrowser(BrowserType browser)`	Specify which browser to use, google chrome is the default.
`final Integer`	`getScrollCount()`	The count to scroll down after a page being opened in a browser.
`final Unit`	`setScrollCount(Integer scrollCount)`	The count to scroll down after a page being opened in a browser.
`final Duration`	`getScrollInterval()`	The interval to scroll down after a page being opened in a browser.
`final Unit`	`setScrollInterval(Duration scrollInterval)`	The interval to scroll down after a page being opened in a browser.
`final Duration`	`getScriptTimeout()`	The maximum time to perform javascript injected into the browser.
`final Unit`	`setScriptTimeout(Duration scriptTimeout)`	The maximum time to perform javascript injected into the browser.
`final Duration`	`getPageLoadTimeout()`	The maximum time to wait for a page to finish.
`final Unit`	`setPageLoadTimeout(Duration pageLoadTimeout)`	The maximum time to wait for a page to finish.
`final BrowserType`	`getItemBrowser()`	The browser used to visit the item pages.
`final Unit`	`setItemBrowser(BrowserType itemBrowser)`	The browser used to visit the item pages.
`final Duration`	`getItemExpires()`	The same as expires, but only works for item pages.
`final Unit`	`setItemExpires(Duration itemExpires)`	The same as expires, but only works for item pages.
`final Instant`	`getItemExpireAt()`	If an item page is expired, it should be fetched from the web again.
`final Unit`	`setItemExpireAt(Instant itemExpireAt)`	If an item page is expired, it should be fetched from the web again.
`final Integer`	`getItemScrollCount()`	The same as scrollCount, but only works for item pages.
`final Unit`	`setItemScrollCount(Integer itemScrollCount)`	The same as scrollCount, but only works for item pages.
`final Duration`	`getItemScrollInterval()`	The same as scrollInterval, but only works for item pages.
`final Unit`	`setItemScrollInterval(Duration itemScrollInterval)`	The same as scrollInterval, but only works for item pages.
`final Duration`	`getItemScriptTimeout()`	The same as scriptTimeout, but only works for item pages.
`final Unit`	`setItemScriptTimeout(Duration itemScriptTimeout)`	The same as scriptTimeout, but only works for item pages.
`final Duration`	`getItemPageLoadTimeout()`	The same as pageLoadTimeout, but only works for item pages.
`final Unit`	`setItemPageLoadTimeout(Duration itemPageLoadTimeout)`	The same as pageLoadTimeout, but only works for item pages.
`final String`	`getItemRequireNotBlank()`	Re-fetch the item pages if the required text is blank.
`final Unit`	`setItemRequireNotBlank(String itemRequireNotBlank)`	Re-fetch the item pages if the required text is blank.
`final Integer`	`getItemRequireSize()`	Re-fetch item pages smaller than requireSize.
`final Unit`	`setItemRequireSize(Integer itemRequireSize)`	Re-fetch item pages smaller than requireSize.
`final Integer`	`getItemRequireImages()`	Re-fetch item pages who's images is less than requireImages.
`final Unit`	`setItemRequireImages(Integer itemRequireImages)`	Re-fetch item pages who's images is less than requireImages.
`final Integer`	`getItemRequireAnchors()`	Re-fetch item pages who's anchors is less than requireAnchors.
`final Unit`	`setItemRequireAnchors(Integer itemRequireAnchors)`	Re-fetch item pages who's anchors is less than requireAnchors.
`final Boolean`	`getPersist()`	Persist fetched pages as soon as possible.
`final Unit`	`setPersist(Boolean persist)`	Persist fetched pages as soon as possible.
`final Boolean`	`getStoreContent()`	If false, persist the page without the content which is usually very large
`final Unit`	`setStoreContent(Boolean storeContent)`	If false, persist the page without the content which is usually very large
`final Boolean`	`getRefresh()`	Refresh the fetch state of a page, clear the retry counters.
`final Unit`	`setRefresh(Boolean refresh)`
`final Boolean`	`getRetryFailed()`	Force retry fetching the page if it's failed last time, or it's marked as gone.
`final Unit`	`setRetryFailed(@Deprecated(message = Replaced by ignoreFailure, will be removed in further versions) Boolean retryFailed)`	Force retry fetching the page if it's failed last time, or it's marked as gone.
`final Boolean`	`getIgnoreFailure()`	Retry fetching the page even if it's failed last time.
`final Unit`	`setIgnoreFailure(Boolean ignoreFailure)`	Retry fetching the page even if it's failed last time.
`final Integer`	`getNMaxRetry()`	Retry to fetch at most n times, if page.fetchRetries nMaxRetry, the page is marked as gone and do not fetch it again until -refresh is set to clear page.
`final Unit`	`setNMaxRetry(Integer nMaxRetry)`	Retry to fetch at most n times, if page.fetchRetries nMaxRetry, the page is marked as gone and do not fetch it again until -refresh is set to clear page.
`final Integer`	`getNJitRetry()`	Retry at most n times at fetch phrase immediately if RETRY(1601) code return.
`final Unit`	`setNJitRetry(Integer nJitRetry)`	Retry at most n times at fetch phrase immediately if RETRY(1601) code return.
`final Boolean`	`getLazyFlush()`	If false, pages are flushed into database as soon as possible.
`final Unit`	`setLazyFlush(Boolean lazyFlush)`	If false, pages are flushed into database as soon as possible.
`final Boolean`	`getPreferParallel()`	Parallel fetch pages whenever applicable.
`final Unit`	`setPreferParallel(@Deprecated(message = Not used since there is no non-parallel fetching) Boolean preferParallel)`	Parallel fetch pages whenever applicable.
`final Boolean`	`getIncognito()`	Run browser in incognito mode.
`final Unit`	`setIncognito(@Deprecated(message = Not used since the browser is always running in temporary contexts) Boolean incognito)`	Run browser in incognito mode.
`final Boolean`	`getNoRedirect()`	Do not redirect.
`final Unit`	`setNoRedirect(Boolean noRedirect)`	Do not redirect.
`final Boolean`	`getHardRedirect()`	If false, return the original page record but the redirect target's content, otherwise, return the page record of the redirected target.
`final Unit`	`setHardRedirect(Boolean hardRedirect)`	If false, return the original page record but the redirect target's content, otherwise, return the page record of the redirected target.
`final Boolean`	`getParse()`	If true, run the parse phrase.
`final Unit`	`setParse(Boolean parse)`	If true, run the parse phrase.
`final Boolean`	`getReparseLinks()`	Re-parse all links if the page is parsed.
`final Unit`	`setReparseLinks(Boolean reparseLinks)`	Re-parse all links if the page is parsed.
`final Boolean`	`getIgnoreUrlQuery()`	If true, remove the query parameters in the url.
`final Unit`	`setIgnoreUrlQuery(Boolean ignoreUrlQuery)`	If true, remove the query parameters in the url.
`final Boolean`	`getNoNorm()`	No normalizer is applied to parse links.
`final Unit`	`setNoNorm(Boolean noNorm)`	No normalizer is applied to parse links.
`final Boolean`	`getNoFilter()`	No filter is applied to parse links.
`final Unit`	`setNoFilter(Boolean noFilter)`	No filter is applied to parse links.
`final Condition`	`getNetCondition()`	Specify the network condition.
`final Unit`	`setNetCondition(Condition netCondition)`	Specify the network condition.
`final Integer`	`getTest()`	The test level, 0 to disable, we will talk more in test mode.
`final Unit`	`setTest(Integer test)`	The test level, 0 to disable, we will talk more in test mode.
`final String`	`getVersion()`	The load option version.
`final Unit`	`setVersion(String version)`	The load option version.
`final String`	`getOutLinkSelectorOrNull()`
`final String`	`getReferrer()`	The page referrer.
`final Unit`	`setReferrer(String referrer)`	The page referrer.
`Params`	`getModifiedParams()`
`Map<String, Object>`	`getModifiedOptions()`
`final VolatileConfig`	`getConf()`
`final PulsarEventHandler`	`getEventHandler()`
`final Unit`	`setEventHandler(PulsarEventHandler eventHandler)`
`final PulsarEventHandler`	`getItemEventHandler()`
`final Unit`	`setItemEventHandler(PulsarEventHandler itemEventHandler)`
`Boolean`	`getIsHelp()`
`Unit`	`setIsHelp(Boolean isHelp)`
`final Boolean`	`getExpandAtSign()`
`final Unit`	`setExpandAtSign(Boolean expandAtSign)`
`final Boolean`	`getAcceptUnknownOptions()`
`final Unit`	`setAcceptUnknownOptions(Boolean acceptUnknownOptions)`
`final Boolean`	`getAllowParameterOverwriting()`
`final Unit`	`setAllowParameterOverwriting(Boolean allowParameterOverwriting)`
`final String`	`getArgs()`
`final Array<String>`	`getArgv()`	The argument vector
`final PulsarEventHandler`	`ensureEventHandler()`	Ensure the EventHandler is created.
`final PulsarEventHandler`	`ensureItemEventHandler()`
`Boolean`	`parse()`	Parse the arguments into LoadOptions with JCommander and with bug fixes.
`LoadOptions`	`createItemOptions()`	Create options for item pages.
`final Boolean`	`isExpired(Instant prevFetchTime)`	Check if the page has been expired.
`final Boolean`	`isDead()`	If the page is dead, do not fetch it from the web.
`Unit`	`itemOptions2MajorOptions()`	Convert the item options to major options.
`final VolatileConfig`	`overrideConfiguration()`	Write some option values to conf.
`final VolatileConfig`	`overrideConfiguration(VolatileConfig conf)`	Write some option values to conf.
`Boolean`	`isDefault(String option)`	Check if the option value is the default.
`Params`	`getParams()`	Convert the LoadOptions to be a Params.
`String`	`toString()`	Convert the LoadOptions to be a string.
`Boolean`	`equals(Object other)`	The equality check, two LoadOptions are equal only when the normalized arguments string are equal.
`Integer`	`hashCode()`
`LoadOptions`	`clone()`	Create a new LoadOptions object.

Methods inherited from class ai.platon.pulsar.common.options.CommonOptions
addObjects, parseOrExit, setObjects, toArgsMap, toArgv, toCmdLine, toMutableArgsMap, usage
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Constructor Detail
  - LoadOptions
```
LoadOptions(Array<String> argv, VolatileConfig conf, PulsarEventHandler eventHandler, PulsarEventHandler itemEventHandler)
```
- Method Detail
  - getLabel
```
 final String getLabel()
```
    The label of this load task.
  - setLabel
```
 final Unit setLabel(String label)
```
    The label of this load task.
  - getTaskId
```
 final String getTaskId()
```
    The task id. A task can contain multiple loads.
  - setTaskId
```
 final Unit setTaskId(String taskId)
```
    The task id. A task can contain multiple loads.
  - getTaskTime
```
 final Instant getTaskTime()
```
    The task time, we usually use a task time to indicate the name of a batch task.
  - setTaskTime
```
 final Unit setTaskTime(Instant taskTime)
```
    The task time, we usually use a task time to indicate the name of a batch task.
  - getDeadTime
```
 final Instant getDeadTime()
```
    The dead time, if now deadTime, the task should be discarded as soon as possible.
  - setDeadTime
```
 final Unit setDeadTime(Instant deadTime)
```
    The dead time, if now deadTime, the task should be discarded as soon as possible.
  - getAuthToken
```
 final String getAuthToken()
```
    The auth token for this task.
  - setAuthToken
```
 final Unit setAuthToken(String authToken)
```
    The auth token for this task.
  - getReadonly
```
 final Boolean getReadonly()
```
    If true, get a copy of the webpage and do not modify it.
  - setReadonly
```
 final Unit setReadonly(Boolean readonly)
```
    If true, get a copy of the webpage and do not modify it.
  - getIsResource
```
 final Boolean getIsResource()
```
    If true, fetch the page content as a resource without browser rendering.
  - setIsResource
```
 final Unit setIsResource(Boolean isResource)
```
    If true, fetch the page content as a resource without browser rendering.
  - getExpires
```
 final Duration getExpires()
```
    The expiry time. If a page is expired, it should be fetched from the web.
    The term "expires" usually be used for an expiry time, for example, http-equiv, or in cookie specification, guess it means "expires at".
    The expires field supports both ISO-8601 standard and hadoop time duration format: ISO-8601 standard : PnDTnHnMn.nS Hadoop time duration format : Valid units are : ns, us, ms, s, m, h, d.
  - setExpires
```
 final Unit setExpires(Duration expires)
```
    The expiry time. If a page is expired, it should be fetched from the web.
    The term "expires" usually be used for an expiry time, for example, http-equiv, or in cookie specification, guess it means "expires at".
    The expires field supports both ISO-8601 standard and hadoop time duration format: ISO-8601 standard : PnDTnHnMn.nS Hadoop time duration format : Valid units are : ns, us, ms, s, m, h, d.
  - getExpireAt
```
 final Instant getExpireAt()
```
    The time point to expire. If a page is expired, it should be fetched from the web.
  - setExpireAt
```
 final Unit setExpireAt(Instant expireAt)
```
    The time point to expire. If a page is expired, it should be fetched from the web.
  - getFetchInterval
```
 final Duration getFetchInterval()
```
    The fetch interval, used for periodically tasks.
  - setFetchInterval
```
 final Unit setFetchInterval(Duration fetchInterval)
```
    The fetch interval, used for periodically tasks.
  - getOutLinkSelector
```
 final String getOutLinkSelector()
```
    The CSS selector to select out links in the portal page.
  - setOutLinkSelector
```
 final Unit setOutLinkSelector(String outLinkSelector)
```
    The CSS selector to select out links in the portal page.
  - getOutLinkPattern
```
 final String getOutLinkPattern()
```
    The pattern to select out links in the portal page.
  - setOutLinkPattern
```
 final Unit setOutLinkPattern(String outLinkPattern)
```
    The pattern to select out links in the portal page.
  - getClickTarget
```
 final String getClickTarget()
```
    The selector of the element to click for out links, if it's blank, no element should be clicked. TODO: not implemented yet
  - setClickTarget
```
 final Unit setClickTarget(String clickTarget)
```
    The selector of the element to click for out links, if it's blank, no element should be clicked. TODO: not implemented yet
  - getNextPageSelector
```
 final String getNextPageSelector()
```
    The css selector of next page anchor. TODO: not implemented yet
  - setNextPageSelector
```
 final Unit setNextPageSelector(String nextPageSelector)
```
    The css selector of next page anchor. TODO: not implemented yet
  - getIframe
```
 final Integer getIframe()
```
    The iframe id to switch to.
  - setIframe
```
 final Unit setIframe(Integer iframe)
```
    The iframe id to switch to.
  - getTopLinks
```
 final Integer getTopLinks()
```
    Specify how many links to select if we deal with out-pages.
  - setTopLinks
```
 final Unit setTopLinks(Integer topLinks)
```
    Specify how many links to select if we deal with out-pages.
  - getTopNAnchorGroups
```
 final Integer getTopNAnchorGroups()
```
    Try the top N anchor groups.
  - setTopNAnchorGroups
```
 final Unit setTopNAnchorGroups(Integer topNAnchorGroups)
```
    Try the top N anchor groups.
  - getWaitNonBlank
```
 final String getWaitNonBlank()
```
    Wait for ajax content until the element is filled by a non-blank text.
  - setWaitNonBlank
```
 final Unit setWaitNonBlank(String waitNonBlank)
```
    Wait for ajax content until the element is filled by a non-blank text.
  - getRequireNotBlank
```
 final String getRequireNotBlank()
```
    Keep the pages only if the required text is not blank.
  - setRequireNotBlank
```
 final Unit setRequireNotBlank(String requireNotBlank)
```
    Keep the pages only if the required text is not blank.
  - getRequireSize
```
 final Integer getRequireSize()
```
    Fetch pages smaller than requireSize in bytes.
  - setRequireSize
```
 final Unit setRequireSize(Integer requireSize)
```
    Fetch pages smaller than requireSize in bytes.
  - getRequireImages
```
 final Integer getRequireImages()
```
    Re-fetch a page who's images less than requireImages.
  - setRequireImages
```
 final Unit setRequireImages(Integer requireImages)
```
    Re-fetch a page who's images less than requireImages.
  - getRequireAnchors
```
 final Integer getRequireAnchors()
```
    Re-fetch a page who's anchors less than requireAnchors.
  - setRequireAnchors
```
 final Unit setRequireAnchors(Integer requireAnchors)
```
    Re-fetch a page who's anchors less than requireAnchors.
  - getFetchMode
```
 final FetchMode getFetchMode()
```
    The fetch mode.
  - setFetchMode
```
 final Unit setFetchMode(FetchMode fetchMode)
```
    The fetch mode.
  - getBrowser
```
 final BrowserType getBrowser()
```
    Specify which browser to use, google chrome is the default. TODO: session scope browser choice is not support by now
  - setBrowser
```
 final Unit setBrowser(BrowserType browser)
```
    Specify which browser to use, google chrome is the default. TODO: session scope browser choice is not support by now
  - getScrollCount
```
 final Integer getScrollCount()
```
    The count to scroll down after a page being opened in a browser.
  - setScrollCount
```
 final Unit setScrollCount(Integer scrollCount)
```
    The count to scroll down after a page being opened in a browser.
  - getScrollInterval
```
 final Duration getScrollInterval()
```
    The interval to scroll down after a page being opened in a browser.
  - setScrollInterval
```
 final Unit setScrollInterval(Duration scrollInterval)
```
    The interval to scroll down after a page being opened in a browser.
  - getScriptTimeout
```
 final Duration getScriptTimeout()
```
    The maximum time to perform javascript injected into the browser.
  - setScriptTimeout
```
 final Unit setScriptTimeout(Duration scriptTimeout)
```
    The maximum time to perform javascript injected into the browser.
  - getPageLoadTimeout
```
 final Duration getPageLoadTimeout()
```
    The maximum time to wait for a page to finish.
  - setPageLoadTimeout
```
 final Unit setPageLoadTimeout(Duration pageLoadTimeout)
```
    The maximum time to wait for a page to finish.
  - getItemBrowser
```
 final BrowserType getItemBrowser()
```
    The browser used to visit the item pages. TODO: session scope browser choice is not support by now
  - setItemBrowser
```
 final Unit setItemBrowser(BrowserType itemBrowser)
```
    The browser used to visit the item pages. TODO: session scope browser choice is not support by now
  - getItemExpires
```
 final Duration getItemExpires()
```
    The same as expires, but only works for item pages.
  - setItemExpires
```
 final Unit setItemExpires(Duration itemExpires)
```
    The same as expires, but only works for item pages.
  - getItemExpireAt
```
 final Instant getItemExpireAt()
```
    If an item page is expired, it should be fetched from the web again.
  - setItemExpireAt
```
 final Unit setItemExpireAt(Instant itemExpireAt)
```
    If an item page is expired, it should be fetched from the web again.
  - getItemScrollCount
```
 final Integer getItemScrollCount()
```
    The same as scrollCount, but only works for item pages.
  - setItemScrollCount
```
 final Unit setItemScrollCount(Integer itemScrollCount)
```
    The same as scrollCount, but only works for item pages.
  - getItemScrollInterval
```
 final Duration getItemScrollInterval()
```
    The same as scrollInterval, but only works for item pages.
  - setItemScrollInterval
```
 final Unit setItemScrollInterval(Duration itemScrollInterval)
```
    The same as scrollInterval, but only works for item pages.
  - getItemScriptTimeout
```
 final Duration getItemScriptTimeout()
```
    The same as scriptTimeout, but only works for item pages.
  - setItemScriptTimeout
```
 final Unit setItemScriptTimeout(Duration itemScriptTimeout)
```
    The same as scriptTimeout, but only works for item pages.
  - getItemPageLoadTimeout
```
 final Duration getItemPageLoadTimeout()
```
    The same as pageLoadTimeout, but only works for item pages.
  - setItemPageLoadTimeout
```
 final Unit setItemPageLoadTimeout(Duration itemPageLoadTimeout)
```
    The same as pageLoadTimeout, but only works for item pages.
  - getItemRequireNotBlank
```
 final String getItemRequireNotBlank()
```
    Re-fetch the item pages if the required text is blank.
  - setItemRequireNotBlank
```
 final Unit setItemRequireNotBlank(String itemRequireNotBlank)
```
    Re-fetch the item pages if the required text is blank.
  - getItemRequireSize
```
 final Integer getItemRequireSize()
```
    Re-fetch item pages smaller than requireSize.
  - setItemRequireSize
```
 final Unit setItemRequireSize(Integer itemRequireSize)
```
    Re-fetch item pages smaller than requireSize.
  - getItemRequireImages
```
 final Integer getItemRequireImages()
```
    Re-fetch item pages who's images is less than requireImages.
  - setItemRequireImages
```
 final Unit setItemRequireImages(Integer itemRequireImages)
```
    Re-fetch item pages who's images is less than requireImages.
  - getItemRequireAnchors
```
 final Integer getItemRequireAnchors()
```
    Re-fetch item pages who's anchors is less than requireAnchors.
  - setItemRequireAnchors
```
 final Unit setItemRequireAnchors(Integer itemRequireAnchors)
```
    Re-fetch item pages who's anchors is less than requireAnchors.
  - getPersist
```
 final Boolean getPersist()
```
    Persist fetched pages as soon as possible.
  - setPersist
```
 final Unit setPersist(Boolean persist)
```
    Persist fetched pages as soon as possible.
  - getStoreContent
```
 final Boolean getStoreContent()
```
    If false, persist the page without the content which is usually very large
  - setStoreContent
```
 final Unit setStoreContent(Boolean storeContent)
```
    If false, persist the page without the content which is usually very large
  - getRefresh
```
 final Boolean getRefresh()
```
    Refresh the fetch state of a page, clear the retry counters. If true, the page should be fetched, just like we click the refresh button on a real browser. The option can be explained as follows: -refresh = -ignoreFailure -i 0s and set page.fetchRetries = 0.
    TODO: consider add an option itemRefresh
  - setRefresh
```
 final Unit setRefresh(Boolean refresh)
```
  - getRetryFailed
```
 final Boolean getRetryFailed()
```
    Force retry fetching the page if it's failed last time, or it's marked as gone.
  - setRetryFailed
```
 final Unit setRetryFailed(@Deprecated(message = Replaced by ignoreFailure, will be removed in further versions) Boolean retryFailed)
```
    Force retry fetching the page if it's failed last time, or it's marked as gone.
  - getIgnoreFailure
```
 final Boolean getIgnoreFailure()
```
    Retry fetching the page even if it's failed last time.
  - setIgnoreFailure
```
 final Unit setIgnoreFailure(Boolean ignoreFailure)
```
    Retry fetching the page even if it's failed last time.
  - getNMaxRetry
```
 final Integer getNMaxRetry()
```
    Retry to fetch at most n times, if page.fetchRetries nMaxRetry, the page is marked as gone and do not fetch it again until -refresh is set to clear page.fetchRetries
  - setNMaxRetry
```
 final Unit setNMaxRetry(Integer nMaxRetry)
```
    Retry to fetch at most n times, if page.fetchRetries nMaxRetry, the page is marked as gone and do not fetch it again until -refresh is set to clear page.fetchRetries
  - getNJitRetry
```
 final Integer getNJitRetry()
```
    Retry at most n times at fetch phrase immediately if RETRY(1601) code return.
  - setNJitRetry
```
 final Unit setNJitRetry(Integer nJitRetry)
```
    Retry at most n times at fetch phrase immediately if RETRY(1601) code return.
  - getLazyFlush
```
 final Boolean getLazyFlush()
```
    If false, pages are flushed into database as soon as possible.
  - setLazyFlush
```
 final Unit setLazyFlush(Boolean lazyFlush)
```
    If false, pages are flushed into database as soon as possible.
  - getPreferParallel
```
 final Boolean getPreferParallel()
```
    Parallel fetch pages whenever applicable.
  - setPreferParallel
```
 final Unit setPreferParallel(@Deprecated(message = Not used since there is no non-parallel fetching) Boolean preferParallel)
```
    Parallel fetch pages whenever applicable.
  - getIncognito
```
 final Boolean getIncognito()
```
    Run browser in incognito mode. Not used since the browser is always running in temporary contexts
  - setIncognito
```
 final Unit setIncognito(@Deprecated(message = Not used since the browser is always running in temporary contexts) Boolean incognito)
```
    Run browser in incognito mode. Not used since the browser is always running in temporary contexts
  - getNoRedirect
```
 final Boolean getNoRedirect()
```
    Do not redirect. Ignored in browser mode since the browser handles the redirection itself.
  - setNoRedirect
```
 final Unit setNoRedirect(Boolean noRedirect)
```
    Do not redirect. Ignored in browser mode since the browser handles the redirection itself.
  - getHardRedirect
```
 final Boolean getHardRedirect()
```
    If false, return the original page record but the redirect target's content, otherwise, return the page record of the redirected target. If we use a browser, redirections are handled by the browser so the flag is ignored.
  - setHardRedirect
```
 final Unit setHardRedirect(Boolean hardRedirect)
```
    If false, return the original page record but the redirect target's content, otherwise, return the page record of the redirected target. If we use a browser, redirections are handled by the browser so the flag is ignored.
  - getParse
```
 final Boolean getParse()
```
    If true, run the parse phrase.
  - setParse
```
 final Unit setParse(Boolean parse)
```
    If true, run the parse phrase.
  - getReparseLinks
```
 final Boolean getReparseLinks()
```
    Re-parse all links if the page is parsed.
  - setReparseLinks
```
 final Unit setReparseLinks(Boolean reparseLinks)
```
    Re-parse all links if the page is parsed.
  - getIgnoreUrlQuery
```
 final Boolean getIgnoreUrlQuery()
```
    If true, remove the query parameters in the url.
  - setIgnoreUrlQuery
```
 final Unit setIgnoreUrlQuery(Boolean ignoreUrlQuery)
```
    If true, remove the query parameters in the url.
  - getNoNorm
```
 final Boolean getNoNorm()
```
    No normalizer is applied to parse links.
  - setNoNorm
```
 final Unit setNoNorm(Boolean noNorm)
```
    No normalizer is applied to parse links.
  - getNoFilter
```
 final Boolean getNoFilter()
```
    No filter is applied to parse links.
  - setNoFilter
```
 final Unit setNoFilter(Boolean noFilter)
```
    No filter is applied to parse links.
  - getNetCondition
```
 final Condition getNetCondition()
```
    Specify the network condition.
  - setNetCondition
```
 final Unit setNetCondition(Condition netCondition)
```
    Specify the network condition.
  - getTest
```
 final Integer getTest()
```
    The test level, 0 to disable, we will talk more in test mode.
  - setTest
```
 final Unit setTest(Integer test)
```
    The test level, 0 to disable, we will talk more in test mode.
  - getVersion
```
 final String getVersion()
```
    The load option version.
  - setVersion
```
 final Unit setVersion(String version)
```
    The load option version.
  - getOutLinkSelectorOrNull
```
 final String getOutLinkSelectorOrNull()
```
  - getReferrer
```
 final String getReferrer()
```
    The page referrer.
  - setReferrer
```
 final Unit setReferrer(String referrer)
```
    The page referrer.
  - getModifiedParams
```
 Params getModifiedParams()
```
  - getModifiedOptions
```
 Map<String, Object> getModifiedOptions()
```
  - getConf
```
 final VolatileConfig getConf()
```
  - getEventHandler
```
 final PulsarEventHandler getEventHandler()
```
  - setEventHandler
```
 final Unit setEventHandler(PulsarEventHandler eventHandler)
```
  - getItemEventHandler
```
 final PulsarEventHandler getItemEventHandler()
```
  - setItemEventHandler
```
 final Unit setItemEventHandler(PulsarEventHandler itemEventHandler)
```
  - getIsHelp
```
 Boolean getIsHelp()
```
  - setIsHelp
```
 Unit setIsHelp(Boolean isHelp)
```
  - getExpandAtSign
```
 final Boolean getExpandAtSign()
```
  - setExpandAtSign
```
 final Unit setExpandAtSign(Boolean expandAtSign)
```
  - getAcceptUnknownOptions
```
 final Boolean getAcceptUnknownOptions()
```
  - setAcceptUnknownOptions
```
 final Unit setAcceptUnknownOptions(Boolean acceptUnknownOptions)
```
  - getAllowParameterOverwriting
```
 final Boolean getAllowParameterOverwriting()
```
  - setAllowParameterOverwriting
```
 final Unit setAllowParameterOverwriting(Boolean allowParameterOverwriting)
```
  - getArgs
```
 final String getArgs()
```
  - getArgv
```
 final Array<String> getArgv()
```
    The argument vector
  - ensureEventHandler
```
 final PulsarEventHandler ensureEventHandler()
```
    Ensure the EventHandler is created.
  - ensureItemEventHandler
```
 final PulsarEventHandler ensureItemEventHandler()
```
  - parse
```
 Boolean parse()
```
    Parse the arguments into LoadOptions with JCommander and with bug fixes.
  - createItemOptions
```
 LoadOptions createItemOptions()
```
    Create options for item pages.
  - isExpired
```
 final Boolean isExpired(Instant prevFetchTime)
```
    Check if the page has been expired. A page is expired if
    the last fetch time is before expireAt and now is after expireAt
    (the last fetch time + expires) is passed
  - isDead
```
 final Boolean isDead()
```
    If the page is dead, do not fetch it from the web.
  - itemOptions2MajorOptions
```
 Unit itemOptions2MajorOptions()
```
    Convert the item options to major options. The system do not use item options directly, we have to do the convert before we process item pages.
  - overrideConfiguration
```
 final VolatileConfig overrideConfiguration()
```
    Write some option values to conf.
    LoadOptions is not globally visible, we have to pass some values to modules who can not see it through a VolatileConfig object.
  - overrideConfiguration
```
 final VolatileConfig overrideConfiguration(VolatileConfig conf)
```
    Write some option values to conf.
    LoadOptions is not globally visible, we have to pass some values to modules who can not see it through a VolatileConfig object.
  - isDefault
```
 Boolean isDefault(String option)
```
    Check if the option value is the default.
  - getParams
```
 Params getParams()
```
    Convert the LoadOptions to be a Params.
  - toString
```
 String toString()
```
    Convert the LoadOptions to be a string. The operation should be reversible:
    val args = "..." val options1 = LoadOptions.parse(args) val normalizedArgs = options1 val options2 = LoadOptions.parse(normalizedArgs) require(normalizedArgs == options2.toString())
  - equals
```
 Boolean equals(Object other)
```
    The equality check, two LoadOptions are equal only when the normalized arguments string are equal.
  - hashCode
```
 Integer hashCode()
```
  - clone
```
 LoadOptions clone()
```
    Create a new LoadOptions object.

Class LoadOptions

Nested Class Summary

Field Summary

Constructor Summary

Method Summary

Methods inherited from class ai.platon.pulsar.common.options.CommonOptions

Methods inherited from class java.lang.Object

Constructor Detail

LoadOptions

Method Detail

getLabel

setLabel

getTaskId

setTaskId

getTaskTime

setTaskTime

getDeadTime

setDeadTime

getAuthToken

setAuthToken

getReadonly

setReadonly

getIsResource

setIsResource

getExpires

setExpires

getExpireAt

setExpireAt

getFetchInterval

setFetchInterval

getOutLinkSelector

setOutLinkSelector

getOutLinkPattern

setOutLinkPattern

getClickTarget

setClickTarget

getNextPageSelector

setNextPageSelector

getIframe

setIframe

getTopLinks

setTopLinks

getTopNAnchorGroups

setTopNAnchorGroups

getWaitNonBlank

setWaitNonBlank

getRequireNotBlank

setRequireNotBlank

getRequireSize

setRequireSize

getRequireImages

setRequireImages

getRequireAnchors

setRequireAnchors

getFetchMode

setFetchMode

getBrowser

setBrowser

getScrollCount

setScrollCount

getScrollInterval

setScrollInterval

getScriptTimeout

setScriptTimeout

getPageLoadTimeout

setPageLoadTimeout

getItemBrowser

setItemBrowser

getItemExpires

setItemExpires

getItemExpireAt

setItemExpireAt

getItemScrollCount

setItemScrollCount

getItemScrollInterval

setItemScrollInterval

getItemScriptTimeout

setItemScriptTimeout

getItemPageLoadTimeout

setItemPageLoadTimeout