All Implemented Interfaces:: ai.platon.pulsar.common.config.Parameterized


public class LoadOptions
extends CommonOptions

A LoadOptions object contains a set of control parameters that affect how we load a webpage.

The load options, or load arguments, can be a plain string in the form of command line parameters, and can be parsed into a LoadOptions object.

// parse a string into a LoadOptions object
val options = session.options('-expires 1d -itemExpires 1d -ignoreFailure -parse -storeContent')
// fetch after 1d since last fetch
session.load('https://www.jd.com', '-expires 1d')
// fetch immediately ignoring errors
session.load('https://www.jd.com', '-refresh')
// do not fetch after dead time
session.load('https://www.jd.com', '-deadline 2022-04-15T18:36:54.941Z')
// activate the parse phase
session.load('https://www.jd.com', '-parse')
// write the page content into storage
session.load('https://www.jd.com', '-storeContent')

TODO: support new options: -priority TODO: keep unsupported options

Nested Class Summary

Nested Classes
Modifier and Type	Class	Description
`public class`	`LoadOptions.Companion`

Field Summary

Fields
Modifier and Type	Field	Description
`private String`	`entity`
`private String`	`label`
`private String`	`taskId`
`private Instant`	`taskTime`
`private Instant`	`deadline`
`private final Instant`	`deadTime`
`private String`	`authToken`
`private Boolean`	`readonly`
`private Boolean`	`isResource`
`private Duration`	`expires`
`private Instant`	`expireAt`
`private Duration`	`fetchInterval`
`private String`	`outLinkSelector`
`private String`	`outLinkPattern`
`private String`	`clickTarget`
`private String`	`nextPageSelector`
`private Integer`	`iframe`
`private Integer`	`topLinks`
`private Integer`	`topNAnchorGroups`
`private String`	`waitNonBlank`
`private String`	`requireNotBlank`
`private Integer`	`requireSize`
`private Integer`	`requireImages`
`private Integer`	`requireAnchors`
`private FetchMode`	`fetchMode`
`private BrowserType`	`browser`
`private Integer`	`scrollCount`
`private Duration`	`scrollInterval`
`private Duration`	`scriptTimeout`
`private Duration`	`pageLoadTimeout`
`private BrowserType`	`itemBrowser`
`private Duration`	`itemExpires`
`private Instant`	`itemExpireAt`
`private Integer`	`itemScrollCount`
`private Duration`	`itemScrollInterval`
`private Duration`	`itemScriptTimeout`
`private Duration`	`itemPageLoadTimeout`
`private String`	`itemRequireNotBlank`
`private Integer`	`itemRequireSize`
`private Integer`	`itemRequireImages`
`private Integer`	`itemRequireAnchors`
`private Boolean`	`persist`
`private Boolean`	`storeContent`
`private Boolean`	`dropContent`
`private Boolean`	`refresh`
`private Boolean`	`ignoreFailure`
`private Integer`	`nMaxRetry`
`private Integer`	`nJitRetry`
`private Boolean`	`lazyFlush`
`private Boolean`	`incognito`
`private Boolean`	`noRedirect`
`private Boolean`	`hardRedirect`
`private Boolean`	`parse`
`private Boolean`	`reparseLinks`
`private Boolean`	`ignoreUrlQuery`
`private Boolean`	`noNorm`
`private Boolean`	`noFilter`
`private Condition`	`netCondition`
`private Integer`	`test`
`private String`	`version`
`private final String`	`outLinkSelectorOrNull`
`private String`	`referrer`
`private final PageEvent`	`event`
`private final PageEvent`	`itemEvent`
`private final Params`	`modifiedParams`
`private final Map<String, Object>`	`modifiedOptions`
`private final VolatileConfig`	`conf`
`private PageEvent`	`rawEvent`
`private PageEvent`	`rawItemEvent`
`private Boolean`	`isHelp`
`private Boolean`	`expandAtSign`
`private Boolean`	`acceptUnknownOptions`
`private Boolean`	`allowParameterOverwriting`
`private final String`	`args`
`private final Array<String>`	`argv`
`public final static LoadOptions.Companion`	`Companion`

Constructor Summary

Constructors
Constructor	Description
`LoadOptions(Array<String> argv, VolatileConfig conf, PageEvent rawEvent, PageEvent rawItemEvent)`

Enum Constant Summary

Enum Constants
Enum Constant	Description

Method Summary

Modifier and Type	Method	Description
`final String`	`getEntity()`	The entity name of the page, for example, article, product, hotel, flower, etc, it's optional.
`final Unit`	`setEntity(String entity)`	The entity name of the page, for example, article, product, hotel, flower, etc, it's optional.
`final String`	`getLabel()`	The task label, it's optional and can be used to group tasks.
`final Unit`	`setLabel(String label)`	The task label, it's optional and can be used to group tasks.
`final String`	`getTaskId()`	The task id, it's optional and can be used to distinguish tasks.
`final Unit`	`setTaskId(String taskId)`	The task id, it's optional and can be used to distinguish tasks.
`final Instant`	`getTaskTime()`	The task time, we usually use task time to indicate the name of a task batch.
`final Unit`	`setTaskTime(Instant taskTime)`	The task time, we usually use task time to indicate the name of a task batch.
`final Instant`	`getDeadline()`	The deadline to finish the task, if the deadline is exceeded, the task should be discarded as soon as possible.
`final Unit`	`setDeadline(Instant deadline)`	The deadline to finish the task, if the deadline is exceeded, the task should be discarded as soon as possible.
`final Instant`	`getDeadTime()`
`final String`	`getAuthToken()`	The auth token, used for authorization purpose.
`final Unit`	`setAuthToken(String authToken)`	The auth token, used for authorization purpose.
`final Boolean`	`getReadonly()`	Specify if the load execution is read only or not.
`final Unit`	`setReadonly(Boolean readonly)`	Specify if the load execution is read only or not.
`final Boolean`	`isResource()`	If true, fetch the url as a resource without browser rendering.
`final Unit`	`setResource(Boolean isResource)`	If true, fetch the url as a resource without browser rendering.
`final Duration`	`getExpires()`	The expiry duration.
`final Unit`	`setExpires(Duration expires)`	The expiry duration.
`final Instant`	`getExpireAt()`	The expiry time point.
`final Unit`	`setExpireAt(Instant expireAt)`	The expiry time point.
`final Duration`	`getFetchInterval()`	The fetch interval, used for periodically fetch tasks.
`final Unit`	`setFetchInterval(Duration fetchInterval)`	The fetch interval, used for periodically fetch tasks.
`final String`	`getOutLinkSelector()`	The selector to extract links in portal pages.
`final Unit`	`setOutLinkSelector(String outLinkSelector)`	The selector to extract links in portal pages.
`final String`	`getOutLinkPattern()`	The pattern to select out links in the portal page.
`final Unit`	`setOutLinkPattern(String outLinkPattern)`	The pattern to select out links in the portal page.
`final String`	`getClickTarget()`	The selector for element to click.
`final Unit`	`setClickTarget(String clickTarget)`	The selector for element to click.
`final String`	`getNextPageSelector()`	The selector for next page anchor.
`final Unit`	`setNextPageSelector(String nextPageSelector)`	The selector for next page anchor.
`final Integer`	`getIframe()`	The iframe id to switch to.
`final Unit`	`setIframe(Integer iframe)`	The iframe id to switch to.
`final Integer`	`getTopLinks()`	Specify how many links to extract for out pages.
`final Unit`	`setTopLinks(Integer topLinks)`	Specify how many links to extract for out pages.
`final Integer`	`getTopNAnchorGroups()`	Choose the top N anchor groups for further process.
`final Unit`	`setTopNAnchorGroups(Integer topNAnchorGroups)`	Choose the top N anchor groups for further process.
`final String`	`getWaitNonBlank()`	The selector specified element should have a non-blank text, the system should wait until the element is filled by a non-blank text, or until it times out.
`final Unit`	`setWaitNonBlank(String waitNonBlank)`	The selector specified element should have a non-blank text, the system should wait until the element is filled by a non-blank text, or until it times out.
`final String`	`getRequireNotBlank()`	The selector specified element should have a non-blank text, the task should be retried if the element's text content is empty or blank.
`final Unit`	`setRequireNotBlank(String requireNotBlank)`	The selector specified element should have a non-blank text, the task should be retried if the element's text content is empty or blank.
`final Integer`	`getRequireSize()`	The minimum page size expected, if it is less than that, it will need to be re-fetched.
`final Unit`	`setRequireSize(Integer requireSize)`	The minimum page size expected, if it is less than that, it will need to be re-fetched.
`final Integer`	`getRequireImages()`	The minimum number of images expected in the page, if it is less than that, it will need to be re-fetched.
`final Unit`	`setRequireImages(Integer requireImages)`	The minimum number of images expected in the page, if it is less than that, it will need to be re-fetched.
`final Integer`	`getRequireAnchors()`	The minimum number of anchors expected in the page, if it is less than that, it will need to be re-fetched.
`final Unit`	`setRequireAnchors(Integer requireAnchors)`	The minimum number of anchors expected in the page, if it is less than that, it will need to be re-fetched.
`final FetchMode`	`getFetchMode()`	The fetch mode.
`final Unit`	`setFetchMode(FetchMode fetchMode)`	The fetch mode.
`final BrowserType`	`getBrowser()`	Specify which browser to use, google chrome is the default.
`final Unit`	`setBrowser(BrowserType browser)`	Specify which browser to use, google chrome is the default.
`final Integer`	`getScrollCount()`	The number of times the page should be scrolled down after it has just been opened in a browser.
`final Unit`	`setScrollCount(Integer scrollCount)`	The number of times the page should be scrolled down after it has just been opened in a browser.
`final Duration`	`getScrollInterval()`	The interval to scroll down.
`final Unit`	`setScrollInterval(Duration scrollInterval)`	The interval to scroll down.
`final Duration`	`getScriptTimeout()`	The maximum time to perform javascript injected into the browser.
`final Unit`	`setScriptTimeout(Duration scriptTimeout)`	The maximum time to perform javascript injected into the browser.
`final Duration`	`getPageLoadTimeout()`	The maximum time to wait for a page to finish.
`final Unit`	`setPageLoadTimeout(Duration pageLoadTimeout)`	The maximum time to wait for a page to finish.
`final BrowserType`	`getItemBrowser()`	The browser used to visit the item pages.
`final Unit`	`setItemBrowser(BrowserType itemBrowser)`	The browser used to visit the item pages.
`final Duration`	`getItemExpires()`	The same as expires, but only works for item pages.
`final Unit`	`setItemExpires(Duration itemExpires)`	The same as expires, but only works for item pages.
`final Instant`	`getItemExpireAt()`	If an item page is expired, it should be fetched from the web again.
`final Unit`	`setItemExpireAt(Instant itemExpireAt)`	If an item page is expired, it should be fetched from the web again.
`final Integer`	`getItemScrollCount()`	The same as scrollCount, but only works for item pages.
`final Unit`	`setItemScrollCount(Integer itemScrollCount)`	The same as scrollCount, but only works for item pages.
`final Duration`	`getItemScrollInterval()`	The same as scrollInterval, but only works for item pages.
`final Unit`	`setItemScrollInterval(Duration itemScrollInterval)`	The same as scrollInterval, but only works for item pages.
`final Duration`	`getItemScriptTimeout()`	The same as scriptTimeout, but only works for item pages.
`final Unit`	`setItemScriptTimeout(Duration itemScriptTimeout)`	The same as scriptTimeout, but only works for item pages.
`final Duration`	`getItemPageLoadTimeout()`	The same as pageLoadTimeout, but only works for item pages.
`final Unit`	`setItemPageLoadTimeout(Duration itemPageLoadTimeout)`	The same as pageLoadTimeout, but only works for item pages.
`final String`	`getItemRequireNotBlank()`	Re-fetch the item pages if the required text is blank.
`final Unit`	`setItemRequireNotBlank(String itemRequireNotBlank)`	Re-fetch the item pages if the required text is blank.
`final Integer`	`getItemRequireSize()`	Re-fetch item pages smaller than requireSize.
`final Unit`	`setItemRequireSize(Integer itemRequireSize)`	Re-fetch item pages smaller than requireSize.
`final Integer`	`getItemRequireImages()`	Re-fetch item pages who's images is less than requireImages.
`final Unit`	`setItemRequireImages(Integer itemRequireImages)`	Re-fetch item pages who's images is less than requireImages.
`final Integer`	`getItemRequireAnchors()`	Re-fetch item pages whos anchors is less than requireAnchors.
`final Unit`	`setItemRequireAnchors(Integer itemRequireAnchors)`	Re-fetch item pages whos anchors is less than requireAnchors.
`final Boolean`	`getPersist()`	Persist fetched pages as soon as possible.
`final Unit`	`setPersist(Boolean persist)`	Persist fetched pages as soon as possible.
`final Boolean`	`getStoreContent()`	If false, do not persist the page content which is usually very large.
`final Unit`	`setStoreContent(Boolean storeContent)`	If false, do not persist the page content which is usually very large.
`final Boolean`	`getDropContent()`	If the option is set, do not persist the page content which is usually very large.
`final Unit`	`setDropContent(Boolean dropContent)`	If the option is set, do not persist the page content which is usually very large.
`final Boolean`	`getRefresh()`	Refresh the fetch state of a page, clear the retry counters.
`final Unit`	`setRefresh(Boolean refresh)`
`final Boolean`	`getIgnoreFailure()`	Retry fetching the page even if it's failed last time.
`final Unit`	`setIgnoreFailure(Boolean ignoreFailure)`	Retry fetching the page even if it's failed last time.
`final Integer`	`getNMaxRetry()`	Retry to fetch at most n times, if page.fetchRetries nMaxRetry, the page is marked as gone and do not fetch it again until -refresh is set to clear page.
`final Unit`	`setNMaxRetry(Integer nMaxRetry)`	Retry to fetch at most n times, if page.fetchRetries nMaxRetry, the page is marked as gone and do not fetch it again until -refresh is set to clear page.
`final Integer`	`getNJitRetry()`	Retry at most n times at fetch phase immediately if RETRY(1601) code return.
`final Unit`	`setNJitRetry(Integer nJitRetry)`	Retry at most n times at fetch phase immediately if RETRY(1601) code return.
`final Boolean`	`getLazyFlush()`	If false, pages are flushed into database as soon as possible.
`final Unit`	`setLazyFlush(Boolean lazyFlush)`	If false, pages are flushed into database as soon as possible.
`final Boolean`	`getIncognito()`	Run browser in incognito mode.
`final Unit`	`setIncognito(Boolean incognito)`	Run browser in incognito mode.
`final Boolean`	`getNoRedirect()`	Do not redirect.
`final Unit`	`setNoRedirect(Boolean noRedirect)`	Do not redirect.
`final Boolean`	`getHardRedirect()`	If false, return the original page record but the redirect target's content, otherwise, return the page record of the redirected target.
`final Unit`	`setHardRedirect(Boolean hardRedirect)`	If false, return the original page record but the redirect target's content, otherwise, return the page record of the redirected target.
`final Boolean`	`getParse()`	If true, parse the page when it's just be fetched.
`final Unit`	`setParse(Boolean parse)`	If true, parse the page when it's just be fetched.
`final Boolean`	`getReparseLinks()`	Re-parse links if the page has been parsed before.
`final Unit`	`setReparseLinks(Boolean reparseLinks)`	Re-parse links if the page has been parsed before.
`final Boolean`	`getIgnoreUrlQuery()`	If true, remove the query parameters in the url.
`final Unit`	`setIgnoreUrlQuery(Boolean ignoreUrlQuery)`	If true, remove the query parameters in the url.
`final Boolean`	`getNoNorm()`	If true, no normalizer will be applied when parse links.
`final Unit`	`setNoNorm(Boolean noNorm)`	If true, no normalizer will be applied when parse links.
`final Boolean`	`getNoFilter()`	If true, no filter will be applied when parse links.
`final Unit`	`setNoFilter(Boolean noFilter)`	If true, no filter will be applied when parse links.
`final Condition`	`getNetCondition()`	Indicates the network condition.
`final Unit`	`setNetCondition(Condition netCondition)`	Indicates the network condition.
`final Integer`	`getTest()`	The test level, 0 to disable, we will talk more in test mode.
`final Unit`	`setTest(Integer test)`	The test level, 0 to disable, we will talk more in test mode.
`final String`	`getVersion()`	The load option version.
`final Unit`	`setVersion(String version)`	The load option version.
`final String`	`getOutLinkSelectorOrNull()`
`final String`	`getReferrer()`	The page referrer.
`final Unit`	`setReferrer(String referrer)`	The page referrer.
`final PageEvent`	`getEvent()`
`final PageEvent`	`getItemEvent()`
`Params`	`getModifiedParams()`
`Map<String, Object>`	`getModifiedOptions()`
`final VolatileConfig`	`getConf()`
`final PageEvent`	`getRawEvent()`
`final Unit`	`setRawEvent(PageEvent rawEvent)`
`final PageEvent`	`getRawItemEvent()`
`final Unit`	`setRawItemEvent(PageEvent rawItemEvent)`
`Boolean`	`parse()`	Parse the arguments into LoadOptions with JCommander and with bug fixes.
`LoadOptions`	`createItemOptions()`	Create a new LoadOptions object for item pages.
`final Boolean`	`isExpired(Instant prevFetchTime)`	Check if the page expires.
`final Boolean`	`isDead()`	If the page is dead, drop the task as soon as possible.
`Unit`	`itemOptions2MajorOptions()`	Convert the item options to major options.
`final VolatileConfig`	`overrideConfiguration()`	Write option values to conf.
`final VolatileConfig`	`overrideConfiguration(VolatileConfig conf)`	Write option values to conf.
`Boolean`	`isDefault(String option)`	Check if the option value is the default.
`Params`	`getParams()`	Convert the LoadOptions to be a Params.
`String`	`toString()`	Convert the LoadOptions to a string.
`Boolean`	`equals(Object other)`	The equality check, two LoadOptions are equal only when the normalized arguments string are equal.
`Integer`	`hashCode()`
`LoadOptions`	`clone()`	Create a new LoadOptions object with the same arguments string and event handlers.

Methods inherited from class ai.platon.pulsar.common.options.CommonOptions
isHelp, setHelp
Methods inherited from class ai.platon.pulsar.common.options.PulsarOptions
addObjects, getAcceptUnknownOptions, getAllowParameterOverwriting, getArgs, getArgv, getExpandAtSign, parseOrExit, setAcceptUnknownOptions, setAllowParameterOverwriting, setExpandAtSign, setObjects, toArgsMap, toArgv, toCmdLine, toMutableArgsMap, usage
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Constructor Detail
  - LoadOptions
```
LoadOptions(Array<String> argv, VolatileConfig conf, PageEvent rawEvent, PageEvent rawItemEvent)
```
- Method Detail
  - getEntity
```
 final String getEntity()
```
    The entity name of the page, for example, article, product, hotel, flower, etc, it's optional.
  - setEntity
```
 final Unit setEntity(String entity)
```
    The entity name of the page, for example, article, product, hotel, flower, etc, it's optional.
  - getLabel
```
 final String getLabel()
```
    The task label, it's optional and can be used to group tasks.
  - setLabel
```
 final Unit setLabel(String label)
```
    The task label, it's optional and can be used to group tasks.
  - getTaskId
```
 final String getTaskId()
```
    The task id, it's optional and can be used to distinguish tasks.
  - setTaskId
```
 final Unit setTaskId(String taskId)
```
    The task id, it's optional and can be used to distinguish tasks.
  - getTaskTime
```
 final Instant getTaskTime()
```
    The task time, we usually use task time to indicate the name of a task batch.
  - setTaskTime
```
 final Unit setTaskTime(Instant taskTime)
```
    The task time, we usually use task time to indicate the name of a task batch.
  - getDeadline
```
 final Instant getDeadline()
```
    The deadline to finish the task, if the deadline is exceeded, the task should be discarded as soon as possible.
    :::
    NOTICE: The arguments "-deadTime", "--dead-time" are deprecated, will be removed in the further.
    :::
  - setDeadline
```
 final Unit setDeadline(Instant deadline)
```
    The deadline to finish the task, if the deadline is exceeded, the task should be discarded as soon as possible.
    :::
    NOTICE: The arguments "-deadTime", "--dead-time" are deprecated, will be removed in the further.
    :::
  - getDeadTime
```
 final Instant getDeadTime()
```
  - getAuthToken
```
 final String getAuthToken()
```
    The auth token, used for authorization purpose.
  - setAuthToken
```
 final Unit setAuthToken(String authToken)
```
    The auth token, used for authorization purpose.
  - getReadonly
```
 final Boolean getReadonly()
```
    Specify if the load execution is read only or not. If a load execution is read only, the webpage loaded should not be changed by the execution.
  - setReadonly
```
 final Unit setReadonly(Boolean readonly)
```
    Specify if the load execution is read only or not. If a load execution is read only, the webpage loaded should not be changed by the execution.
  - isResource
```
 final Boolean isResource()
```
    If true, fetch the url as a resource without browser rendering.
  - setResource
```
 final Unit setResource(Boolean isResource)
```
    If true, fetch the url as a resource without browser rendering.
  - getExpires
```
 final Duration getExpires()
```
    The expiry duration. If the expiry time is exceeded, the page should be fetched from the Internet.
    The term "expires" usually be used for an expiry time, for example, http-equiv, or in cookie specification, guess it means "expires at".
    The expires field supports both ISO-8601 standard and hadoop time duration format:
    ISO-8601 standard : PnDTnHnMn.nS
    Hadoop time duration format: 100s, 1m, 1h, 1d, valid units are : ns, us, ms, s, m, h, d.
  - setExpires
```
 final Unit setExpires(Duration expires)
```
    The expiry duration. If the expiry time is exceeded, the page should be fetched from the Internet.
    The term "expires" usually be used for an expiry time, for example, http-equiv, or in cookie specification, guess it means "expires at".
    The expires field supports both ISO-8601 standard and hadoop time duration format:
    ISO-8601 standard : PnDTnHnMn.nS
    Hadoop time duration format: 100s, 1m, 1h, 1d, valid units are : ns, us, ms, s, m, h, d.
  - getExpireAt
```
 final Instant getExpireAt()
```
    The expiry time point. If the expiry time is exceeded, the page should be fetched from the Internet.
    Accept the following format:
    yyyy-MM-dd[ HH[:mm:ss]]
    ISO_INSTANT, or yyyy-MM-ddTHH:mm:ssZ
  - setExpireAt
```
 final Unit setExpireAt(Instant expireAt)
```
    The expiry time point. If the expiry time is exceeded, the page should be fetched from the Internet.
    Accept the following format:
    yyyy-MM-dd[ HH[:mm:ss]]
    ISO_INSTANT, or yyyy-MM-ddTHH:mm:ssZ
  - getFetchInterval
```
 final Duration getFetchInterval()
```
    The fetch interval, used for periodically fetch tasks.
  - setFetchInterval
```
 final Unit setFetchInterval(Duration fetchInterval)
```
    The fetch interval, used for periodically fetch tasks.
  - getOutLinkSelector
```
 final String getOutLinkSelector()
```
    The selector to extract links in portal pages.
  - setOutLinkSelector
```
 final Unit setOutLinkSelector(String outLinkSelector)
```
    The selector to extract links in portal pages.
  - getOutLinkPattern
```
 final String getOutLinkPattern()
```
    The pattern to select out links in the portal page.
  - setOutLinkPattern
```
 final Unit setOutLinkPattern(String outLinkPattern)
```
    The pattern to select out links in the portal page.
  - getClickTarget
```
 final String getClickTarget()
```
    The selector for element to click. TODO: not implemented yet
  - setClickTarget
```
 final Unit setClickTarget(String clickTarget)
```
    The selector for element to click. TODO: not implemented yet
  - getNextPageSelector
```
 final String getNextPageSelector()
```
    The selector for next page anchor. TODO: not implemented yet
  - setNextPageSelector
```
 final Unit setNextPageSelector(String nextPageSelector)
```
    The selector for next page anchor. TODO: not implemented yet
  - getIframe
```
 final Integer getIframe()
```
    The iframe id to switch to. TODO: not implemented yet
  - setIframe
```
 final Unit setIframe(Integer iframe)
```
    The iframe id to switch to. TODO: not implemented yet
  - getTopLinks
```
 final Integer getTopLinks()
```
    Specify how many links to extract for out pages.
  - setTopLinks
```
 final Unit setTopLinks(Integer topLinks)
```
    Specify how many links to extract for out pages.
  - getTopNAnchorGroups
```
 final Integer getTopNAnchorGroups()
```
    Choose the top N anchor groups for further process. Used by auto web mining project.
  - setTopNAnchorGroups
```
 final Unit setTopNAnchorGroups(Integer topNAnchorGroups)
```
    Choose the top N anchor groups for further process. Used by auto web mining project.
  - getWaitNonBlank
```
 final String getWaitNonBlank()
```
    The selector specified element should have a non-blank text, the system should wait until the element is filled by a non-blank text, or until it times out. TODO: not implemented yet
  - setWaitNonBlank
```
 final Unit setWaitNonBlank(String waitNonBlank)
```
    The selector specified element should have a non-blank text, the system should wait until the element is filled by a non-blank text, or until it times out. TODO: not implemented yet
  - getRequireNotBlank
```
 final String getRequireNotBlank()
```
    The selector specified element should have a non-blank text, the task should be retried if the element's text content is empty or blank. TODO: not implemented yet
  - setRequireNotBlank
```
 final Unit setRequireNotBlank(String requireNotBlank)
```
    The selector specified element should have a non-blank text, the task should be retried if the element's text content is empty or blank. TODO: not implemented yet
  - getRequireSize
```
 final Integer getRequireSize()
```
    The minimum page size expected, if it is less than that, it will need to be re-fetched.
    The unit is byte.
  - setRequireSize
```
 final Unit setRequireSize(Integer requireSize)
```
    The minimum page size expected, if it is less than that, it will need to be re-fetched.
    The unit is byte.
  - getRequireImages
```
 final Integer getRequireImages()
```
    The minimum number of images expected in the page, if it is less than that, it will need to be re-fetched.
  - setRequireImages
```
 final Unit setRequireImages(Integer requireImages)
```
    The minimum number of images expected in the page, if it is less than that, it will need to be re-fetched.
  - getRequireAnchors
```
 final Integer getRequireAnchors()
```
    The minimum number of anchors expected in the page, if it is less than that, it will need to be re-fetched.
  - setRequireAnchors
```
 final Unit setRequireAnchors(Integer requireAnchors)
```
    The minimum number of anchors expected in the page, if it is less than that, it will need to be re-fetched.
  - getFetchMode
```
 final FetchMode getFetchMode()
```
    The fetch mode.
    Only BROWSER is supported currently.
  - setFetchMode
```
 final Unit setFetchMode(FetchMode fetchMode)
```
    The fetch mode.
    Only BROWSER is supported currently.
  - getBrowser
```
 final BrowserType getBrowser()
```
    Specify which browser to use, google chrome is the default. TODO: session scope browser choice is not support by now
  - setBrowser
```
 final Unit setBrowser(BrowserType browser)
```
    Specify which browser to use, google chrome is the default. TODO: session scope browser choice is not support by now
  - getScrollCount
```
 final Integer getScrollCount()
```
    The number of times the page should be scrolled down after it has just been opened in a browser.
  - setScrollCount
```
 final Unit setScrollCount(Integer scrollCount)
```
    The number of times the page should be scrolled down after it has just been opened in a browser.
  - getScrollInterval
```
 final Duration getScrollInterval()
```
    The interval to scroll down.
  - setScrollInterval
```
 final Unit setScrollInterval(Duration scrollInterval)
```
    The interval to scroll down.
  - getScriptTimeout
```
 final Duration getScriptTimeout()
```
    The maximum time to perform javascript injected into the browser.
  - setScriptTimeout
```
 final Unit setScriptTimeout(Duration scriptTimeout)
```
    The maximum time to perform javascript injected into the browser.
  - getPageLoadTimeout
```
 final Duration getPageLoadTimeout()
```
    The maximum time to wait for a page to finish.
  - setPageLoadTimeout
```
 final Unit setPageLoadTimeout(Duration pageLoadTimeout)
```
    The maximum time to wait for a page to finish.
  - getItemBrowser
```
 final BrowserType getItemBrowser()
```
    The browser used to visit the item pages. TODO: session scope browser choice is not support by now
  - setItemBrowser
```
 final Unit setItemBrowser(BrowserType itemBrowser)
```
    The browser used to visit the item pages. TODO: session scope browser choice is not support by now
  - getItemExpires
```
 final Duration getItemExpires()
```
    The same as expires, but only works for item pages.
  - setItemExpires
```
 final Unit setItemExpires(Duration itemExpires)
```
    The same as expires, but only works for item pages.
  - getItemExpireAt
```
 final Instant getItemExpireAt()
```
    If an item page is expired, it should be fetched from the web again.
  - setItemExpireAt
```
 final Unit setItemExpireAt(Instant itemExpireAt)
```
    If an item page is expired, it should be fetched from the web again.
  - getItemScrollCount
```
 final Integer getItemScrollCount()
```
    The same as scrollCount, but only works for item pages.
  - setItemScrollCount
```
 final Unit setItemScrollCount(Integer itemScrollCount)
```
    The same as scrollCount, but only works for item pages.
  - getItemScrollInterval
```
 final Duration getItemScrollInterval()
```
    The same as scrollInterval, but only works for item pages.
  - setItemScrollInterval
```
 final Unit setItemScrollInterval(Duration itemScrollInterval)
```
    The same as scrollInterval, but only works for item pages.
  - getItemScriptTimeout
```
 final Duration getItemScriptTimeout()
```
    The same as scriptTimeout, but only works for item pages.
  - setItemScriptTimeout
```
 final Unit setItemScriptTimeout(Duration itemScriptTimeout)
```
    The same as scriptTimeout, but only works for item pages.
  - getItemPageLoadTimeout
```
 final Duration getItemPageLoadTimeout()
```
    The same as pageLoadTimeout, but only works for item pages.
  - setItemPageLoadTimeout
```
 final Unit setItemPageLoadTimeout(Duration itemPageLoadTimeout)
```
    The same as pageLoadTimeout, but only works for item pages.
  - getItemRequireNotBlank
```
 final String getItemRequireNotBlank()
```
    Re-fetch the item pages if the required text is blank.
  - setItemRequireNotBlank
```
 final Unit setItemRequireNotBlank(String itemRequireNotBlank)
```
    Re-fetch the item pages if the required text is blank.
  - getItemRequireSize
```
 final Integer getItemRequireSize()
```
    Re-fetch item pages smaller than requireSize.
  - setItemRequireSize
```
 final Unit setItemRequireSize(Integer itemRequireSize)
```
    Re-fetch item pages smaller than requireSize.
  - getItemRequireImages
```
 final Integer getItemRequireImages()
```
    Re-fetch item pages who's images is less than requireImages.
  - setItemRequireImages
```
 final Unit setItemRequireImages(Integer itemRequireImages)
```
    Re-fetch item pages who's images is less than requireImages.
  - getItemRequireAnchors
```
 final Integer getItemRequireAnchors()
```
    Re-fetch item pages whos anchors is less than requireAnchors.
  - setItemRequireAnchors
```
 final Unit setItemRequireAnchors(Integer itemRequireAnchors)
```
    Re-fetch item pages whos anchors is less than requireAnchors.
  - getPersist
```
 final Boolean getPersist()
```
    Persist fetched pages as soon as possible.
  - setPersist
```
 final Unit setPersist(Boolean persist)
```
    Persist fetched pages as soon as possible.
  - getStoreContent
```
 final Boolean getStoreContent()
```
    If false, do not persist the page content which is usually very large.
  - setStoreContent
```
 final Unit setStoreContent(Boolean storeContent)
```
    If false, do not persist the page content which is usually very large.
  - getDropContent
```
 final Boolean getDropContent()
```
    If the option is set, do not persist the page content which is usually very large. If the option is set, it overrides storeContent.
  - setDropContent
```
 final Unit setDropContent(Boolean dropContent)
```
    If the option is set, do not persist the page content which is usually very large. If the option is set, it overrides storeContent.
  - getRefresh
```
 final Boolean getRefresh()
```
    Refresh the fetch state of a page, clear the retry counters. If true, the page should be fetched, just like we click the refresh button on a real browser. The option can be explained as follows: -refresh = -ignoreFailure -i 0s and set page.fetchRetries = 0.
    TODO: consider add an option itemRefresh
  - setRefresh
```
 final Unit setRefresh(Boolean refresh)
```
  - getIgnoreFailure
```
 final Boolean getIgnoreFailure()
```
    Retry fetching the page even if it's failed last time.
  - setIgnoreFailure
```
 final Unit setIgnoreFailure(Boolean ignoreFailure)
```
    Retry fetching the page even if it's failed last time.
  - getNMaxRetry
```
 final Integer getNMaxRetry()
```
    Retry to fetch at most n times, if page.fetchRetries nMaxRetry, the page is marked as gone and do not fetch it again until -refresh is set to clear page.fetchRetries
  - setNMaxRetry
```
 final Unit setNMaxRetry(Integer nMaxRetry)
```
    Retry to fetch at most n times, if page.fetchRetries nMaxRetry, the page is marked as gone and do not fetch it again until -refresh is set to clear page.fetchRetries
  - getNJitRetry
```
 final Integer getNJitRetry()
```
    Retry at most n times at fetch phase immediately if RETRY(1601) code return.
  - setNJitRetry
```
 final Unit setNJitRetry(Integer nJitRetry)
```
    Retry at most n times at fetch phase immediately if RETRY(1601) code return.
  - getLazyFlush
```
 final Boolean getLazyFlush()
```
    If false, pages are flushed into database as soon as possible.
  - setLazyFlush
```
 final Unit setLazyFlush(Boolean lazyFlush)
```
    If false, pages are flushed into database as soon as possible.
  - getIncognito
```
 final Boolean getIncognito()
```
    Run browser in incognito mode. Not used since the browser always running in temporary contexts.
  - setIncognito
```
 final Unit setIncognito(Boolean incognito)
```
    Run browser in incognito mode. Not used since the browser always running in temporary contexts.
  - getNoRedirect
```
 final Boolean getNoRedirect()
```
    Do not redirect. Ignored in browser mode since the browser handles the redirection itself.
  - setNoRedirect
```
 final Unit setNoRedirect(Boolean noRedirect)
```
    Do not redirect. Ignored in browser mode since the browser handles the redirection itself.
  - getHardRedirect
```
 final Boolean getHardRedirect()
```
    If false, return the original page record but the redirect target's content, otherwise, return the page record of the redirected target. If we use a browser, redirections are handled by the browser so the flag is ignored.
  - setHardRedirect
```
 final Unit setHardRedirect(Boolean hardRedirect)
```
    If false, return the original page record but the redirect target's content, otherwise, return the page record of the redirected target. If we use a browser, redirections are handled by the browser so the flag is ignored.
  - getParse
```
 final Boolean getParse()
```
    If true, parse the page when it's just be fetched.
  - setParse
```
 final Unit setParse(Boolean parse)
```
    If true, parse the page when it's just be fetched.
  - getReparseLinks
```
 final Boolean getReparseLinks()
```
    Re-parse links if the page has been parsed before.
  - setReparseLinks
```
 final Unit setReparseLinks(Boolean reparseLinks)
```
    Re-parse links if the page has been parsed before.
  - getIgnoreUrlQuery
```
 final Boolean getIgnoreUrlQuery()
```
    If true, remove the query parameters in the url.
  - setIgnoreUrlQuery
```
 final Unit setIgnoreUrlQuery(Boolean ignoreUrlQuery)
```
    If true, remove the query parameters in the url.
  - getNoNorm
```
 final Boolean getNoNorm()
```
    If true, no normalizer will be applied when parse links.
  - setNoNorm
```
 final Unit setNoNorm(Boolean noNorm)
```
    If true, no normalizer will be applied when parse links.
  - getNoFilter
```
 final Boolean getNoFilter()
```
    If true, no filter will be applied when parse links.
  - setNoFilter
```
 final Unit setNoFilter(Boolean noFilter)
```
    If true, no filter will be applied when parse links.
  - getNetCondition
```
 final Condition getNetCondition()
```
    Indicates the network condition.
  - setNetCondition
```
 final Unit setNetCondition(Condition netCondition)
```
    Indicates the network condition.
  - getTest
```
 final Integer getTest()
```
    The test level, 0 to disable, we will talk more in test mode.
  - setTest
```
 final Unit setTest(Integer test)
```
    The test level, 0 to disable, we will talk more in test mode.
  - getVersion
```
 final String getVersion()
```
    The load option version.
  - setVersion
```
 final Unit setVersion(String version)
```
    The load option version.
  - getOutLinkSelectorOrNull
```
 final String getOutLinkSelectorOrNull()
```
  - getReferrer
```
 final String getReferrer()
```
    The page referrer.
  - setReferrer
```
 final Unit setReferrer(String referrer)
```
    The page referrer.
  - getEvent
```
 final PageEvent getEvent()
```
  - getItemEvent
```
 final PageEvent getItemEvent()
```
  - getModifiedParams
```
 Params getModifiedParams()
```
  - getModifiedOptions
```
 Map<String, Object> getModifiedOptions()
```
  - getConf
```
 final VolatileConfig getConf()
```
  - getRawEvent
```
 final PageEvent getRawEvent()
```
  - setRawEvent
```
 final Unit setRawEvent(PageEvent rawEvent)
```
  - getRawItemEvent
```
 final PageEvent getRawItemEvent()
```
  - setRawItemEvent
```
 final Unit setRawItemEvent(PageEvent rawItemEvent)
```
  - parse
```
 Boolean parse()
```
    Parse the arguments into LoadOptions with JCommander and with bug fixes.
  - createItemOptions
```
 LoadOptions createItemOptions()
```
    Create a new LoadOptions object for item pages.
  - isExpired
```
 final Boolean isExpired(Instant prevFetchTime)
```
    Check if the page expires.
    A page is expired when:
    the last fetch time is before expireAt and now is after expireAt
    (the last fetch time + expires) is exceeded
  - isDead
```
 final Boolean isDead()
```
    If the page is dead, drop the task as soon as possible.
  - itemOptions2MajorOptions
```
 Unit itemOptions2MajorOptions()
```
    Convert the item options to major options. The system do not use item options directly, we have to do the convert before we process item pages.
  - overrideConfiguration
```
 final VolatileConfig overrideConfiguration()
```
    Write option values to conf.
    LoadOptions is not globally visible, we have to pass values to modules which can not see it through a VolatileConfig object.
  - overrideConfiguration
```
 final VolatileConfig overrideConfiguration(VolatileConfig conf)
```
    Write option values to conf.
    LoadOptions is not globally visible, we have to pass values to modules which can not see it through a VolatileConfig object.
  - isDefault
```
 Boolean isDefault(String option)
```
    Check if the option value is the default.
  - getParams
```
 Params getParams()
```
    Convert the LoadOptions to be a Params.
  - toString
```
 String toString()
```
    Convert the LoadOptions to a string. The two operations parse and toString are reversible:
    val args = "..." val options1 = LoadOptions.parse(args) val normalizedArgs = options1 val options2 = LoadOptions.parse(normalizedArgs) require(normalizedArgs == options2.toString())
  - equals
```
 Boolean equals(Object other)
```
    The equality check, two LoadOptions are equal only when the normalized arguments string are equal.
  - hashCode
```
 Integer hashCode()
```
  - clone
```
 LoadOptions clone()
```
    Create a new LoadOptions object with the same arguments string and event handlers.

Class LoadOptions

Nested Class Summary

Field Summary

Constructor Summary

Enum Constant Summary

Method Summary

Methods inherited from class ai.platon.pulsar.common.options.CommonOptions

Methods inherited from class ai.platon.pulsar.common.options.PulsarOptions

Methods inherited from class java.lang.Object

Constructor Detail

LoadOptions

Method Detail

getEntity

setEntity

getLabel

setLabel

getTaskId

setTaskId

getTaskTime

setTaskTime

getDeadline

setDeadline

getDeadTime

getAuthToken

setAuthToken

getReadonly

setReadonly

isResource

setResource

getExpires

setExpires

getExpireAt

setExpireAt

getFetchInterval

setFetchInterval

getOutLinkSelector

setOutLinkSelector

getOutLinkPattern

setOutLinkPattern

getClickTarget

setClickTarget

getNextPageSelector

setNextPageSelector

getIframe

setIframe

getTopLinks

setTopLinks

getTopNAnchorGroups

setTopNAnchorGroups

getWaitNonBlank

setWaitNonBlank

getRequireNotBlank

setRequireNotBlank

getRequireSize

setRequireSize

getRequireImages

setRequireImages

getRequireAnchors

setRequireAnchors

getFetchMode

setFetchMode

getBrowser

setBrowser

getScrollCount

setScrollCount

getScrollInterval

setScrollInterval

getScriptTimeout

setScriptTimeout

getPageLoadTimeout

setPageLoadTimeout

getItemBrowser

setItemBrowser

getItemExpires

setItemExpires

getItemExpireAt

setItemExpireAt

getItemScrollCount

setItemScrollCount

getItemScrollInterval