Package ai.platon.pulsar.skeleton.common.options

Class LoadOptions

  • All Implemented Interfaces:
    ai.platon.pulsar.common.config.Parameterized 
    
public class LoadOptions
extends CommonOptions

    Load options are a set of parameters that define how a page should be loaded, including any specific behaviors or conditions that should be applied during the loading process.

    The load options, or load arguments, can be a plain string in the form of command line parameters, and can be parsed into a LoadOptions object.

    // parse a string into a LoadOptions object
val options = session.options('-expires 1d -itemExpires 1d -ignoreFailure -parse -storeContent')
// fetch after 1d since last fetch
session.load('https://www.jd.com', '-expires 1d')
// fetch immediately ignoring errors
session.load('https://www.jd.com', '-refresh')
// do not fetch after dead time
session.load('https://www.jd.com', '-deadline 2022-04-15T18:36:54.941Z')
// activate the parse phase
session.load('https://www.jd.com', '-parse')
// write the page content into storage
session.load('https://www.jd.com', '-storeContent')

    • Method Detail

      • getEntity

         final String getEntity()

        The entity name of the page, for example, article, product, hotel, flower, etc., it's optional.

      • setEntity

         final Unit setEntity(String entity)

        The entity name of the page, for example, article, product, hotel, flower, etc., it's optional.

      • getLabel

         final String getLabel()

        The task label is optional and may be utilized to categorize tasks into groups.

      • setLabel

         final Unit setLabel(String label)

        The task label is optional and may be utilized to categorize tasks into groups.

      • getTaskId

         final String getTaskId()

        The taskId is optional and serves to differentiate tasks if needed.

      • setTaskId

         final Unit setTaskId(String taskId)

        The taskId is optional and serves to differentiate tasks if needed.

      • getTaskTime

         final Instant getTaskTime()

        Task time is usually used to denote the name of a batch of tasks.

        The task time is initialized to Instant.EPOCH, so parse() and toString() operations are symmetric.

      • setTaskTime

         final Unit setTaskTime(Instant taskTime)

        Task time is usually used to denote the name of a batch of tasks.

        The task time is initialized to Instant.EPOCH, so parse() and toString() operations are symmetric.

      • getDeadline

         final Instant getDeadline()

        The task's deadline indicates the time by which it should be completed. If this deadline is surpassed, the task must be promptly discarded.

      • setDeadline

         final Unit setDeadline(Instant deadline)

        The task's deadline indicates the time by which it should be completed. If this deadline is surpassed, the task must be promptly discarded.

      • getAuthToken

         final String getAuthToken()

        The auth token, used for authorization purpose.

      • getReadonly

         final Boolean getReadonly()

        Specify whether the load execution is read-only or not. When a load execution is read-only, it ensures that the webpage loaded remains unchanged by the execution.

      • setReadonly

         final Unit setReadonly(Boolean readonly)

        Specify whether the load execution is read-only or not. When a load execution is read-only, it ensures that the webpage loaded remains unchanged by the execution.

      • isResource

         final Boolean isResource()

        If true, fetch the url as a resource without browser rendering.

      • setResource

         final Unit setResource(Boolean isResource)

        If true, fetch the url as a resource without browser rendering.

      • getPriority

         final Integer getPriority()

        Represents the priority of a task, determining the order of execution.

        The priority value is an integer where a smaller value indicates a higher priority. This is consistent with java.util.concurrent.PriorityBlockingQueue.

        If the priority value is not within the range defined by Priority13, it will be adjusted to the nearest valid value. For example, a priority of -2001 will be adjusted to Priority13.HIGHER2.

        Note: The priority specified in args or LoadOptions takes precedence over the priority in UrlAware, Hyperlink, etc.

        Priority can be set in the following ways:

        • In the url, for example, http://example.com -priority -2000

        • In the args, for example, Hyperlink("http://example.com", "", args = "-priority -2000")

        • Int the LoadOptions object, for example, session.load("http://example.com", options.apply { priority = -2000 })

        • In the UrlAware object, for example, Hyperlink("http://example.com", "", priority = -2000)

        If a url is normalized like this:

        session.normalize(url: UrlAware, options: LoadOptions)

        The priority will be set in the following order:

        • The priority in the url

        • The priority in the args

        • The priority in the options

        Note: Consider use url args to set priority only.

        Example:

        session.load(url, "-priority 2000")

val options = session.options("-priority 2000")
session.load(url, options)

      • setPriority

         final Unit setPriority(Integer priority)

        Represents the priority of a task, determining the order of execution.

        The priority value is an integer where a smaller value indicates a higher priority. This is consistent with java.util.concurrent.PriorityBlockingQueue.

        If the priority value is not within the range defined by Priority13, it will be adjusted to the nearest valid value. For example, a priority of -2001 will be adjusted to Priority13.HIGHER2.

        Note: The priority specified in args or LoadOptions takes precedence over the priority in UrlAware, Hyperlink, etc.

        Priority can be set in the following ways:

        • In the url, for example, http://example.com -priority -2000

        • In the args, for example, Hyperlink("http://example.com", "", args = "-priority -2000")

        • Int the LoadOptions object, for example, session.load("http://example.com", options.apply { priority = -2000 })

        • In the UrlAware object, for example, Hyperlink("http://example.com", "", priority = -2000)

        If a url is normalized like this:

        session.normalize(url: UrlAware, options: LoadOptions)

        The priority will be set in the following order:

        • The priority in the url

        • The priority in the args

        • The priority in the options

        Note: Consider use url args to set priority only.

        Example:

        session.load(url, "-priority 2000")

val options = session.options("-priority 2000")
session.load(url, options)

      • getExpires

         final Duration getExpires()

        The expiry duration. If the expiry time is exceeded, the page should be fetched from the Internet.

        The term "expires" usually be used for an expiry time, for example, http-equiv, or in cookie specification, guess it means "expires at".

        The expires field supports both ISO-8601 standard and hadoop time duration format:

        • ISO-8601 standard : PnDTnHnMn.nS

        • Hadoop time duration format: 100s, 1m, 1h, 1d, valid units are : ns, us, ms, s, m, h, d.

      • setExpires

         final Unit setExpires(Duration expires)

        The expiry duration. If the expiry time is exceeded, the page should be fetched from the Internet.

        The term "expires" usually be used for an expiry time, for example, http-equiv, or in cookie specification, guess it means "expires at".

        The expires field supports both ISO-8601 standard and hadoop time duration format:

        • ISO-8601 standard : PnDTnHnMn.nS

        • Hadoop time duration format: 100s, 1m, 1h, 1d, valid units are : ns, us, ms, s, m, h, d.

      • getExpireAt

         final Instant getExpireAt()

        The expiry time point. If the expiry time is exceeded, the page should be fetched from the Internet.

        Accept the following format:

        • yyyy-MM-dd[ HH[:mm:ss]]

        • ISO_INSTANT, or yyyy-MM-ddTHH:mm:ssZ

      • setExpireAt

         final Unit setExpireAt(Instant expireAt)

        The expiry time point. If the expiry time is exceeded, the page should be fetched from the Internet.

        Accept the following format:

        • yyyy-MM-dd[ HH[:mm:ss]]

        • ISO_INSTANT, or yyyy-MM-ddTHH:mm:ssZ

      • getClickTarget

         final String getClickTarget()

        The selector for element to click. TODO: not implemented yet

      • setClickTarget

         final Unit setClickTarget(String clickTarget)

        The selector for element to click. TODO: not implemented yet

      • getIframe

         final Integer getIframe()

        The iframe id to switch to. TODO: not implemented yet

      • setIframe

         final Unit setIframe(Integer iframe)

        The iframe id to switch to. TODO: not implemented yet

      • getTopLinks

         final Integer getTopLinks()

        Specify how many links to extract for out pages.

      • getWaitNonBlank

         final String getWaitNonBlank()

        The selector specified element should have a non-blank text, the system should wait until the element is filled by a non-blank text, or until it times out.

      • setWaitNonBlank

         final Unit setWaitNonBlank(String waitNonBlank)

        The selector specified element should have a non-blank text, the system should wait until the element is filled by a non-blank text, or until it times out.

      • getRequireNotBlank

         final String getRequireNotBlank()

        The selector specified element should have a non-blank text, the task should be retried if the element's text content is empty or blank.

      • setRequireNotBlank

         final Unit setRequireNotBlank(String requireNotBlank)

        The selector specified element should have a non-blank text, the task should be retried if the element's text content is empty or blank.

      • getRequireSize

         final Integer getRequireSize()

        The minimum page size expected, if it is less than that, it will need to be re-fetched.

        The unit is byte.

      • setRequireSize

         final Unit setRequireSize(Integer requireSize)

        The minimum page size expected, if it is less than that, it will need to be re-fetched.

        The unit is byte.

      • getRequireImages

         final Integer getRequireImages()

        The minimum number of images expected in the page, if it is less than that, it will need to be re-fetched.

      • setRequireImages

         final Unit setRequireImages(Integer requireImages)

        The minimum number of images expected in the page, if it is less than that, it will need to be re-fetched.

      • getRequireAnchors

         final Integer getRequireAnchors()

        The minimum number of anchors expected in the page, if it is less than that, it will need to be re-fetched.

      • setRequireAnchors

         final Unit setRequireAnchors(Integer requireAnchors)

        The minimum number of anchors expected in the page, if it is less than that, it will need to be re-fetched.

      • getFetchMode

         final FetchMode getFetchMode()

        The fetch mode.

        Only BROWSER is supported currently.

      • setFetchMode

         final Unit setFetchMode(FetchMode fetchMode)

        The fetch mode.

        Only BROWSER is supported currently.

      • getBrowser

         final BrowserType getBrowser()

        Specify which browser to use, google chrome is the default. TODO: session scope browser choice is not support by now

      • setBrowser

         final Unit setBrowser(BrowserType browser)

        Specify which browser to use, google chrome is the default. TODO: session scope browser choice is not support by now

      • getScrollCount

         final Integer getScrollCount()

        The number of times the page should be scrolled down after it has just been opened in a browser.

      • setScrollCount

         final Unit setScrollCount(Integer scrollCount)

        The number of times the page should be scrolled down after it has just been opened in a browser.

      • getItemBrowser

         final BrowserType getItemBrowser()

        The browser used to visit the item pages. TODO: session scope browser choice is not support by now

      • setItemBrowser

         final Unit setItemBrowser(BrowserType itemBrowser)

        The browser used to visit the item pages. TODO: session scope browser choice is not support by now

      • getItemExpireAt

         final Instant getItemExpireAt()

        If an item page is expired, it should be fetched from the web again.

      • setItemExpireAt

         final Unit setItemExpireAt(Instant itemExpireAt)

        If an item page is expired, it should be fetched from the web again.

      • getItemWaitNonBlank

         final String getItemWaitNonBlank()

        The selector specified element should have a non-blank text, the system should wait until the element is filled by a non-blank text, or until it times out.

      • setItemWaitNonBlank

         final Unit setItemWaitNonBlank(String itemWaitNonBlank)

        The selector specified element should have a non-blank text, the system should wait until the element is filled by a non-blank text, or until it times out.

      • getItemRequireAnchors

         final Integer getItemRequireAnchors()

        Re-fetch item pages with fewer anchors than the required number specified by requireAnchors.

      • setItemRequireAnchors

         final Unit setItemRequireAnchors(Integer itemRequireAnchors)

        Re-fetch item pages with fewer anchors than the required number specified by requireAnchors.

      • getStoreContent

         final Boolean getStoreContent()

        If false, do not persist the page content which is usually very large.

      • setStoreContent

         final Unit setStoreContent(Boolean storeContent)

        If false, do not persist the page content which is usually very large.

      • getDropContent

         final Boolean getDropContent()

        If the option is set, do not persist the page content which is usually very large. If the option is true, it overrides storeContent.

        Example:

        session.load(url, "-dropContent")

val options = session.options("-dropContent")
session.load(url, options)

      • setDropContent

         final Unit setDropContent(Boolean dropContent)

        If the option is set, do not persist the page content which is usually very large. If the option is true, it overrides storeContent.

        Example:

        session.load(url, "-dropContent")

val options = session.options("-dropContent")
session.load(url, options)

      • getRefresh

         final Boolean getRefresh()

        Refresh the fetch state of a page, clear the retry counters. If true, the page should be fetched, just like we click the refresh button on a real browser. The option can be explained as follows: -refresh = -ignoreFailure -i 0s and set page.fetchRetries = 0.

        TODO: consider add an option itemRefresh

      • getNMaxRetry

         final Integer getNMaxRetry()

        Retry to fetch at most n times, if page.fetchRetries nMaxRetry, the page is marked as gone and do not fetch it again until -refresh is set to clear page.fetchRetries

      • setNMaxRetry

         final Unit setNMaxRetry(Integer nMaxRetry)

        Retry to fetch at most n times, if page.fetchRetries nMaxRetry, the page is marked as gone and do not fetch it again until -refresh is set to clear page.fetchRetries

      • getNJitRetry

         final Integer getNJitRetry()

        Retry at most n times at fetch phase immediately if RETRY(1601) code return.

      • setNJitRetry

         final Unit setNJitRetry(Integer nJitRetry)

        Retry at most n times at fetch phase immediately if RETRY(1601) code return.

      • getLazyFlush

         final Boolean getLazyFlush()

        If false, pages are flushed into database as soon as possible.

      • setLazyFlush

         final Unit setLazyFlush(Boolean lazyFlush)

        If false, pages are flushed into database as soon as possible.

      • getIncognito

         final Boolean getIncognito()

        Run browser in incognito mode. Not used since the browser always running in temporary contexts.

      • setIncognito

         final Unit setIncognito(Boolean incognito)

        Run browser in incognito mode. Not used since the browser always running in temporary contexts.

      • getNoRedirect

         final Boolean getNoRedirect()

        Do not redirect. Ignored in browser mode since the browser handles the redirection itself.

      • setNoRedirect

         final Unit setNoRedirect(Boolean noRedirect)

        Do not redirect. Ignored in browser mode since the browser handles the redirection itself.

      • getHardRedirect

         final Boolean getHardRedirect()

        If false, return the original page record but the redirect target's content, otherwise, return the page record of the redirected target. If we use a browser, redirections are handled by the browser so the flag is ignored.

      • setHardRedirect

         final Unit setHardRedirect(Boolean hardRedirect)

        If false, return the original page record but the redirect target's content, otherwise, return the page record of the redirected target. If we use a browser, redirections are handled by the browser so the flag is ignored.

      • getParse

         final Boolean getParse()

        If true, parse the page when it's just be fetched.

      • setParse

         final Unit setParse(Boolean parse)

        If true, parse the page when it's just be fetched.

      • getNoNorm

         final Boolean getNoNorm()

        If true, no normalizer will be applied when parse links.

      • setNoNorm

         final Unit setNoNorm(Boolean noNorm)

        If true, no normalizer will be applied when parse links.

      • getNoFilter

         final Boolean getNoFilter()

        If true, no filter will be applied when parse links.

      • getTest

         final Integer getTest()

        The test level, 0 to disable, we will talk more in test mode.

      • setTest

         final Unit setTest(Integer test)

        The test level, 0 to disable, we will talk more in test mode.

      • getConf

         final VolatileConfig getConf()

      • isDead

         final Boolean isDead()

        If the page is dead, drop the task as soon as possible.

      • itemOptions2MajorOptions

         Unit itemOptions2MajorOptions()

        Convert the item options to major options. The system do not use item options directly, we have to do the convert before we process item pages.

      • overrideConfiguration

         final VolatileConfig overrideConfiguration()

        Write option values to conf.

        LoadOptions is not globally visible, we have to pass values to modules which can not see it through a VolatileConfig object.

      • overrideConfiguration

         final VolatileConfig overrideConfiguration(VolatileConfig conf)

        Write option values to conf.

        LoadOptions is not globally visible, we have to pass values to modules which can not see it through a VolatileConfig object.

      • isDefault

         Boolean isDefault(String optionName)

        Check if the option value is the default.

        Parameters:
        optionName - the option name, without the leading -

      • toString

         String toString()

        Convert the LoadOptions to a string. The two operations parse and toString are reversible:

        val args = "..."
val options1 = LoadOptions.parse(args)
val normalizedArgs = options1
val options2 = LoadOptions.parse(normalizedArgs)
require(normalizedArgs == options2.toString())