Package ai.platon.pulsar.skeleton.common.options

Class LoadOptions

  • All Implemented Interfaces:
    ai.platon.pulsar.common.config.Parameterized 
    
public class LoadOptions
extends CommonOptions

    Load options define the parameters controlling how web pages are fetched, processed, and stored.

    These options can be specified as command-line style parameters and parsed into a LoadOptions object. They control aspects such as caching behavior, page interaction, content requirements, and more.

    Examples:

    // Parse a string into a LoadOptions object
val options = session.options('-expires 1d -itemExpires 1d -ignoreFailure -parse -storeContent')

// Fetch after 1 day since last fetch
session.load('https://www.jd.com', '-expires 1d')

// Force immediate fetch regardless of cache
session.load('https://www.jd.com', '-refresh')

// Don't fetch after specified deadline
session.load('https://www.jd.com', '-deadline 2022-04-15T18:36:54.941Z')

// Enable parsing phase
session.load('https://www.jd.com', '-parse')

// Store page content in storage
session.load('https://www.jd.com', '-storeContent')

    • Method Detail

      • getEntity

         final String getEntity()

        The entity name representing the type of content being crawled (e.g., article, product, hotel). Used for classification and specialized processing of different content types.

      • setEntity

         final Unit setEntity(String entity)

        The entity name representing the type of content being crawled (e.g., article, product, hotel). Used for classification and specialized processing of different content types.

      • getLabel

         final String getLabel()

        Optional label used to categorize tasks into logical groups. Useful for organizing related crawl tasks and filtering/querying results.

      • setLabel

         final Unit setLabel(String label)

        Optional label used to categorize tasks into logical groups. Useful for organizing related crawl tasks and filtering/querying results.

      • getTaskId

         final String getTaskId()

        Optional identifier to distinguish between separate tasks. Provides a unique reference for individual crawl operations within the system.

      • setTaskId

         final Unit setTaskId(String taskId)

        Optional identifier to distinguish between separate tasks. Provides a unique reference for individual crawl operations within the system.

      • getTaskTime

         final Instant getTaskTime()

        Timestamp used to identify a batch of related tasks.

        Initialized to Instant.EPOCH to maintain symmetry between parse() and toString() operations. When specified, groups tasks that belong to the same logical batch or execution window.

      • setTaskTime

         final Unit setTaskTime(Instant taskTime)

        Timestamp used to identify a batch of related tasks.

        Initialized to Instant.EPOCH to maintain symmetry between parse() and toString() operations. When specified, groups tasks that belong to the same logical batch or execution window.

      • getDeadline

         final Instant getDeadline()

        Absolute deadline after which the task should be discarded.

        If the current time exceeds this deadline, the task will be immediately abandoned. Used to prevent processing of stale or irrelevant tasks.

      • setDeadline

         final Unit setDeadline(Instant deadline)

        Absolute deadline after which the task should be discarded.

        If the current time exceeds this deadline, the task will be immediately abandoned. Used to prevent processing of stale or irrelevant tasks.

      • getAuthToken

         final String getAuthToken()

        Authentication token for authorized access to protected resources. Can be used to access restricted content or APIs requiring authentication.

      • setAuthToken

         final Unit setAuthToken(String authToken)

        Authentication token for authorized access to protected resources. Can be used to access restricted content or APIs requiring authentication.

      • getReadonly

         final Boolean getReadonly()

        When enabled, ensures the crawler operates in a non-destructive mode.

        Prevents modifications to the target page during interactions, making the crawl operation completely passive without side effects on the target site.

      • setReadonly

         final Unit setReadonly(Boolean readonly)

        When enabled, ensures the crawler operates in a non-destructive mode.

        Prevents modifications to the target page during interactions, making the crawl operation completely passive without side effects on the target site.

      • isResource

         final Boolean isResource()

        When true, fetches the URL as a basic resource without browser rendering.

        Useful for simple file downloads, APIs, or static content where browser rendering and JavaScript execution aren't necessary.

      • setResource

         final Unit setResource(Boolean isResource)

        When true, fetches the URL as a basic resource without browser rendering.

        Useful for simple file downloads, APIs, or static content where browser rendering and JavaScript execution aren't necessary.

      • getPriority

         final Integer getPriority()

        Determines task execution priority in the crawl queue.

        Lower numerical values indicate higher priority (consistent with java.util.concurrent.PriorityBlockingQueue). Values outside the valid range defined by Priority13 will be adjusted to the nearest valid value.

        Priority can be specified in multiple ways:

        • In the URL: http://example.com -priority -2000

        • In args: Hyperlink("http://example.com", "", args = "-priority -2000")

        • In LoadOptions: session.load("http://example.com", options.apply { priority = -2000 })

        • In UrlAware: Hyperlink("http://example.com", "", priority = -2000)

        When normalizing URLs, priorities are resolved in this order:

        • Priority in the URL

        • Priority in the args

        • Priority in the options

      • setPriority

         final Unit setPriority(Integer priority)

        Determines task execution priority in the crawl queue.

        Lower numerical values indicate higher priority (consistent with java.util.concurrent.PriorityBlockingQueue). Values outside the valid range defined by Priority13 will be adjusted to the nearest valid value.

        Priority can be specified in multiple ways:

        • In the URL: http://example.com -priority -2000

        • In args: Hyperlink("http://example.com", "", args = "-priority -2000")

        • In LoadOptions: session.load("http://example.com", options.apply { priority = -2000 })

        • In UrlAware: Hyperlink("http://example.com", "", priority = -2000)

        When normalizing URLs, priorities are resolved in this order:

        • Priority in the URL

        • Priority in the args

        • Priority in the options

      • getExpires

         final Duration getExpires()

        Duration after which cached content is considered stale and should be refetched.

        Controls the crawler's caching behavior by specifying how long fetched content remains valid. When the duration since last fetch exceeds this value, the page will be fetched again.

        Supports two time format standards:

        • ISO-8601 duration format: PnDTnHnMn.nS

        • Hadoop time duration format: 100s, 1m, 1h, 1d (units: ns, us, ms, s, m, h, d)

      • setExpires

         final Unit setExpires(Duration expires)

        Duration after which cached content is considered stale and should be refetched.

        Controls the crawler's caching behavior by specifying how long fetched content remains valid. When the duration since last fetch exceeds this value, the page will be fetched again.

        Supports two time format standards:

        • ISO-8601 duration format: PnDTnHnMn.nS

        • Hadoop time duration format: 100s, 1m, 1h, 1d (units: ns, us, ms, s, m, h, d)

      • getExpireAt

         final Instant getExpireAt()

        Absolute timestamp after which cached content should be refetched.

        Provides an explicit expiration point rather than a relative duration. When current time exceeds this value, the page will be fetched again regardless of when it was last fetched.

        Supports two time format standards:

        • ISO-8601 duration format: PnDTnHnMn.nS

        • Hadoop time duration format: 100s, 1m, 1h, 1d (units: ns, us, ms, s, m, h, d)

      • setExpireAt

         final Unit setExpireAt(Instant expireAt)

        Absolute timestamp after which cached content should be refetched.

        Provides an explicit expiration point rather than a relative duration. When current time exceeds this value, the page will be fetched again regardless of when it was last fetched.

        Supports two time format standards:

        • ISO-8601 duration format: PnDTnHnMn.nS

        • Hadoop time duration format: 100s, 1m, 1h, 1d (units: ns, us, ms, s, m, h, d)

      • getOutLinkSelector

         final String getOutLinkSelector()

        CSS selector used to identify and extract links from portal/index pages.

        Specifies which elements containing links should be extracted for subsequent crawling. If not specified, default link extraction behavior will be used.

      • setOutLinkSelector

         final Unit setOutLinkSelector(String outLinkSelector)

        CSS selector used to identify and extract links from portal/index pages.

        Specifies which elements containing links should be extracted for subsequent crawling. If not specified, default link extraction behavior will be used.

      • getOutLinkPattern

         final String getOutLinkPattern()

        Regular expression pattern to filter extracted outlinks.

        Only links matching this pattern will be followed after extraction. Default pattern (.+) matches all URLs.

      • setOutLinkPattern

         final Unit setOutLinkPattern(String outLinkPattern)

        Regular expression pattern to filter extracted outlinks.

        Only links matching this pattern will be followed after extraction. Default pattern (.+) matches all URLs.

      • getClickTarget

         final String getClickTarget()

        CSS selector for elements to be clicked during page interaction.

        Used to simulate user clicks on specific elements (e.g., "Load more" buttons). TODO: This feature is planned but not fully implemented yet.

      • setClickTarget

         final Unit setClickTarget(@Beta() String clickTarget)

        CSS selector for elements to be clicked during page interaction.

        Used to simulate user clicks on specific elements (e.g., "Load more" buttons). TODO: This feature is planned but not fully implemented yet.

      • getNextPageSelector

         final String getNextPageSelector()

        CSS selector to identify the "next page" link for pagination handling.

        Used to navigate through multi-page content by automatically following pagination links. TODO: This feature is planned but not fully implemented yet.

      • setNextPageSelector

         final Unit setNextPageSelector(@Beta() String nextPageSelector)

        CSS selector to identify the "next page" link for pagination handling.

        Used to navigate through multi-page content by automatically following pagination links. TODO: This feature is planned but not fully implemented yet.

      • getIframe

         final Integer getIframe()

        The index of the iframe to focus on during page interaction.

        Used to target specific iframes within complex pages for content extraction. TODO: This feature is planned but not fully implemented yet.

      • setIframe

         final Unit setIframe(@Beta() Integer iframe)

        The index of the iframe to focus on during page interaction.

        Used to target specific iframes within complex pages for content extraction. TODO: This feature is planned but not fully implemented yet.

      • getTopLinks

         final Integer getTopLinks()

        Maximum number of outlinks to extract and follow from a single page.

        Limits the number of child URLs to be processed from portal/index pages to prevent excessive crawling depth or breadth.

      • setTopLinks

         final Unit setTopLinks(Integer topLinks)

        Maximum number of outlinks to extract and follow from a single page.

        Limits the number of child URLs to be processed from portal/index pages to prevent excessive crawling depth or breadth.

      • getWaitNonBlank

         final String getWaitNonBlank()

        CSS selector for an element that must contain non-blank text before proceeding.

        The crawler will wait until the specified element contains text or until a timeout occurs. Useful for ensuring that dynamic content has properly loaded before processing the page.

      • setWaitNonBlank

         final Unit setWaitNonBlank(@Beta() String waitNonBlank)

        CSS selector for an element that must contain non-blank text before proceeding.

        The crawler will wait until the specified element contains text or until a timeout occurs. Useful for ensuring that dynamic content has properly loaded before processing the page.

      • getRequireNotBlank

         final String getRequireNotBlank()

        CSS selector for an element that must contain non-blank text for valid retrieval.

        If the specified element's text is empty or blank, the page fetch will be considered unsuccessful and retried. Used to validate that critical content is present.

      • setRequireNotBlank

         final Unit setRequireNotBlank(@Beta() String requireNotBlank)

        CSS selector for an element that must contain non-blank text for valid retrieval.

        If the specified element's text is empty or blank, the page fetch will be considered unsuccessful and retried. Used to validate that critical content is present.

      • getRequireSize

         final Integer getRequireSize()

        Minimum acceptable page size in bytes to consider the fetch successful.

        Pages smaller than this threshold will be treated as incomplete and refetched. Helps detect truncated downloads or error pages that are abnormally small.

      • setRequireSize

         final Unit setRequireSize(Integer requireSize)

        Minimum acceptable page size in bytes to consider the fetch successful.

        Pages smaller than this threshold will be treated as incomplete and refetched. Helps detect truncated downloads or error pages that are abnormally small.

      • getRequireImages

         final Integer getRequireImages()

        Minimum number of images required for a page to be considered complete.

        Pages with fewer images than this threshold will be refetched. Useful for ensuring media-rich content has properly loaded.

      • setRequireImages

         final Unit setRequireImages(Integer requireImages)

        Minimum number of images required for a page to be considered complete.

        Pages with fewer images than this threshold will be refetched. Useful for ensuring media-rich content has properly loaded.

      • getRequireAnchors

         final Integer getRequireAnchors()

        Minimum number of anchor (link) elements required for a page to be considered complete.

        Pages with fewer links than this threshold will be refetched. Useful for validating that navigational elements have properly loaded.

      • setRequireAnchors

         final Unit setRequireAnchors(Integer requireAnchors)

        Minimum number of anchor (link) elements required for a page to be considered complete.

        Pages with fewer links than this threshold will be refetched. Useful for validating that navigational elements have properly loaded.

      • getFetchMode

         final FetchMode getFetchMode()

        Specifies the mechanism used to fetch web content.

        Currently, only BROWSER mode is fully supported, which uses a headless browser for rendering and JavaScript execution during content retrieval.

      • setFetchMode

         final Unit setFetchMode(FetchMode fetchMode)

        Specifies the mechanism used to fetch web content.

        Currently, only BROWSER mode is fully supported, which uses a headless browser for rendering and JavaScript execution during content retrieval.

      • getBrowser

         final BrowserType getBrowser()

        Specifies which browser engine to use for rendering pages.

        Google Chrome is the default and recommended browser engine. NOTE: Session-scope browser selection is not yet fully supported.

      • setBrowser

         final Unit setBrowser(BrowserType browser)

        Specifies which browser engine to use for rendering pages.

        Google Chrome is the default and recommended browser engine. NOTE: Session-scope browser selection is not yet fully supported.

      • getScrollCount

         final Integer getScrollCount()

        Number of times to scroll down the page after initial load.

        Controls how aggressively the crawler attempts to load additional content by simulating scroll events. Higher values trigger more dynamic content loading.

      • setScrollCount

         final Unit setScrollCount(Integer scrollCount)

        Number of times to scroll down the page after initial load.

        Controls how aggressively the crawler attempts to load additional content by simulating scroll events. Higher values trigger more dynamic content loading.

      • getScrollInterval

         final Duration getScrollInterval()

        Time interval between successive scroll actions.

        Controls the pace of scrolling to allow time for content to load between scrolls. Shorter intervals may miss content that takes longer to load, while longer intervals increase overall crawl time.

      • setScrollInterval

         final Unit setScrollInterval(Duration scrollInterval)

        Time interval between successive scroll actions.

        Controls the pace of scrolling to allow time for content to load between scrolls. Shorter intervals may miss content that takes longer to load, while longer intervals increase overall crawl time.

      • getScriptTimeout

         final Duration getScriptTimeout()

        Maximum time allowed for injected JavaScript execution before timeout.

        Limits how long custom scripts can run during page interaction to prevent hanging on problematic scripts.

      • setScriptTimeout

         final Unit setScriptTimeout(Duration scriptTimeout)

        Maximum time allowed for injected JavaScript execution before timeout.

        Limits how long custom scripts can run during page interaction to prevent hanging on problematic scripts.

      • getPageLoadTimeout

         final Duration getPageLoadTimeout()

        Maximum time to wait for page loading to complete.

        Controls how long the crawler waits for a page to fully load before considering it ready for processing or timing out if loading takes too long.

      • setPageLoadTimeout

         final Unit setPageLoadTimeout(Duration pageLoadTimeout)

        Maximum time to wait for page loading to complete.

        Controls how long the crawler waits for a page to fully load before considering it ready for processing or timing out if loading takes too long.

      • getItemBrowser

         final BrowserType getItemBrowser()

        Browser to use specifically for item detail pages (as opposed to index pages).

        Allows using different browser configurations for detail pages vs. index pages. NOTE: Session-scope browser selection is not yet fully supported.

      • setItemBrowser

         final Unit setItemBrowser(BrowserType itemBrowser)

        Browser to use specifically for item detail pages (as opposed to index pages).

        Allows using different browser configurations for detail pages vs. index pages. NOTE: Session-scope browser selection is not yet fully supported.

      • getItemExpires

         final Duration getItemExpires()

        Cache expiration duration specifically for item detail pages.

        Controls how long item detail pages remain valid in the cache before requiring refresh. Works identically to expires but applies only to item pages, not index pages.

      • setItemExpires

         final Unit setItemExpires(Duration itemExpires)

        Cache expiration duration specifically for item detail pages.

        Controls how long item detail pages remain valid in the cache before requiring refresh. Works identically to expires but applies only to item pages, not index pages.

      • getItemExpireAt

         final Instant getItemExpireAt()

        Absolute timestamp after which item detail pages should be refetched.

        Works identically to expireAt but applies only to item pages, not index pages. Provides a fixed point in time after which cached item pages are considered invalid.

      • setItemExpireAt

         final Unit setItemExpireAt(Instant itemExpireAt)

        Absolute timestamp after which item detail pages should be refetched.

        Works identically to expireAt but applies only to item pages, not index pages. Provides a fixed point in time after which cached item pages are considered invalid.

      • getItemScrollCount

         final Integer getItemScrollCount()

        Number of scroll actions for item detail pages.

        Works identically to scrollCount but applies only to item pages. Controls how thoroughly dynamic content is loaded on item detail pages.

      • setItemScrollCount

         final Unit setItemScrollCount(Integer itemScrollCount)

        Number of scroll actions for item detail pages.

        Works identically to scrollCount but applies only to item pages. Controls how thoroughly dynamic content is loaded on item detail pages.

      • getItemScrollInterval

         final Duration getItemScrollInterval()

        Time interval between scrolls for item detail pages.

        Works identically to scrollInterval but applies only to item pages. Controls the pace of content loading on item detail pages.

      • setItemScrollInterval

         final Unit setItemScrollInterval(Duration itemScrollInterval)

        Time interval between scrolls for item detail pages.

        Works identically to scrollInterval but applies only to item pages. Controls the pace of content loading on item detail pages.

      • getItemScriptTimeout

         final Duration getItemScriptTimeout()

        JavaScript execution timeout for item detail pages.

        Works identically to scriptTimeout but applies only to item pages. Controls how long scripts can run when processing item detail pages.

      • setItemScriptTimeout

         final Unit setItemScriptTimeout(Duration itemScriptTimeout)

        JavaScript execution timeout for item detail pages.

        Works identically to scriptTimeout but applies only to item pages. Controls how long scripts can run when processing item detail pages.

      • getItemPageLoadTimeout

         final Duration getItemPageLoadTimeout()

        Page load timeout for item detail pages.

        Works identically to pageLoadTimeout but applies only to item pages. Controls maximum wait time for item detail pages to load.

      • setItemPageLoadTimeout

         final Unit setItemPageLoadTimeout(Duration itemPageLoadTimeout)

        Page load timeout for item detail pages.

        Works identically to pageLoadTimeout but applies only to item pages. Controls maximum wait time for item detail pages to load.

      • getItemWaitNonBlank

         final String getItemWaitNonBlank()

        CSS selector for non-blank text validation on item detail pages.

        Works identically to waitNonBlank but applies only to item pages. Makes the crawler wait until the specified element contains text.

      • setItemWaitNonBlank

         final Unit setItemWaitNonBlank(String itemWaitNonBlank)

        CSS selector for non-blank text validation on item detail pages.

        Works identically to waitNonBlank but applies only to item pages. Makes the crawler wait until the specified element contains text.

      • getItemRequireNotBlank

         final String getItemRequireNotBlank()

        CSS selector for content validation on item detail pages.

        Works identically to requireNotBlank but applies only to item pages. Pages without text in this selector will be considered invalid and refetched.

      • setItemRequireNotBlank

         final Unit setItemRequireNotBlank(@Beta() String itemRequireNotBlank)

        CSS selector for content validation on item detail pages.

        Works identically to requireNotBlank but applies only to item pages. Pages without text in this selector will be considered invalid and refetched.

      • getItemRequireSize

         final Integer getItemRequireSize()

        Minimum page size required for item detail pages.

        Works identically to requireSize but applies only to item pages. Item pages smaller than this threshold (in bytes) will be refetched.

      • setItemRequireSize

         final Unit setItemRequireSize(Integer itemRequireSize)

        Minimum page size required for item detail pages.

        Works identically to requireSize but applies only to item pages. Item pages smaller than this threshold (in bytes) will be refetched.

      • getItemRequireImages

         final Integer getItemRequireImages()

        Minimum number of images required for item detail pages.

        Works identically to requireImages but applies only to item pages. Item pages with fewer images than this threshold will be refetched.

      • setItemRequireImages

         final Unit setItemRequireImages(Integer itemRequireImages)

        Minimum number of images required for item detail pages.

        Works identically to requireImages but applies only to item pages. Item pages with fewer images than this threshold will be refetched.

      • getItemRequireAnchors

         final Integer getItemRequireAnchors()

        Minimum number of links required for item detail pages.

        Works identically to requireAnchors but applies only to item pages. Item pages with fewer links than this threshold will be refetched.

      • setItemRequireAnchors

         final Unit setItemRequireAnchors(Integer itemRequireAnchors)

        Minimum number of links required for item detail pages.

        Works identically to requireAnchors but applies only to item pages. Item pages with fewer links than this threshold will be refetched.

      • getPersist

         final Boolean getPersist()

        Controls whether fetched pages are immediately persisted to storage.

        When enabled, pages are saved to the database as soon as they're fetched. Disabling this can improve performance for tasks where persistence isn't needed.

      • setPersist

         final Unit setPersist(Boolean persist)

        Controls whether fetched pages are immediately persisted to storage.

        When enabled, pages are saved to the database as soon as they're fetched. Disabling this can improve performance for tasks where persistence isn't needed.

      • getStoreContent

         final Boolean getStoreContent()

        Controls whether page content (HTML) is stored in the database.

        The page content is typically the largest part of a page record. Disabling storage can significantly reduce database size when only metadata is needed.

      • setStoreContent

         final Unit setStoreContent(Boolean storeContent)

        Controls whether page content (HTML) is stored in the database.

        The page content is typically the largest part of a page record. Disabling storage can significantly reduce database size when only metadata is needed.

      • getDropContent

         final Boolean getDropContent()

        When enabled, page content (HTML) will not be stored in the database.

        This is the inverse of storeContent and takes precedence when both are specified. Useful for saving storage space when only page metadata is needed.

        Example:

        session.load(url, "-dropContent")

val options = session.options("-dropContent")
session.load(url, options)

      • setDropContent

         final Unit setDropContent(Boolean dropContent)

        When enabled, page content (HTML) will not be stored in the database.

        This is the inverse of storeContent and takes precedence when both are specified. Useful for saving storage space when only page metadata is needed.

        Example:

        session.load(url, "-dropContent")

val options = session.options("-dropContent")
session.load(url, options)

      • getRefresh

         final Boolean getRefresh()

        Forces an immediate fetch of the page, ignoring cache state and past failures.

        Acts as a shorthand for setting multiple options:

        • Sets expires to 0s (immediate expiration)

        • Enables ignoreFailure

        • Resets fetch retry counters

        Equivalent to clicking refresh in a browser - fetches the latest version regardless of cache.

      • getIgnoreFailure

         final Boolean getIgnoreFailure()

        Attempts to fetch a page even if previous fetch attempts have failed.

        By default, pages that have failed to fetch will be skipped until refresh is called. This option overrides that behavior and retries failed pages.

      • setIgnoreFailure

         final Unit setIgnoreFailure(Boolean ignoreFailure)

        Attempts to fetch a page even if previous fetch attempts have failed.

        By default, pages that have failed to fetch will be skipped until refresh is called. This option overrides that behavior and retries failed pages.

      • getNMaxRetry

         final Integer getNMaxRetry()

        Maximum number of fetch retries before marking a page as permanently failed.

        Once a page has been retried this many times without success, it will be considered "gone" and won't be attempted again unless explicitly refreshed.

      • setNMaxRetry

         final Unit setNMaxRetry(Integer nMaxRetry)

        Maximum number of fetch retries before marking a page as permanently failed.

        Once a page has been retried this many times without success, it will be considered "gone" and won't be attempted again unless explicitly refreshed.

      • getNJitRetry

         final Integer getNJitRetry()

        Maximum number of immediate retries during a single fetch operation.

        When a RETRY status code (1601) is encountered, the system will retry immediately up to this many times before giving up on the current attempt.

      • setNJitRetry

         final Unit setNJitRetry(Integer nJitRetry)

        Maximum number of immediate retries during a single fetch operation.

        When a RETRY status code (1601) is encountered, the system will retry immediately up to this many times before giving up on the current attempt.

      • getLazyFlush

         final Boolean getLazyFlush()

        Controls when pages are flushed to the database.

        When enabled, page writes are batched and delayed for better performance. When disabled, pages are written immediately for better data safety.

      • setLazyFlush

         final Unit setLazyFlush(Boolean lazyFlush)

        Controls when pages are flushed to the database.

        When enabled, page writes are batched and delayed for better performance. When disabled, pages are written immediately for better data safety.

      • getIncognito

         final Boolean getIncognito()

        Enables browser incognito/private mode for page fetching.

        NOTE: This setting has limited effect since browsers always run in temporary contexts in the current implementation.

      • setIncognito

         final Unit setIncognito(@Beta() Boolean incognito)

        Enables browser incognito/private mode for page fetching.

        NOTE: This setting has limited effect since browsers always run in temporary contexts in the current implementation.

      • getParse

         final Boolean getParse()

        Enables immediate parsing of fetched pages.

        When enabled, pages are parsed into a ai.platon.pulsar.dom.FeaturedDocument as soon as they're fetched, extracting links and other structured data. This is typically enabled for crawler operations.

      • setParse

         final Unit setParse(Boolean parse)

        Enables immediate parsing of fetched pages.

        When enabled, pages are parsed into a ai.platon.pulsar.dom.FeaturedDocument as soon as they're fetched, extracting links and other structured data. This is typically enabled for crawler operations.

      • getReparseLinks

         final Boolean getReparseLinks()

        Forces re-parsing of links even for previously parsed pages.

        When enabled, link extraction will be performed even if the page has been parsed before. Useful when link extraction rules have changed.

      • setReparseLinks

         final Unit setReparseLinks(@Beta() Boolean reparseLinks)

        Forces re-parsing of links even for previously parsed pages.

        When enabled, link extraction will be performed even if the page has been parsed before. Useful when link extraction rules have changed.

      • getIgnoreUrlQuery

         final Boolean getIgnoreUrlQuery()

        Removes query parameters from URLs during processing.

        When enabled, query strings (parameters after '?') are stripped from URLs. Useful for treating URLs with different query parameters as the same resource.

      • setIgnoreUrlQuery

         final Unit setIgnoreUrlQuery(@Beta() Boolean ignoreUrlQuery)

        Removes query parameters from URLs during processing.

        When enabled, query strings (parameters after '?') are stripped from URLs. Useful for treating URLs with different query parameters as the same resource.

      • getNoNorm

         final Boolean getNoNorm()

        Disables URL normalization during link processing.

        When enabled, extracted links are used exactly as found without normalization. This can lead to duplicate URLs in different formats being treated as distinct.

      • setNoNorm

         final Unit setNoNorm(@Beta() Boolean noNorm)

        Disables URL normalization during link processing.

        When enabled, extracted links are used exactly as found without normalization. This can lead to duplicate URLs in different formats being treated as distinct.

      • getNoFilter

         final Boolean getNoFilter()

        Disables URL filtering during link processing.

        When enabled, all extracted links are processed without filtering. This can lead to following links that would normally be excluded (like external domains).

      • setNoFilter

         final Unit setNoFilter(@Beta() Boolean noFilter)

        Disables URL filtering during link processing.

        When enabled, all extracted links are processed without filtering. This can lead to following links that would normally be excluded (like external domains).

      • setNetCondition

         final Unit setNetCondition(@Deprecated(message = "Use interactLevel instead", replaceWith = @ReplaceWith(imports = {}, expression = "options.interactLevel")) Condition netCondition)

        Deprecated network condition parameter, use interactLevel instead.

      • getInteractLevel

         final InteractLevel getInteractLevel()

        Controls the level of interaction with web pages during crawling.

        This setting balances content quality against performance:

        • Higher levels: More thorough interaction (scrolling, waiting, etc.) produces better content extraction but slower performance

        • Lower levels: Minimal interaction for faster crawling but potentially missing dynamic content

        This is the primary setting for controlling how aggressively the crawler interacts with pages to discover content.

      • setInteractLevel

         final Unit setInteractLevel(InteractLevel interactLevel)

        Controls the level of interaction with web pages during crawling.

        This setting balances content quality against performance:

        • Higher levels: More thorough interaction (scrolling, waiting, etc.) produces better content extraction but slower performance

        • Lower levels: Minimal interaction for faster crawling but potentially missing dynamic content

        This is the primary setting for controlling how aggressively the crawler interacts with pages to discover content.

      • getTest

         final Integer getTest()

        Enables test mode with various verbosity levels.

        Higher values produce more detailed logging and output. Set to 0 to disable test mode completely.

      • setTest

         final Unit setTest(Integer test)

        Enables test mode with various verbosity levels.

        Higher values produce more detailed logging and output. Set to 0 to disable test mode completely.

      • getVersion

         final String getVersion()

        Version identifier for the load options format.

        Used to track compatibility between different versions of the load options parser.

      • setVersion

         final Unit setVersion(String version)

        Version identifier for the load options format.

        Used to track compatibility between different versions of the load options parser.

      • getConf

         final VolatileConfig getConf()

      • parserEngaged

         final Boolean parserEngaged()

        Checks if the parser phase should be activated based on current settings.

        Returns true if either explicit parsing is enabled or if content validation requiring parsing is requested.

      • parse

         Boolean parse()

        Parses command-line arguments into this LoadOptions object.

        In addition to standard parameter parsing, it also:

        • Fixes special handling for zero-arity boolean parameters

        • Corrects the outLinkSelector format

        Returns:

        true if parsing was successful

      • createItemOptions

         LoadOptions createItemOptions()

        Creates a new LoadOptions instance specifically for item page processing.

        The new instance applies item-specific options to the main options for processing detail pages rather than index pages.

        Returns:

        a new LoadOptions instance optimized for item page processing

      • isExpired

         final Boolean isExpired(Instant prevFetchTime)

        Determines if a page should be considered expired based on its last fetch time.

        A page is considered expired when any of these conditions is true:

        • The refresh flag is set (forcing immediate refresh)

        • Current time is after expireAt and the last fetch was before expireAt

        • The time since last fetch exceeds the expires duration

        Parameters:
        prevFetchTime - when the page was last fetched
        Returns:

        true if the page should be fetched again

      • isDead

         final Boolean isDead()

        Checks if the task's deadline has passed and it should be abandoned.

        Returns:

        true if the current time is after the deadline

      • itemOptions2MajorOptions

         Unit itemOptions2MajorOptions()

        Converts item-specific options to main options for item page processing.

        This method is called when transitioning from processing index pages to detail pages, applying the item-specific settings as the new main settings.

      • overrideConfiguration

         final VolatileConfig overrideConfiguration()

        Copies option values to the configuration object for use by other components.

        Returns:

        the updated configuration object

      • overrideConfiguration

         final VolatileConfig overrideConfiguration(VolatileConfig conf)

        Copies option values to the provided configuration object for use by other components.

        Since LoadOptions isn't globally accessible, this method transfers settings to a VolatileConfig that can be passed to other modules.

        Parameters:
        conf - the configuration to update
        Returns:

        the updated configuration object

      • isDefault

         Boolean isDefault(String optionName)

        Checks if an option has its default value.

        Parameters:
        optionName - the option name without leading dash
        Returns:

        true if the option has its default value

      • getParams

         Params getParams()

        Converts this LoadOptions to a Params object containing all options.

        Returns:

        a Params object representing all options

      • toString

         String toString()

        Converts this LoadOptions to a normalized command-line string.

        Only includes options that differ from defaults. The resulting string can be parsed back into an equivalent LoadOptions object.

        Returns:

        a normalized command-line string representing these options

      • equals

         Boolean equals(Object other)

        Compares this LoadOptions with another object for equality.

        Two LoadOptions are considered equal if their normalized string representations are identical.

        Parameters:
        other - the object to compare with
        Returns:

        true if the objects are equal

      • hashCode

         Integer hashCode()

        Returns a hash code for this LoadOptions.

        Returns:

        the hash code

      • clone

         LoadOptions clone()

        Creates a copy of this LoadOptions with the same settings.

        Returns:

        a new LoadOptions instance with the same settings