Package ai.platon.pulsar.skeleton.common.options

Object LoadOptionDefaults

  • All Implemented Interfaces:
    
public class LoadOptionDefaults

    The default load options, be careful if you have to change the default behaviour.

    • Nested Class Summary

      Nested Classes 
      Modifier and Type Class Description

    • Constructor Summary

      Constructors 
      Constructor Description

    • Enum Constant Summary

      Enum Constants 
      Enum Constant Description

    • Method Summary

      Modifier and Type Method Description
      final Unit reset() Reset all the options to default.
      final Duration getEXPIRES() The default expiry time, some time we may need expire all pages by default, for example, in test mode
      final Instant getEXPIRE_AT() The default time to expire
      final BrowserType getBROWSER() The default browser is chrome with pulsar implemented web driver.
      final Duration getExpires() The default expiry time, some time we may need expire all pages by default, for example, in test mode
      final Unit setExpires(Duration expires) The default expiry time, some time we may need expire all pages by default, for example, in test mode
      final Instant getExpireAt() The default time to expire
      final Unit setExpireAt(Instant expireAt) The default time to expire
      final Boolean getLazyFlush() Lazy flush.
      final Unit setLazyFlush(Boolean lazyFlush) Lazy flush.
      final Boolean getParse() Trigger the parse phase or not.
      final Unit setParse(Boolean parse) Trigger the parse phase or not.
      final Boolean getStoreContent() Store webpage content or not.
      final Unit setStoreContent(Boolean storeContent) Store webpage content or not.
      final Boolean getIgnoreFailure() If true, still fetch the page even if it is gone.
      final Unit setIgnoreFailure(Boolean ignoreFailure) If true, still fetch the page even if it is gone.
      final Integer getNJitRetry() There are several cases to enable jit retry.
      final Unit setNJitRetry(Integer nJitRetry) There are several cases to enable jit retry.
      final BrowserType getBrowser() The default browser is chrome with pulsar implemented web driver.
      final Unit setBrowser(BrowserType browser) The default browser is chrome with pulsar implemented web driver.
      final Integer getTest() Set to be 0 if we are doing unit test or other test.
      final Unit setTest(Integer test) Set to be 0 if we are doing unit test or other test.

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    • Constructor Detail

    • Method Detail

      • reset

         final Unit reset()

        Reset all the options to default.

      • getEXPIRES

         final Duration getEXPIRES()

        The default expiry time, some time we may need expire all pages by default, for example, in test mode

      • getEXPIRE_AT

         final Instant getEXPIRE_AT()

        The default time to expire

      • getBROWSER

         final BrowserType getBROWSER()

        The default browser is chrome with pulsar implemented web driver.

      • getExpires

         final Duration getExpires()

        The default expiry time, some time we may need expire all pages by default, for example, in test mode

      • setExpires

         final Unit setExpires(Duration expires)

        The default expiry time, some time we may need expire all pages by default, for example, in test mode

      • getExpireAt

         final Instant getExpireAt()

        The default time to expire

      • setExpireAt

         final Unit setExpireAt(Instant expireAt)

        The default time to expire

      • getParse

         final Boolean getParse()

        Trigger the parse phase or not.

        Do not parse by default, since there are may ways to trigger a webpage parsing:

        • use session.parse()

        • add a -parse option

        • use a ai.platon.pulsar.crawl.common.url.ParsableHyperlink

      • setParse

         final Unit setParse(Boolean parse)

        Trigger the parse phase or not.

        Do not parse by default, since there are may ways to trigger a webpage parsing:

        • use session.parse()

        • add a -parse option

        • use a ai.platon.pulsar.crawl.common.url.ParsableHyperlink

      • getStoreContent

         final Boolean getStoreContent()

        Store webpage content or not.

        Store webpage content by default. If we are running a public cloud, this option might be changed to false.

      • setStoreContent

         final Unit setStoreContent(Boolean storeContent)

        Store webpage content or not.

        Store webpage content by default. If we are running a public cloud, this option might be changed to false.

      • getIgnoreFailure

         final Boolean getIgnoreFailure()

        If true, still fetch the page even if it is gone.

      • setIgnoreFailure

         final Unit setIgnoreFailure(Boolean ignoreFailure)

        If true, still fetch the page even if it is gone.

      • getNJitRetry

         final Integer getNJitRetry()

        There are several cases to enable jit retry. For example, in a test environment.

      • setNJitRetry

         final Unit setNJitRetry(Integer nJitRetry)

        There are several cases to enable jit retry. For example, in a test environment.

      • getBrowser

         final BrowserType getBrowser()

        The default browser is chrome with pulsar implemented web driver.

      • setBrowser

         final Unit setBrowser(BrowserType browser)

        The default browser is chrome with pulsar implemented web driver.

      • getTest

         final Integer getTest()

        Set to be 0 if we are doing unit test or other test. We will talk more, log more and trace more in test mode.

      • setTest

         final Unit setTest(Integer test)

        Set to be 0 if we are doing unit test or other test. We will talk more, log more and trace more in test mode.