Constructor Detail

Method Detail

• getTaskTime

   final Instant getTaskTime()

  The default task time.

• setTaskTime

   final Unit setTaskTime(Instant taskTime)

  The default task time.

• getExpires

   final Duration getExpires()

  The default expiry time, some time we may need expire all pages by default, for example, in test mode

• setExpires

   final Unit setExpires(Duration expires)

  The default expiry time, some time we may need expire all pages by default, for example, in test mode

• getExpireAt

   final Instant getExpireAt()

  The default time to expire

• setExpireAt

   final Unit setExpireAt(Instant expireAt)

  The default time to expire

• getLazyFlush

   final Boolean getLazyFlush()

  Lazy flush.

• setLazyFlush

   final Unit setLazyFlush(Boolean lazyFlush)

  Lazy flush.

• getParse

   final Boolean getParse()

  Trigger the parse phase or not.

  Do not parse by default, since there are may ways to trigger a webpage parsing:

  • use session.parse()

  • add a -parse option

  • use a ai.platon.pulsar.crawl.common.url.ParsableHyperlink

• setParse

   final Unit setParse(Boolean parse)

  Trigger the parse phase or not.

  Do not parse by default, since there are may ways to trigger a webpage parsing:

  • use session.parse()

  • add a -parse option

  • use a ai.platon.pulsar.crawl.common.url.ParsableHyperlink

• getStoreContent

   final Boolean getStoreContent()

  Store webpage content or not.

  Store webpage content by default. If we are running a public cloud, this option might be changed to false.

• setStoreContent

   final Unit setStoreContent(Boolean storeContent)

  Store webpage content or not.

  Store webpage content by default. If we are running a public cloud, this option might be changed to false.

• getIgnoreFailure

   final Boolean getIgnoreFailure()

  If true, still fetch the page even if it is gone.

• setIgnoreFailure

   final Unit setIgnoreFailure(Boolean ignoreFailure)

  If true, still fetch the page even if it is gone.

• getNJitRetry

   final Integer getNJitRetry()

  There are several cases to enable jit retry. For example, in a test environment.

• setNJitRetry

   final Unit setNJitRetry(Integer nJitRetry)

  There are several cases to enable jit retry. For example, in a test environment.

• getBrowser

   final BrowserType getBrowser()

  The default browser is chrome with pulsar implemented web driver.

• setBrowser

   final Unit setBrowser(BrowserType browser)

  The default browser is chrome with pulsar implemented web driver.

• getTest

   final Integer getTest()

  Set to be 0 if we are doing unit test or other test. We will talk more, log more and trace more in test mode.

• setTest

   final Unit setTest(Integer test)

  Set to be 0 if we are doing unit test or other test. We will talk more, log more and trace more in test mode.