Package ai.platon.pulsar.skeleton.crawl.schedule

Class AbstractFetchSchedule

  • All Implemented Interfaces:
    ai.platon.pulsar.common.config.Parameterized , ai.platon.pulsar.skeleton.crawl.schedule.FetchSchedule 
    
public abstract class AbstractFetchSchedule
 implements FetchSchedule

    • Nested Class Summary

      Nested Classes 
      Modifier and Type Class Description

    • Enum Constant Summary

      Enum Constants 
      Enum Constant Description

    • Method Summary

      Modifier and Type Method Description
      Duration getMaxFetchInterval()
      final ImmutableConfig getConf()
      final MiscMessageWriter getMessageWriter()
      Unit initializeSchedule(WebPage page)
      Unit setFetchSchedule(WebPage page, ModifyInfo m)
      Unit setPageRetrySchedule(WebPage page, Instant prevFetchTime, Instant prevModifiedTime, Instant fetchTime) This method adjusts the fetch schedule if fetching needs to be re-tried due to transient errors.
      Unit setPageGoneSchedule(WebPage page, Instant prevFetchTime, Instant prevModifiedTime, Instant fetchTime) This method specifies how to schedule refetching of pages marked as GONE.
      Instant estimatePrevFetchTime(WebPage page) This method return the last fetch time of the WebPage
      Boolean shouldFetch(WebPage page, Instant now) This method provides information whether the page is suitable for selection in the current fetchlist.
      Unit forceRefetch(WebPage page, Instant prevFetchTime, Boolean asap) This method resets fetchTime, fetchInterval, modifiedTime, retriesSinceFetch and page text, so that it forces refetching.

      • Methods inherited from class ai.platon.pulsar.common.config.Parameterized

        getParams

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    • Constructor Detail

      • AbstractFetchSchedule

        AbstractFetchSchedule(ImmutableConfig conf, MiscMessageWriter messageWriter)

    • Method Detail

      • getConf

         final ImmutableConfig getConf()

      • setPageRetrySchedule

         Unit setPageRetrySchedule(WebPage page, Instant prevFetchTime, Instant prevModifiedTime, Instant fetchTime)

        This method adjusts the fetch schedule if fetching needs to be re-tried due to transient errors. The default implementation sets the next fetch time 1 day in the future and increases the retry counter.

        Parameters:
        page - WebPage to retry
        prevFetchTime - previous fetch time
        prevModifiedTime - previous modified time
        fetchTime - current fetch time

      • setPageGoneSchedule

         Unit setPageGoneSchedule(WebPage page, Instant prevFetchTime, Instant prevModifiedTime, Instant fetchTime)

        This method specifies how to schedule refetching of pages marked as GONE. Default implementation increases fetchInterval by 50% but the value may never exceed maxInterval.

        Returns:

        adjusted page information, including all original information. NOTE: this may be a different instance than

      • estimatePrevFetchTime

         Instant estimatePrevFetchTime(WebPage page)

        This method return the last fetch time of the WebPage

        Returns:

        the date as a long.

      • shouldFetch

         Boolean shouldFetch(WebPage page, Instant now)

        This method provides information whether the page is suitable for selection in the current fetchlist. NOTE: a true return value does not guarantee that the page will be fetched, it just allows it to be included in the further selection process based on scores. The default implementation checks fetchTime, if it is higher than the

        Parameters:
        page - Web page to fetch
        now - reference time (usually set to the time when the fetchlist generation process was started).
        Returns:

        true, if the page should be considered for inclusion in the current fetchlist, otherwise false.

      • forceRefetch

         Unit forceRefetch(WebPage page, Instant prevFetchTime, Boolean asap)

        This method resets fetchTime, fetchInterval, modifiedTime, retriesSinceFetch and page text, so that it forces refetching.

        Parameters:
        asap - if true, force refetch as soon as possible - this sets the fetchTime to now.