Package ai.platon.pulsar.crawl.schedule

Class NewsFetchSchedule

  • All Implemented Interfaces:
    ai.platon.pulsar.common.config.Parameterized , ai.platon.pulsar.crawl.schedule.FetchSchedule 
    
public final class NewsFetchSchedule
extends AdaptiveFetchSchedule

    This class implements an adaptive re-fetch algorithm.

    NOTE: values of DEC_FACTOR and INC_FACTOR higher than 0.4f may destabilize the algorithm, so that the fetch interval either increases or decreases infinitely, with little relevance to the page changes

    • Nested Class Summary

      Nested Classes 
      Modifier and Type Class Description

    • Enum Constant Summary

      Enum Constants 
      Enum Constant Description

    • Method Summary

      Modifier and Type Method Description
      Unit setFetchSchedule(WebPage page, ModifyInfo m) Sets the fetchInterval and fetchTime on a successfully fetched page.

      • Methods inherited from class ai.platon.pulsar.crawl.schedule.AbstractFetchSchedule

        estimatePrevFetchTime, forceRefetch, getConf, getMaxFetchInterval, getMessageWriter, initializeSchedule, setPageRetrySchedule, shouldFetch

      • Methods inherited from class ai.platon.pulsar.crawl.schedule.AdaptiveFetchSchedule

        getParams, setPageGoneSchedule

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    • Constructor Detail

      • NewsFetchSchedule

        NewsFetchSchedule(ImmutableConfig conf, MiscMessageWriter messageWriter)

    • Method Detail

      • setFetchSchedule

         Unit setFetchSchedule(WebPage page, ModifyInfo m)

        Sets the fetchInterval and fetchTime on a successfully fetched page. NOTE: this implementation resets the retry counter - extending classes should call super.setFetchSchedule() to preserve this behavior.

        Parameters:
        page - The Web page
        m - The modification info