-
- All Implemented Interfaces:
-
ai.platon.pulsar.crawl.Crawler,java.lang.AutoCloseable
public abstract class AbstractCrawler implements Crawler
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description public classAbstractCrawler.Companion
-
Field Summary
Fields Modifier and Type Field Description private final Integeridprivate final Stringnameprivate Function2<Integer, UrlAware, Duration>retryDelayPolicyprivate final AtomicBooleanclosedprivate final BooleanisActiveprivate final PulsarSessionsessionprivate final LoadOptionsdefaultOptionsprivate final BooleanautoCloseprivate final StringdefaultArgs
-
Constructor Summary
Constructors Constructor Description AbstractCrawler(PulsarContext context)AbstractCrawler(PulsarSession session, LoadOptions defaultOptions, Boolean autoClose)
-
Method Summary
Modifier and Type Method Description IntegergetId()The crawl id StringgetName()Function2<Integer, UrlAware, Duration>getRetryDelayPolicy()Delay policy for retry tasks UnitsetRetryDelayPolicy(Function2<Integer, UrlAware, Duration> retryDelayPolicy)Delay policy for retry tasks final AtomicBooleangetClosed()BooleangetIsActive()final PulsarSessiongetSession()LoadOptionsgetDefaultOptions()The default load options final BooleangetAutoClose()StringgetDefaultArgs()Unitclose()-
-
Constructor Detail
-
AbstractCrawler
AbstractCrawler(PulsarContext context)
-
AbstractCrawler
AbstractCrawler(PulsarSession session, LoadOptions defaultOptions, Boolean autoClose)
-
-
Method Detail
-
getRetryDelayPolicy
Function2<Integer, UrlAware, Duration> getRetryDelayPolicy()
Delay policy for retry tasks
-
setRetryDelayPolicy
Unit setRetryDelayPolicy(Function2<Integer, UrlAware, Duration> retryDelayPolicy)
Delay policy for retry tasks
-
getClosed
final AtomicBoolean getClosed()
-
getIsActive
Boolean getIsActive()
-
getSession
final PulsarSession getSession()
-
getDefaultOptions
LoadOptions getDefaultOptions()
The default load options
-
getAutoClose
final Boolean getAutoClose()
-
getDefaultArgs
String getDefaultArgs()
-
-
-
-