Package ai.platon.pulsar.crawl.component
Class BatchFetchComponent
-
- All Implemented Interfaces:
-
java.lang.AutoCloseable
public final class BatchFetchComponent extends FetchComponent
-
-
Field Summary
Fields Modifier and Type Field Description private final GlobalCacheglobalCacheprivate final WebDbwebDbprivate final GlobalCacheFactoryglobalCacheFactoryprivate final BooleanisActiveprivate final CoreMetricscoreMetricsprivate final ProtocolFactoryprotocolFactoryprivate final ImmutableConfigimmutableConfig
-
Constructor Summary
Constructors Constructor Description BatchFetchComponent(WebDb webDb, ImmutableConfig immutableConfig)BatchFetchComponent(WebDb webDb, GlobalCacheFactory globalCacheFactory, CoreMetrics coreMetrics, ProtocolFactory protocolFactory, ImmutableConfig immutableConfig)
-
Method Summary
Modifier and Type Method Description final GlobalCachegetGlobalCache()final WebDbgetWebDb()final GlobalCacheFactorygetGlobalCacheFactory()final Collection<WebPage>fetchAll(Iterable<String> urls, LoadOptions options)Fetch all the urls, config property 'fetch.concurrency' controls the concurrency level. final Collection<WebPage>parallelFetchAll(Iterable<String> urls, LoadOptions options)Parallel fetch all the urlsEager fetch only some urls to response as soon as possible, the rest urls will be fetched in background later final Collection<WebPage>parallelFetchAllGroupedBySchema(Iterable<String> urls, LoadOptions options)Group all urls by URL schema, and parallel fetch each group. -
Methods inherited from class ai.platon.pulsar.crawl.component.FetchComponent
close, fetch, fetch, fetchContent, fetchContent, fetchContentDeferred, getCoreMetrics, getImmutableConfig, getProtocolFactory, isActive -
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
-
-
Constructor Detail
-
BatchFetchComponent
BatchFetchComponent(WebDb webDb, ImmutableConfig immutableConfig)
-
BatchFetchComponent
BatchFetchComponent(WebDb webDb, GlobalCacheFactory globalCacheFactory, CoreMetrics coreMetrics, ProtocolFactory protocolFactory, ImmutableConfig immutableConfig)
-
-
Method Detail
-
getGlobalCache
final GlobalCache getGlobalCache()
-
getWebDb
final WebDb getWebDb()
-
getGlobalCacheFactory
final GlobalCacheFactory getGlobalCacheFactory()
-
fetchAll
final Collection<WebPage> fetchAll(Iterable<String> urls, LoadOptions options)
Fetch all the urls, config property 'fetch.concurrency' controls the concurrency level. If concurrency level is not great than 1, fetch all urls in the caller thread
Eager fetch only some urls to response as soon as possible, the rest urls will be fetched in background later
- Parameters:
urls- The urls to fetchoptions- The options
-
parallelFetchAll
final Collection<WebPage> parallelFetchAll(Iterable<String> urls, LoadOptions options)
Parallel fetch all the urls
Eager fetch only some urls to response as soon as possible, the rest urls will be fetched in background later
- Parameters:
urls- The urls to fetchoptions- The options
-
parallelFetchAllGroupedBySchema
final Collection<WebPage> parallelFetchAllGroupedBySchema(Iterable<String> urls, LoadOptions options)
Group all urls by URL schema, and parallel fetch each group.
Eager fetch only some urls to response as soon as possible, the rest urls will be fetched in background later
- Parameters:
urls- The urls to fetchoptions- The options
-
-
-
-