Package ai.platon.pulsar.skeleton.crawl.common

Class GlobalCache

  • All Implemented Interfaces:
    
public class GlobalCache

    The global cache.

    • Nested Class Summary

      Nested Classes 
      Modifier and Type Class Description

    • Constructor Summary

      Constructors 
      Constructor Description
      GlobalCache(ImmutableConfig conf)

    • Enum Constant Summary

      Enum Constants 
      Enum Constant Description

    • Method Summary

      Modifier and Type Method Description
      UrlPool getUrlPool() A url pool contains many url caches, the urls added to the pool will be processed in crawl loops.
      Unit setUrlPool(UrlPool urlPool) A url pool contains many url caches, the urls added to the pool will be processed in crawl loops.
      FetchingCache getFetchingCache() Fetching cache holds the URLs being fetched.
      ConcurrentExpiringLRUCache<String, WebPage> getPageCache() The global page cache, a page will be removed automatically if it's expired or the cache is full.
      ConcurrentExpiringLRUCache<String, FeaturedDocument> getDocumentCache() The global document cache, a document will be removed automatically if it's expired or the cache is full.
      final ImmutableConfig getConf()
      final Unit resetCaches() Reset all caches.
      final Unit clearCaches() Clear all caches.
      final Unit clearPDCaches() Clear page cache and document cache.
      final Unit putPDCache(WebPage page, FeaturedDocument document) Put the page and the document in the cache.
      final Unit removePDCache(String url) Remove items specified by the url from page cache and document cache.

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    • Constructor Detail

      • GlobalCache

        GlobalCache(ImmutableConfig conf)

    • Method Detail

      • getUrlPool

         UrlPool getUrlPool()

        A url pool contains many url caches, the urls added to the pool will be processed in crawl loops.

      • setUrlPool

         Unit setUrlPool(UrlPool urlPool)

        A url pool contains many url caches, the urls added to the pool will be processed in crawl loops.

      • getFetchingCache

         FetchingCache getFetchingCache()

        Fetching cache holds the URLs being fetched.

        URLs are cached before being fetched and removed from the cache after retrieval.

        The cache is used to avoid fetching the same URL multiple times.

      • getPageCache

         ConcurrentExpiringLRUCache<String, WebPage> getPageCache()

        The global page cache, a page will be removed automatically if it's expired or the cache is full.

      • getDocumentCache

         ConcurrentExpiringLRUCache<String, FeaturedDocument> getDocumentCache()

        The global document cache, a document will be removed automatically if it's expired or the cache is full.

      • getConf

         final ImmutableConfig getConf()

      • resetCaches

         final Unit resetCaches()

        Reset all caches. After this operation, all caches will be empty.

      • clearCaches

         final Unit clearCaches()

        Clear all caches. After this operation, all caches will be empty.

      • clearPDCaches

         final Unit clearPDCaches()

        Clear page cache and document cache. After this operation, page cache and document cache will be empty.

      • putPDCache

         final Unit putPDCache(WebPage page, FeaturedDocument document)

        Put the page and the document in the cache.

      • removePDCache

         final Unit removePDCache(String url)

        Remove items specified by the url from page cache and document cache.