Package ai.platon.pulsar.context

Interface PulsarContext

  • All Implemented Interfaces:
    java.lang.AutoCloseable 
    
public interface PulsarContext
 implements AutoCloseable

    PulsarContext consists of a set of highly customizable components that provide the core set of interfaces of the system and is used to create PulsarSession.

    • Constructor Detail

    • Method Detail

      • getBean

         abstract <T extends Any> T getBean(KClass<T> requiredType)

        Get a bean with the specified class, throws BeansException if the bean doesn't exist

      • getBeanOrNull

         abstract <T extends Any> T getBeanOrNull(KClass<T> requiredType)

        Get a bean with the specified class, returns null if the bean doesn't exist

      • normalize

         abstract NormUrl normalize(String url, LoadOptions options, Boolean toItemOption)

        Normalize an url, the url can be in one of the following forms:

        • a normal url

        • a configured url

        • a base64 encoded url

        • a base64 encoded configured url

        An url can be configured by appending arguments to the url, and it also can be used with a LoadOptions, If both tailing arguments and LoadOptions are present, the LoadOptions overrides the tailing arguments, but default values in LoadOptions are ignored.

      • normalize

         abstract List<NormUrl> normalize(Iterable<String> urls, LoadOptions options, Boolean toItemOption)

        Normalize urls, remove invalid ones

        Parameters:
        urls - The urls to normalize
        options - The LoadOptions applied to each url
        toItemOption - If the LoadOptions is converted to item load options
        Returns:

        All normalized urls, all invalid input urls are removed

      • normalize

         abstract NormUrl normalize(UrlAware url, LoadOptions options, Boolean toItemOption)

        Normalize an url.

        If both url arguments and LoadOptions are present, the LoadOptions overrides the tailing arguments, but default values in LoadOptions are ignored.

      • normalize

         abstract List<NormUrl> normalize(Collection<UrlAware> urls, LoadOptions options, Boolean toItemOption)

        Normalize urls, remove invalid ones

        Parameters:
        urls - The urls to normalize
        options - The LoadOptions applied to each url
        toItemOption - If the LoadOptions is converted to item load options
        Returns:

        All normalized urls, all invalid input urls are removed

      • inject

         abstract WebPage inject(String url)

        Inject an url

        Parameters:
        url - The url followed by config options
        Returns:

        The web page created

      • inject

         abstract WebPage inject(NormUrl url)

        Inject an url

        Parameters:
        url - The url followed by config options
        Returns:

        The web page created

      • get

         abstract WebPage get(String url)

        Get a webpage from the storage

        Parameters:
        url - The url of the page to retrieve
        Returns:

        The web page, if the page doesn't exist, a NIL page is return

      • get

         abstract WebPage get(String url, String fields)

        Get a page from storage.

        Parameters:
        url - The url
        fields - The fields to load from local storage
        Returns:

        The webpage in storage if exists, otherwise returns a NIL page

      • getOrNull

         abstract WebPage getOrNull(String url)

        Get a webpage from the storage

        Parameters:
        url - The url of the page to retrieve
        Returns:

        The web page or null

      • getOrNull

         abstract WebPage getOrNull(String url, String fields)

        Get a page from storage.

        Parameters:
        url - The url
        fields - The fields to load from local storage
        Returns:

        The page in storage if exists, otherwise returns null

      • getContent

         abstract ByteBuffer getContent(String url)

        Get the content of the page from the storage

        Parameters:
        url - The url of the page to retrieve
        Returns:

        The page content or null

      • getContentAsString

        @Beta() abstract String getContentAsString(String url)

        Get the content of the page from the storage

        Parameters:
        url - The url of the page to retrieve
        Returns:

        The page content in string format or null

      • exists

         abstract Boolean exists(String url)

        Get a webpage from the storage

        Parameters:
        url - The url of the page to check
        Returns:

        True if the page exists in the storage

      • fetchState

         abstract CheckState fetchState(WebPage page, LoadOptions options)

        Check the fetch state of a webpage

        Parameters:
        page - The webpage to check
        Returns:

        The fetch state of the webpage

      • scan

         abstract Iterator<WebPage> scan(String urlPrefix)

        Scan webpages in the storage whose url start with urlPrefix

        Parameters:
        urlPrefix - The url prefix
        Returns:

        The iterator of the webpages whose url start with urlPrefix

      • load

         abstract WebPage load(String url, LoadOptions options)

        Load a url with specified options, see LoadOptions for all options

        Parameters:
        url - The url followed by options
        options - The options
        Returns:

        The WebPage. If there is no web page at local storage nor remote location, WebPage.NIL is returned

      • load

         abstract WebPage load(URL url, LoadOptions options)

        Load a url with specified options, see LoadOptions for all options

        Parameters:
        url - The url followed by options
        options - The options
        Returns:

        The WebPage. If there is no web page at local storage nor remote location, WebPage.NIL is returned

      • load

         abstract WebPage load(NormUrl url)

        Load a url, options can be specified following the url, see LoadOptions for all options

        Parameters:
        url - The url followed by options
        Returns:

        The WebPage. If there is no web page at local storage nor remote location, WebPage.NIL is returned

      • loadDeferred

         abstract WebPage loadDeferred(NormUrl url)

        Load a url, options can be specified following the url, see LoadOptions for all options

        Parameters:
        url - The url followed by options
        Returns:

        The WebPage. If there is no web page at local storage nor remote location, WebPage.NIL is returned

      • loadAll

         abstract Collection<WebPage> loadAll(Iterable<String> urls, LoadOptions options)

        Load a batch of urls with the specified options.

        If the option indicates prefer parallel, urls are fetched in a parallel manner whenever applicable. If the batch is too large, only a random part of the urls is fetched immediately, all the rest urls are put into a pending fetch list and will be fetched in background later.

        If a page exists neither in local storage nor at the given remote location, WebPage.NIL is returned

        Parameters:
        urls - The urls to load
        options - The options
        Returns:

        Pages for all urls.

      • loadAll

         abstract Collection<WebPage> loadAll(Iterable<NormUrl> urls)

        Load a batch of urls with the specified options.

        If the option indicates prefer parallel, urls are fetched in a parallel manner whenever applicable. If the batch is too large, only a random part of the urls is fetched immediately, all the rest urls are put into a pending fetch list and will be fetched in background later.

        If a page exists neither in local storage nor at the given remote location, WebPage.NIL is returned

        Parameters:
        urls - The urls to load
        Returns:

        Pages for all urls.

      • loadAsync

         abstract CompletableFuture<WebPage> loadAsync(NormUrl url)

        Load a url asynchronously, the url is added to the task queue, and will be executed asynchronously

      • submit

         abstract PulsarContext submit(UrlAware url)

        Submit a url, the url will be added to the task queue, and will be executed asynchronously.

        The url should be standard or degenerate, otherwise it will be discarded.

        Parameters:
        url - The url to submit, which will be added to the task queue
        Returns:

        The PulsarContext itself to enabled chained operations

      • submitAll

         abstract PulsarContext submitAll(Iterable<UrlAware> urls)

        Submit a batch of urls, the urls are added to the task queue, and will be executed asynchronously.

        The urls should be standard or degenerate, otherwise they will be discarded.

        Parameters:
        urls - The urls to submit, which will be added to the task queue
        Returns:

        The PulsarContext itself to enabled chained operations

      • parse

         abstract FeaturedDocument parse(WebPage page)

        Parse the WebPage using ParseComponent

      • persist

         abstract Unit persist(WebPage page)

        Persist the webpage into the storage immediately. By default, the backend storage is the local file system, if mongodb is detected, the mongodb will be used as the backend storage.

      • delete

         abstract Unit delete(String url)

        Delete the webpage from the backend storage

      • delete

         abstract Unit delete(WebPage page)

        Delete the webpage from the backend storage

      • flush

         abstract Unit flush()

        Flush the backend storage

      • await

         abstract Unit await()

        Wait until all tasks are done.

      • getUnmodifiedConfig

         abstract ImmutableConfig getUnmodifiedConfig()

        An immutable config loaded from the config file at startup, and never changes

      • getCrawlPool

         abstract UrlPool getCrawlPool()

        The url pool to fetch