Package ai.platon.pulsar.skeleton.context

Interface PulsarContext

  • All Implemented Interfaces:
    java.lang.AutoCloseable 
    
public interface PulsarContext
 implements AutoCloseable

    PulsarContext consists of a set of highly customizable components that provide the core set of interfaces of the system and is used to create PulsarSession.

    • Nested Class Summary

      Nested Classes 
      Modifier and Type Class Description

    • Field Summary

      Fields 
      Modifier and Type Field Description

    • Constructor Summary

      Constructors 
      Constructor Description

    • Enum Constant Summary

      Enum Constants 
      Enum Constant Description

    • Constructor Detail

    • Method Detail

      • getBean

         abstract <T extends Any> T getBean(KClass<T> requiredType)

        Get a bean with the specified class, throws BeansException if the bean doesn't exist

      • getBeanOrNull

         abstract <T extends Any> T getBeanOrNull(KClass<T> requiredType)

        Get a bean with the specified class, returns null if the bean doesn't exist

      • registerClosable

         abstract Unit registerClosable(AutoCloseable closable, Integer priority)

        Register closable objects, the objects will be closed when the context closes. It's safe to register the same object multiple times, the object will be closed only once.

        Parameters:
        closable - The object to close
        priority - The priority of the object, the higher the priority, the earlier the object is closed

      • normalize

         abstract NormURL normalize(String url, LoadOptions options, Boolean toItemOption)

        Normalize an url, the url can be in one of the following forms:

        • a normal url

        • a configured url

        • a base64 encoded url

        • a base64 encoded configured url

        An url can be configured by appending arguments to the url, and it also can be used with a LoadOptions, If both tailing arguments and LoadOptions are present, the LoadOptions overrides the tailing arguments, but default values in LoadOptions are ignored.

        Parameters:
        url - The url to normalize
        options - The LoadOptions applied to the url
        toItemOption - If the LoadOptions is converted to item load options
        Returns:

        The normalized url or NIL if the input url is invalid

      • normalize

         abstract List<NormURL> normalize(Iterable<String> urls, LoadOptions options, Boolean toItemOption)

        Normalize urls, remove invalid ones

        Parameters:
        urls - The urls to normalize
        options - The LoadOptions applied to each url
        toItemOption - If the LoadOptions is converted to item load options
        Returns:

        All normalized urls, all invalid input urls are removed

      • normalize

         abstract NormURL normalize(UrlAware url, LoadOptions options, Boolean toItemOption)

        Normalize an url.

        If both url arguments and LoadOptions are present, the LoadOptions overrides the tailing arguments, but default values in LoadOptions are ignored.

        Parameters:
        url - The url to normalize
        options - The LoadOptions applied to the url
        toItemOption - If the LoadOptions is converted to item load options
        Returns:

        The normalized url or NIL if the input url is invalid

      • normalize

         abstract List<NormURL> normalize(Collection<UrlAware> urls, LoadOptions options, Boolean toItemOption)

        Normalize urls, remove invalid ones

        Parameters:
        urls - The urls to normalize
        options - The LoadOptions applied to each url
        toItemOption - If the LoadOptions is converted to item load options
        Returns:

        All normalized urls, all invalid input urls are removed

      • normalizeOrNull

         abstract NormURL normalizeOrNull(String url, LoadOptions options, Boolean toItemOption)

        Normalize an url, the url can be in one of the following forms:

        • a normal url

        • a configured url

        • a base64 encoded url

        • a base64 encoded configured url

        An url can be configured by appending arguments to the url, and it also can be used with a LoadOptions, If both tailing arguments and LoadOptions are present, the LoadOptions overrides the tailing arguments, but default values in LoadOptions are ignored.

        Parameters:
        url - The url to normalize
        options - The LoadOptions applied to the url
        toItemOption - If the LoadOptions is converted to item load options
        Returns:

        The normalized url or null if the input url is invalid

      • normalizeOrNull

         abstract NormURL normalizeOrNull(UrlAware url, LoadOptions options, Boolean toItemOption)

        Normalize an url, the url can be in one of the following forms:

        • a normal url

        • a configured url

        • a base64 encoded url

        • a base64 encoded configured url

        An url can be configured by appending arguments to the url, and it also can be used with a LoadOptions, If both tailing arguments and LoadOptions are present, the LoadOptions overrides the tailing arguments, but default values in LoadOptions are ignored.

        Parameters:
        url - The url to normalize
        options - The LoadOptions applied to the url
        toItemOption - If the LoadOptions is converted to item load options
        Returns:

        The normalized url or null if the input url is invalid

      • get

         abstract WebPage get(String url)

        Get a webpage from the storage

        Parameters:
        url - The url of the page to retrieve
        Returns:

        The web page, if the page doesn't exist, a NIL page is return

      • get

         abstract WebPage get(String url, String fields)

        Get a page from storage.

        Parameters:
        url - The url
        fields - The fields to load from local storage
        Returns:

        The webpage in storage if exists, otherwise returns a NIL page

      • getOrNull

         abstract WebPage getOrNull(String url)

        Get a webpage from the storage

        Parameters:
        url - The url of the page to retrieve
        Returns:

        The web page or null

      • getOrNull

         abstract WebPage getOrNull(String url, String fields)

        Get a page from storage.

        Parameters:
        url - The url
        fields - The fields to load from local storage
        Returns:

        The page in storage if exists, otherwise returns null

      • getContent

         abstract ByteBuffer getContent(String url)

        Get the content of the page from the storage

        Parameters:
        url - The url of the page to retrieve
        Returns:

        The page content or null

      • getContentAsString

        @Beta() abstract String getContentAsString(String url)

        Get the content of the page from the storage

        Parameters:
        url - The url of the page to retrieve
        Returns:

        The page content in string format or null

      • exists

         abstract Boolean exists(String url)

        Get a webpage from the storage

        Parameters:
        url - The url of the page to check
        Returns:

        True if the page exists in the storage

      • fetchState

         abstract CheckState fetchState(WebPage page, LoadOptions options)

        Check the fetch state of a webpage

        Parameters:
        page - The webpage to check
        Returns:

        The fetch state of the webpage

      • scan

         abstract Iterator<WebPage> scan(String urlPrefix)

        Scan webpages in the storage whose url start with urlPrefix

        Parameters:
        urlPrefix - The url prefix
        Returns:

        The iterator of the webpages whose url start with urlPrefix

      • open

         abstract WebPage open(String url, WebDriver driver, LoadOptions options)

        Open a webpage with options and a web driver

        Parameters:
        url - The url to open
        driver - The web driver
        options - The options
        Returns:

        The web page

      • load

         abstract WebPage load(String url, LoadOptions options)

        Load a url with specified options, see LoadOptions for all options

        Parameters:
        url - The url followed by options
        options - The options
        Returns:

        The WebPage. If there is no web page at local storage nor remote location, WebPageImpl.NIL is returned

      • load

         abstract WebPage load(URL url, LoadOptions options)

        Load a url with specified options, see LoadOptions for all options

        Parameters:
        url - The url followed by options
        options - The options
        Returns:

        The WebPage. If there is no web page at local storage nor remote location, WebPageImpl.NIL is returned

      • load

         abstract WebPage load(NormURL url)

        Load a url, options can be specified following the url, see LoadOptions for all options

        Parameters:
        url - The url followed by options
        Returns:

        The WebPage. If there is no web page at local storage nor remote location, WebPageImpl.NIL is returned

      • loadDeferred

         abstract WebPage loadDeferred(NormURL url)

        Load a url, options can be specified following the url, see LoadOptions for all options

        Parameters:
        url - The url followed by options
        Returns:

        The WebPage. If there is no web page at local storage nor remote location, WebPageImpl.NIL is returned

      • loadAll

         abstract Collection<WebPage> loadAll(Iterable<String> urls, LoadOptions options)

        Load a batch of urls with the specified options.

        If the option indicates prefer parallel, urls are fetched in a parallel manner whenever applicable. If the batch is too large, only a random part of the urls is fetched immediately, all the rest urls are put into a pending fetch list and will be fetched in background later.

        If a page exists neither in local storage nor at the given remote location, WebPageImpl.NIL is returned

        Parameters:
        urls - The urls to load
        options - The options
        Returns:

        Pages for all urls.

      • loadAll

         abstract Collection<WebPage> loadAll(Iterable<NormURL> urls)

        Load a batch of urls with the specified options.

        If the option indicates prefer parallel, urls are fetched in a parallel manner whenever applicable. If the batch is too large, only a random part of the urls is fetched immediately, all the rest urls are put into a pending fetch list and will be fetched in background later.

        If a page exists neither in local storage nor at the given remote location, WebPageImpl.NIL is returned

        Parameters:
        urls - The urls to load
        Returns:

        Pages for all urls.

      • loadAsync

         abstract CompletableFuture<WebPage> loadAsync(NormURL url)

        Load a url asynchronously, the url is added to the task queue, and will be executed asynchronously

      • submit

         abstract PulsarContext submit(UrlAware url)

        Submit a url, the url will be added to the task queue, and will be executed asynchronously.

        The url should be standard or degenerate, otherwise it will be discarded.

        Parameters:
        url - The url to submit, which will be added to the task queue
        Returns:

        The PulsarContext itself to enabled chained operations

      • submitAll

         abstract PulsarContext submitAll(Iterable<UrlAware> urls)

        Submit a batch of urls, the urls are added to the task queue, and will be executed asynchronously.

        The urls should be standard or degenerate, otherwise they will be discarded.

        Parameters:
        urls - The urls to submit, which will be added to the task queue
        Returns:

        The PulsarContext itself to enabled chained operations

      • parse

         abstract FeaturedDocument parse(WebPage page)

        Parse the WebPage using ParseComponent

      • chat

         abstract ModelResponse chat(String prompt)

        Chat with the AI model.

        Parameters:
        prompt - The prompt to chat

      • chat

         abstract ModelResponse chat(String userMessage, String systemMessage)

        Chat with the AI model.

        Parameters:
        userMessage - The user message to chat
        systemMessage - The system message to chat

      • persist

         abstract Unit persist(WebPage page)

        Persist the webpage into the storage immediately. By default, the backend storage is the local file system, if mongodb is detected, the mongodb will be used as the backend storage.

      • delete

         abstract Unit delete(String url)

        Delete the webpage from the backend storage

      • delete

         abstract Unit delete(WebPage page)

        Delete the webpage from the backend storage

      • flush

         abstract Unit flush()

        Flush the backend storage

      • await

         abstract Unit await()

        Wait until all tasks are done.

      • isActive

         abstract Boolean isActive()

        Flag that indicates whether this context is currently active.

      • getUnmodifiedConfig

         abstract ImmutableConfig getUnmodifiedConfig()

        An immutable config loaded from the config file at startup, and never changes

      • getCrawlPool

         abstract UrlPool getCrawlPool()

        The url pool to fetch