Package 

Class AbstractPulsarContext

  • All Implemented Interfaces:
    ai.platon.pulsar.context.PulsarContext , java.lang.AutoCloseable 
    
public abstract class AbstractPulsarContext
 implements PulsarContext, AutoCloseable

    The main entry point for pulsar functionality.

    A PulsarContext can be used to inject, fetch, load, parse, store webpages.

    • Constructor Detail

      • AbstractPulsarContext

        AbstractPulsarContext(AbstractApplicationContext applicationContext, PulsarEnvironment pulsarEnvironment)

    • Method Detail

      • getBean

         <T extends Any> T getBean(KClass<T> requiredType)

        Get a bean with the specified class, throws BeansException if the bean doesn't exist

      • getBean

         final <T extends Any> T getBean()

        Get a bean with the specified class, throws BeansException if the bean doesn't exist

      • getBeanOrNull

         <T extends Any> T getBeanOrNull(KClass<T> requiredType)

        Get a bean with the specified class, returns null if the bean doesn't exist

      • getBeanOrNull

         final <T extends Any> T getBeanOrNull()

        Get a bean with the specified class, returns null if the bean doesn't exist

      • normalize

         NormUrl normalize(String url, LoadOptions options, Boolean toItemOption)

        Normalize an url, the url can be one of the following:

        • a normal url

        • a configured url

        • a base64 encoded url

        • a base64 encoded configured url

        An url can be configured by appending arguments to the url, and it also can be used with a LoadOptions, If both tailing arguments and LoadOptions are present, the LoadOptions overrides the tailing arguments, but default values in LoadOptions are ignored.

      • normalize

         List<NormUrl> normalize(Iterable<String> urls, LoadOptions options, Boolean toItemOption)

        Normalize urls, remove invalid ones

        Parameters:
        urls - The urls to normalize
        options - The LoadOptions applied to each url
        toItemOption - If the LoadOptions is converted to item load options

      • normalize

         NormUrl normalize(UrlAware url, LoadOptions options, Boolean toItemOption)

        Normalize an url.

        If both url arguments and LoadOptions are present, the LoadOptions overrides the tailing arguments, but default values in LoadOptions are ignored.

      • normalize

         List<NormUrl> normalize(Collection<UrlAware> urls, LoadOptions options, Boolean toItemOption)

        Normalize urls, remove invalid ones

        Parameters:
        urls - The urls to normalize
        options - The LoadOptions applied to each url
        toItemOption - If the LoadOptions is converted to item load options

      • inject

         WebPage inject(String url)

        Inject an url

        Parameters:
        url - The url followed by options

      • inject

         WebPage inject(NormUrl url)

        Inject an url

        Parameters:
        url - The url followed by options

      • get

         WebPage get(String url)

        Get a webpage from the storage

        Parameters:
        url - The url of the page to retrieve

      • getOrNull

         WebPage getOrNull(String url)

        Get a webpage from the storage

        Parameters:
        url - The url of the page to retrieve

      • exists

         Boolean exists(String url)

        Check if a page exists in the storage

        Parameters:
        url - The url of the page to check

      • fetchState

         CheckState fetchState(WebPage page, LoadOptions options)

        Check the fetch state of a page

        Parameters:
        page - The webpage to check

      • scan

         Iterator<WebPage> scan(String urlPrefix)

        Scan pages in the storage

        Parameters:
        urlPrefix - The url prefix

      • load

         WebPage load(String url, LoadOptions options)

        Load an page with specified options, see LoadOptions for all options

        Parameters:
        url - The url followed by options
        options - The options

      • load

         WebPage load(URL url, LoadOptions options)

        Load a url with specified options, see LoadOptions for all options

        Parameters:
        url - The url followed by options
        options - The options

      • load

         WebPage load(NormUrl url)

        Load a url, options can be specified following the url, see LoadOptions for all options

        Parameters:
        url - The url followed by options

      • loadDeferred

         WebPage loadDeferred(NormUrl url)

        Load a url, options can be specified following the url, see LoadOptions for all options

        Parameters:
        url - The url followed by options

      • loadAll

         List<WebPage> loadAll(Iterable<String> urls, LoadOptions options)

        Load a batch of urls with the specified options.

        If the option indicates prefer parallel, urls are fetched in a parallel manner whenever applicable. If the batch is too large, only a random part of the urls is fetched immediately, all the rest urls are put into a pending fetch list and will be fetched in background later.

        If a page exists neither in local storage nor at the given remote location, WebPage.NIL is returned

        Parameters:
        urls - The urls to load
        options - The options

      • loadAll

         List<WebPage> loadAll(Iterable<NormUrl> urls)

        Load a batch of urls with the specified options.

        If the option indicates prefer parallel, urls are fetched in a parallel manner whenever applicable. If the batch is too large, only a random part of the urls is fetched immediately, all the rest urls are put into a pending fetch list and will be fetched in background later.

        If a page exists neither in local storage nor at the given remote location, WebPage.NIL is returned

        Parameters:
        urls - The urls to load

      • loadAsync

         CompletableFuture<WebPage> loadAsync(NormUrl url)

        Load a url asynchronously, the url is added to the crawl task queue, and will be executed asynchronously

      • submit

         AbstractPulsarContext submit(UrlAware url)

        Submit a url, the url is added to the crawl task queue, and will be executed asynchronously

      • parse

         FeaturedDocument parse(WebPage page)

        Parse the WebPage content using parseComponent

      • persist

         Unit persist(WebPage page)

        Persist the page into the storage

      • delete

         Unit delete(WebPage page)

        Delete the page from the storage

      • await

         Unit await()

        Wait until there is no tasks in the main loop

      • registerShutdownHook

         Unit registerShutdownHook()

        Register a shutdown hook with the JVM runtime, closing this context on JVM shutdown unless it has already been closed at that time.

        Delegates to doClose() for the actual closing procedure.

      • close

         Unit close()

        Close this pulsar context

        Delegates to doClose() for the actual closing procedure. Also removes a JVM shutdown hook, if registered, as it's not needed anymore.