Constructor Detail

• AbstractPulsarContext

  AbstractPulsarContext(AbstractApplicationContext applicationContext, PulsarEnvironment pulsarEnvironment)

Method Detail

• isActive

   Boolean isActive()

  Flag that indicates whether this context is currently active.

• getId

   Integer getId()

  The context id

• getUnmodifiedConfig

   ImmutableConfig getUnmodifiedConfig()

  An immutable config loaded from the config file at startup, and never changes

• getUrlNormalizer

   ChainedUrlNormalizer getUrlNormalizer()

  The url normalizer

• getUrlNormalizerOrNull

   ChainedUrlNormalizer getUrlNormalizerOrNull()

• getWebDb

   WebDb getWebDb()

• getGlobalCacheFactory

   GlobalCacheFactory getGlobalCacheFactory()

• getFetchComponent

   BatchFetchComponent getFetchComponent()

• getParseComponent

   ParseComponent getParseComponent()

• getUpdateComponent

   UpdateComponent getUpdateComponent()

• getLoadComponent

   LoadComponent getLoadComponent()

• getGlobalCache

   GlobalCache getGlobalCache()

  The global cache

• getCrawlPool

   UrlPool getCrawlPool()

  The url pool to fetch

• getCrawlLoops

   CrawlLoops getCrawlLoops()

  The main loops

• getStartTime

   final Long getStartTime()

  The start time

• getSessions

   final ConcurrentSkipListMap<Integer, AbstractPulsarSession> getSessions()

  All open sessions

• getApplicationContext

   final AbstractApplicationContext getApplicationContext()

• getPulsarEnvironment

   final PulsarEnvironment getPulsarEnvironment()

• getBean

   <T extends Any> T getBean(KClass<T> requiredType)

  Get a bean with the specified class, throws BeansException if the bean doesn't exist

• getBean

   final <T extends Any> T getBean()

  Get a bean with the specified class, throws BeansException if the bean doesn't exist

• getBeanOrNull

   <T extends Any> T getBeanOrNull(KClass<T> requiredType)

  Get a bean with the specified class, returns null if the bean doesn't exist

• getBeanOrNull

   final <T extends Any> T getBeanOrNull()

  Get a bean with the specified class, returns null if the bean doesn't exist

• createSession

   abstract AbstractPulsarSession createSession()

  Create a session

• getOrCreateSession

   AbstractPulsarSession getOrCreateSession()

  Create a pulsar session

• closeSession

   Unit closeSession(PulsarSession session)

  Close the given session

• registerClosable

   Unit registerClosable(AutoCloseable closable, Integer priority)

  Register close objects, the objects will be closed when the context closes

  Parameters:

  closable - The object to close

  priority - The priority of the object, the higher the priority, the earlier the object is closed

• normalize

   NormURL normalize(String url, LoadOptions options, Boolean toItemOption)

  Normalize an url, the url can be in one of the following forms:

  • a normal url

  • a configured url

  • a base64 encoded url

  • a base64 encoded configured url

  An url can be configured by appending arguments to the url, and it also can be used with a LoadOptions, If both tailing arguments and LoadOptions are present, the LoadOptions overrides the tailing arguments, but default values in LoadOptions are ignored.

  Parameters:

  url - The url to normalize

  options - The LoadOptions applied to the url

  toItemOption - If the LoadOptions is converted to item load options

  Returns:

  The normalized url or NIL if the input url is invalid

• normalize

   List<NormURL> normalize(Iterable<String> urls, LoadOptions options, Boolean toItemOption)

  Normalize urls, remove invalid ones

  Parameters:

  urls - The urls to normalize

  options - The LoadOptions applied to each url

  toItemOption - If the LoadOptions is converted to item load options

  Returns:

  All normalized urls, all invalid input urls are removed

• normalize

   NormURL normalize(UrlAware url, LoadOptions options, Boolean toItemOption)

  Normalize an url.

  If both url arguments and LoadOptions are present, the LoadOptions overrides the tailing arguments, but default values in LoadOptions are ignored.

  Parameters:

  url - The url to normalize

  options - The LoadOptions applied to the url

  toItemOption - If the LoadOptions is converted to item load options

  Returns:

  The normalized url or NIL if the input url is invalid

• normalize

   List<NormURL> normalize(Collection<UrlAware> urls, LoadOptions options, Boolean toItemOption)

  Normalize urls, remove invalid ones

  Parameters:

  urls - The urls to normalize

  options - The LoadOptions applied to each url

  toItemOption - If the LoadOptions is converted to item load options

  Returns:

  All normalized urls, all invalid input urls are removed

• normalizeOrNull

   NormURL normalizeOrNull(String url, LoadOptions options, Boolean toItemOption)

  Normalize an url, the url can be in one of the following forms:

  • a normal url

  • a configured url

  • a base64 encoded url

  • a base64 encoded configured url

  An url can be configured by appending arguments to the url, and it also can be used with a LoadOptions, If both tailing arguments and LoadOptions are present, the LoadOptions overrides the tailing arguments, but default values in LoadOptions are ignored.

  Parameters:

  url - The url to normalize

  options - The LoadOptions applied to the url

  toItemOption - If the LoadOptions is converted to item load options

  Returns:

  The normalized url or null if the input url is invalid

• normalizeOrNull

   NormURL normalizeOrNull(UrlAware url, LoadOptions options, Boolean toItemOption)

  Normalize an url, the url can be in one of the following forms:

  • a normal url

  • a configured url

  • a base64 encoded url

  • a base64 encoded configured url

  An url can be configured by appending arguments to the url, and it also can be used with a LoadOptions, If both tailing arguments and LoadOptions are present, the LoadOptions overrides the tailing arguments, but default values in LoadOptions are ignored.

  Parameters:

  url - The url to normalize

  options - The LoadOptions applied to the url

  toItemOption - If the LoadOptions is converted to item load options

  Returns:

  The normalized url or null if the input url is invalid

• get

   WebPage get(String url)

  Get a webpage from the storage.

  Parameters:

  url - The url of the page to retrieve

• get

   WebPage get(String url, String fields)

  Get a page from storage.

  Parameters:

  url - The url

  fields - The fields to load from local storage

  Returns:

  The webpage in storage if exists, otherwise returns a NIL page

• getOrNull

   WebPage getOrNull(String url)

  Get a webpage from the storage.

  Parameters:

  url - The url of the page to retrieve

• getOrNull

   WebPage getOrNull(String url, String fields)

  Get a webpage from the storage.

  Parameters:

  url - The url

  fields - The fields to load from local storage

• getContent

   ByteBuffer getContent(String url)

  Get the content of the page from the storage

  Parameters:

  url - The url of the page to retrieve

  Returns:

  The page content or null

• getContentAsString

   String getContentAsString(String url)

  Get the content of the page from the storage

  Parameters:

  url - The url of the page to retrieve

  Returns:

  The page content in string format or null

• exists

   Boolean exists(String url)

  Check if a page exists in the storage.

  Parameters:

  url - The url of the page to check

• fetchState

   CheckState fetchState(WebPage page, LoadOptions options)

  Check the fetch state of a page.

  Parameters:

  page - The webpage to check

• scan

   Iterator<WebPage> scan(String urlPrefix)

  Scan pages in the storage.

  Parameters:

  urlPrefix - The url prefix

• scan

   Iterator<WebPage> scan(String urlPrefix, Iterable<GWebPage.Field> fields)

  Scan pages in the storage.

• scan

   Iterator<WebPage> scan(String urlPrefix, Array<String> fields)

  Scan pages in the storage.

• open

   WebPage open(String url, WebDriver driver, LoadOptions options)

  Open a web page with a web driver.

  Parameters:

  url - The url to open

  driver - The web driver

  options - The options

• load

   WebPage load(String url, LoadOptions options)

  Load a page with specified options, see LoadOptions for all options.

  Parameters:

  url - The url which can be followed by arguments.

  options - The load options.

  Returns:

  The WebPage. If there is no web page at local storage nor remote location, GoraWebPage.NIL is returned.

• load

   WebPage load(URL url, LoadOptions options)

  Load a url with specified options, see LoadOptions for all options.

  Parameters:

  url - The url which can be followed by arguments.

  options - The load options.

  Returns:

  The WebPage. If there is no web page at local storage nor remote location, GoraWebPage.NIL is returned.

• load

   WebPage load(NormURL url)

  Load a url, options can be specified following the url, see LoadOptions for all options.

  Parameters:

  url - The url which can be followed by arguments.

  Returns:

  The WebPage. If there is no web page at local storage nor remote location, GoraWebPage.NIL is returned.

• loadDeferred

   WebPage loadDeferred(NormURL url)

  Load a url, options can be specified following the url, see LoadOptions for all options

  Parameters:

  url - The url followed by options

  Returns:

  The WebPage. If there is no web page at local storage nor remote location, WebPageImpl.NIL is returned

• loadAll

   List<WebPage> loadAll(Iterable<String> urls, LoadOptions options)

  Load a batch of urls with the specified options.

  If the option indicates prefer parallel, urls are fetched in a parallel manner whenever applicable. If the batch is too large, only a random part of the urls is fetched immediately, all the rest urls are put into a pending fetch list and will be fetched in background later.

  If a page exists neither in local storage nor at the given remote location, GoraWebPage.NIL is returned

  Parameters:

  urls - The urls to load

  options - The load options

  Returns:

  Pages for all urls.

• loadAll

   List<WebPage> loadAll(Iterable<NormURL> urls)

  Load a batch of urls with the specified options.

  If the option indicates prefer parallel, urls are fetched in a parallel manner whenever applicable. If the batch is too large, only a random part of the urls is fetched immediately, all the rest urls are put into a pending fetch list and will be fetched in background later.

  If a page exists neither in local storage nor at the given remote location, WebPageImpl.NIL is returned

  Parameters:

  urls - The urls to load

  Returns:

  Pages for all urls.

• loadAsync

   CompletableFuture<WebPage> loadAsync(NormURL url)

  Load a url asynchronously, the url is added to the task queue, and will be executed asynchronously

• loadAllAsync

   List<CompletableFuture<WebPage>> loadAllAsync(Iterable<NormURL> urls)

  Load a batch of urls asynchronously, the urls are added to the task queue, and will be executed asynchronously

• submit

   AbstractPulsarContext submit(UrlAware url)

  Submit a url, the url will be added to the task queue, and will be executed asynchronously.

  The url should be standard or degenerate, otherwise it will be discarded.

  Parameters:

  url - The url to submit, which will be added to the task queue

  Returns:

  The PulsarContext itself to enabled chained operations

• submitAll

   AbstractPulsarContext submitAll(Iterable<UrlAware> urls)

  Submit a batch of urls, the urls are added to the task queue, and will be executed asynchronously.

  The urls should be standard or degenerate, otherwise they will be discarded.

  Parameters:

  urls - The urls to submit, which will be added to the task queue

  Returns:

  The PulsarContext itself to enabled chained operations

• parse

   FeaturedDocument parse(WebPage page)

  Parse the WebPage content using parseComponent.

• chat

   ModelResponse chat(String prompt)

  Chat with the AI model.

  Parameters:

  prompt - The prompt to chat

• chat

   ModelResponse chat(String userMessage, String systemMessage)

  Chat with the AI model.

  Parameters:

  userMessage - The user message to chat

  systemMessage - The system message to chat

• persist

   Unit persist(WebPage page)

  Persist the webpage into the storage immediately. By default, the backend storage is the local file system, if mongodb is detected, the mongodb will be used as the backend storage.

• delete

   Unit delete(String url)

  Delete the page from the storage.

• delete

   Unit delete(WebPage page)

  Delete the page from the storage.

• flush

   Unit flush()

  Flush the storage.

• await

   Unit await()

  Wait until there is no tasks in the main loop.

• registerShutdownHook

   Unit registerShutdownHook()

  Register a shutdown hook with the JVM runtime, closing this context on JVM shutdown unless it has already been closed at that time.

  Delegates to doClose() for the actual closing procedure.

• close

   Unit close()

  Close this pulsar context.

  Delegates to doClose() for the actual closing procedure. Also removes a JVM shutdown hook, if registered, as it's not needed anymore.