Constructor Detail

• AbstractPulsarContext

  AbstractPulsarContext(AbstractApplicationContext applicationContext, PulsarEnvironment pulsarEnvironment)

Method Detail

• isActive

   Boolean isActive()

  The context id

• getId

   Integer getId()

  The context id

• getUnmodifiedConfig

   ImmutableConfig getUnmodifiedConfig()

  An immutable config loaded from the config file at startup, and never changes

• getUrlNormalizers

   ChainedUrlNormalizer getUrlNormalizers()

• getUrlNormalizer

   ChainedUrlNormalizer getUrlNormalizer()

  The url normalizer

• getUrlNormalizerOrNull

   ChainedUrlNormalizer getUrlNormalizerOrNull()

• getWebDb

   WebDb getWebDb()

• getGlobalCacheFactory

   GlobalCacheFactory getGlobalCacheFactory()

• getInjectComponent

   InjectComponent getInjectComponent()

• getFetchComponent

   BatchFetchComponent getFetchComponent()

• getParseComponent

   ParseComponent getParseComponent()

• getUpdateComponent

   UpdateComponent getUpdateComponent()

• getLoadComponent

   LoadComponent getLoadComponent()

• getGlobalCache

   GlobalCache getGlobalCache()

  The global cache

• getCrawlPool

   UrlPool getCrawlPool()

  The url pool to fetch

• getCrawlLoops

   CrawlLoops getCrawlLoops()

  The main loops

• getStartTime

   final Long getStartTime()

  The start time

• getSessions

   final ConcurrentSkipListMap<Integer, PulsarSession> getSessions()

  All open sessions

• getApplicationContext

   final AbstractApplicationContext getApplicationContext()

• getPulsarEnvironment

   final PulsarEnvironment getPulsarEnvironment()

• getBean

   <T extends Any> T getBean(KClass<T> requiredType)

  Get a bean with the specified class, throws BeansException if the bean doesn't exist

• getBean

   final <T extends Any> T getBean()

  Get a bean with the specified class, throws BeansException if the bean doesn't exist

• getBeanOrNull

   <T extends Any> T getBeanOrNull(KClass<T> requiredType)

  Get a bean with the specified class, returns null if the bean doesn't exist

• getBeanOrNull

   final <T extends Any> T getBeanOrNull()

  Get a bean with the specified class, returns null if the bean doesn't exist

• createSession

   abstract AbstractPulsarSession createSession()

  Create a session

• closeSession

   Unit closeSession(PulsarSession session)

  Close the given session

• registerClosable

   Unit registerClosable(AutoCloseable closable)

  Register close objects, the objects will be closed when the context closes

• normalize

   NormUrl normalize(String url, LoadOptions options, Boolean toItemOption)

  Normalize an url, the url can be one of the following:

  • a normal url

  • a configured url

  • a base64 encoded url

  • a base64 encoded configured url

  An url can be configured by appending arguments to the url, and it also can be used with load options, If both tailing arguments and load options are present, the tailing arguments override the load options.

• normalize

   List<NormUrl> normalize(Iterable<String> urls, LoadOptions options, Boolean toItemOption)

  Normalize urls, remove invalid ones

  Parameters:

  urls - The urls to normalize

  options - The load options applied to each url

  toItemOption - If true, options will be converted to item load options

  Returns:

  All normalized urls, all invalid input urls are removed

• normalize

   NormUrl normalize(UrlAware url, LoadOptions options, Boolean toItemOption)

  Normalize an url.

  If both tailing arguments and load options are present, the tailing arguments override the load options.

• normalize

   List<NormUrl> normalize(Collection<UrlAware> urls, LoadOptions options, Boolean toItemOption)

  Normalize urls, remove invalid ones

  Parameters:

  urls - The urls to normalize

  options - The LoadOptions applied to each url

  toItemOption - If the LoadOptions is converted to item load options

  Returns:

  All normalized urls, all invalid input urls are removed

• normalizeOrNull

   NormUrl normalizeOrNull(String url, LoadOptions options, Boolean toItemOption)

• normalizeOrNull

   NormUrl normalizeOrNull(UrlAware url, LoadOptions options, Boolean toItemOption)

• inject

   WebPage inject(String url)

  Inject an url

  Parameters:

  url - The url which can be followed by arguments

  Returns:

  The web page created

• inject

   WebPage inject(NormUrl url)

  Inject an url

  Parameters:

  url - The url which can be followed by arguments

  Returns:

  The web page created

• get

   WebPage get(String url)

  Get a webpage from the storage

  Parameters:

  url - The url of the page to retrieve

• get

   WebPage get(String url, String fields)

  Get a page from storage.

  Parameters:

  url - The url

  fields - The fields to load from local storage

  Returns:

  The webpage in storage if exists, otherwise returns a NIL page

• getOrNull

   WebPage getOrNull(String url)

  Get a webpage from the storage

  Parameters:

  url - The url of the page to retrieve

• getOrNull

   WebPage getOrNull(String url, String fields)

  Get a webpage from the storage

  Parameters:

  url - The url

  fields - The fields to load from local storage

• getContent

   ByteBuffer getContent(String url)

  Get the content of the page from the storage

  Parameters:

  url - The url of the page to retrieve

  Returns:

  The page content or null

• getContentAsString

   String getContentAsString(String url)

  Get the content of the page from the storage

  Parameters:

  url - The url of the page to retrieve

  Returns:

  The page content in string format or null

• exists

   Boolean exists(String url)

  Check if a page exists in the storage

  Parameters:

  url - The url of the page to check

• fetchState

   CheckState fetchState(WebPage page, LoadOptions options)

  Check the fetch state of a page

  Parameters:

  page - The webpage to check

• scan

   Iterator<WebPage> scan(String urlPrefix)

  Scan pages in the storage

  Parameters:

  urlPrefix - The url prefix

• scan

   Iterator<WebPage> scan(String urlPrefix, Iterable<GWebPage.Field> fields)

  Scan pages in the storage

• scan

   Iterator<WebPage> scan(String urlPrefix, Array<String> fields)

  Scan pages in the storage

• load

   WebPage load(String url, LoadOptions options)

  Load an page with specified options, see LoadOptions for all options

  Parameters:

  url - The url which can be followed by arguments

  options - The load options

  Returns:

  The WebPage. If there is no web page at local storage nor remote location, WebPage.NIL is returned

• load

   WebPage load(URL url, LoadOptions options)

  Load a url with specified options, see LoadOptions for all options

  Parameters:

  url - The url which can be followed by arguments

  options - The load options

  Returns:

  The WebPage. If there is no web page at local storage nor remote location, WebPage.NIL is returned

• load

   WebPage load(NormUrl url)

  Load a url, options can be specified following the url, see LoadOptions for all options

  Parameters:

  url - The url which can be followed by arguments

  Returns:

  The WebPage. If there is no web page at local storage nor remote location, WebPage.NIL is returned

• loadDeferred

   WebPage loadDeferred(NormUrl url)

  Load a url, options can be specified following the url, see LoadOptions for all options

  Parameters:

  url - The url followed by options

  Returns:

  The WebPage. If there is no web page at local storage nor remote location, WebPage.NIL is returned

• loadAll

   List<WebPage> loadAll(Iterable<String> urls, LoadOptions options)

  Load a batch of urls with the specified options.

  If the option indicates prefer parallel, urls are fetched in a parallel manner whenever applicable. If the batch is too large, only a random part of the urls is fetched immediately, all the rest urls are put into a pending fetch list and will be fetched in background later.

  If a page exists neither in local storage nor at the given remote location, WebPage.NIL is returned

  Parameters:

  urls - The urls to load

  options - The load options

  Returns:

  Pages for all urls.

• loadAll

   List<WebPage> loadAll(Iterable<NormUrl> urls)

  Load a batch of urls with the specified options.

  If the option indicates prefer parallel, urls are fetched in a parallel manner whenever applicable. If the batch is too large, only a random part of the urls is fetched immediately, all the rest urls are put into a pending fetch list and will be fetched in background later.

  If a page exists neither in local storage nor at the given remote location, WebPage.NIL is returned

  Parameters:

  urls - The urls to load

  Returns:

  Pages for all urls.

• loadAsync

   CompletableFuture<WebPage> loadAsync(NormUrl url)

  Load a url asynchronously, the url is added to the task queue, and will be executed asynchronously

• loadAllAsync

   List<CompletableFuture<WebPage>> loadAllAsync(Iterable<NormUrl> urls)

  Load a batch of urls asynchronously, the urls are added to the task queue, and will be executed asynchronously

• submit

   AbstractPulsarContext submit(UrlAware url)

  Submit a url, the url will be added to the task queue, and will be executed asynchronously.

  The url should be standard or degenerate, otherwise it will be discarded.

  Parameters:

  url - The url to submit, which will be added to the task queue

  Returns:

  The PulsarContext itself to enabled chained operations

• submitAll

   AbstractPulsarContext submitAll(Iterable<UrlAware> urls)

  Submit a batch of urls, the urls are added to the task queue, and will be executed asynchronously.

  The urls should be standard or degenerate, otherwise they will be discarded.

  Parameters:

  urls - The urls to submit, which will be added to the task queue

  Returns:

  The PulsarContext itself to enabled chained operations

• parse

   FeaturedDocument parse(WebPage page)

  Parse the WebPage content using parseComponent

• persist

   Unit persist(WebPage page)

  Persist the page into the storage

• delete

   Unit delete(String url)

  Delete the page from the storage

• delete

   Unit delete(WebPage page)

  Delete the page from the storage

• flush

   Unit flush()

  Flush the storage

• await

   Unit await()

  Wait until there is no tasks in the main loop

• registerShutdownHook

   Unit registerShutdownHook()

  Register a shutdown hook with the JVM runtime, closing this context on JVM shutdown unless it has already been closed at that time.

  Delegates to doClose() for the actual closing procedure.

• close

   Unit close()

  Close this pulsar context

  Delegates to doClose() for the actual closing procedure. Also removes a JVM shutdown hook, if registered, as it's not needed anymore.