All Implemented Interfaces:

java.lang.AutoCloseable
```
public interface PulsarContext
 implements AutoCloseable
                    
```
PulsarContext consists of a set of highly customizable components that provide the core set of interfaces of the system and is used to create PulsarSession.

Nested Class Summary

Nested Classes
Modifier and Type	Class	Description

Field Summary

Fields
Modifier and Type	Field	Description

Constructor Summary

Constructors
Constructor	Description

Enum Constant Summary

Enum Constants
Enum Constant	Description

Method Summary

Modifier and Type	Method	Description
`abstract <T extends Any> T`	`getBean(KClass<T> requiredType)`	Get a bean with the specified class, throws BeansException if the bean doesn't exist
`abstract <T extends Any> T`	`getBeanOrNull(KClass<T> requiredType)`	Get a bean with the specified class, returns null if the bean doesn't exist
`abstract PulsarSession`	`createSession()`	Create a pulsar session
`abstract Unit`	`closeSession(PulsarSession session)`	Close a pulsar session
`abstract Unit`	`registerClosable(AutoCloseable closable)`	Register close objects, the objects will be closed when the context closes
`abstract NormUrl`	`normalize(String url, LoadOptions options, Boolean toItemOption)`	Normalize an url, the url can be in one of the following forms: a normal url a configured url a base64 encoded url a base64 encoded configured url An url can be configured by appending arguments to the url, and it also can be used with a LoadOptions, If both tailing arguments and LoadOptions are present, the LoadOptions overrides the tailing arguments, but default values in LoadOptions are ignored.
`abstract List<NormUrl>`	`normalize(Iterable<String> urls, LoadOptions options, Boolean toItemOption)`	Normalize urls, remove invalid ones
`abstract NormUrl`	`normalize(UrlAware url, LoadOptions options, Boolean toItemOption)`	Normalize an url.
`abstract List<NormUrl>`	`normalize(Collection<UrlAware> urls, LoadOptions options, Boolean toItemOption)`	Normalize urls, remove invalid ones
`abstract NormUrl`	`normalizeOrNull(String url, LoadOptions options, Boolean toItemOption)`
`abstract NormUrl`	`normalizeOrNull(UrlAware url, LoadOptions options, Boolean toItemOption)`
`abstract WebPage`	`inject(String url)`	Inject an url
`abstract WebPage`	`inject(NormUrl url)`	Inject an url
`abstract WebPage`	`get(String url)`	Get a webpage from the storage
`abstract WebPage`	`get(String url, String fields)`	Get a page from storage.
`abstract WebPage`	`getOrNull(String url)`	Get a webpage from the storage
`abstract WebPage`	`getOrNull(String url, String fields)`	Get a page from storage.
`abstract ByteBuffer`	`getContent(String url)`	Get the content of the page from the storage
`abstract String`	`getContentAsString(String url)`	Get the content of the page from the storage
`abstract Boolean`	`exists(String url)`	Get a webpage from the storage
`abstract CheckState`	`fetchState(WebPage page, LoadOptions options)`	Check the fetch state of a webpage
`abstract Iterator<WebPage>`	`scan(String urlPrefix)`	Scan webpages in the storage whose url start with urlPrefix
`abstract Iterator<WebPage>`	`scan(String urlPrefix, Iterable<GWebPage.Field> fields)`
`abstract Iterator<WebPage>`	`scan(String urlPrefix, Array<String> fields)`
`abstract WebPage`	`load(String url, LoadOptions options)`	Load a url with specified options, see LoadOptions for all options
`abstract WebPage`	`load(URL url, LoadOptions options)`	Load a url with specified options, see LoadOptions for all options
`abstract WebPage`	`load(NormUrl url)`	Load a url, options can be specified following the url, see LoadOptions for all options
`abstract WebPage`	`loadDeferred(NormUrl url)`	Load a url, options can be specified following the url, see LoadOptions for all options
`abstract Collection<WebPage>`	`loadAll(Iterable<String> urls, LoadOptions options)`	Load a batch of urls with the specified options.
`abstract Collection<WebPage>`	`loadAll(Iterable<NormUrl> urls)`	Load a batch of urls with the specified options.
`abstract CompletableFuture<WebPage>`	`loadAsync(NormUrl url)`	Load a url asynchronously, the url is added to the task queue, and will be executed asynchronously
`abstract List<CompletableFuture<WebPage>>`	`loadAllAsync(Iterable<NormUrl> urls)`	Load a batch of urls asynchronously, the urls are added to the task queue, and will be executed asynchronously
`abstract PulsarContext`	`submit(UrlAware url)`	Submit a url, the url will be added to the task queue, and will be executed asynchronously.
`abstract PulsarContext`	`submitAll(Iterable<UrlAware> urls)`	Submit a batch of urls, the urls are added to the task queue, and will be executed asynchronously.
`abstract FeaturedDocument`	`parse(WebPage page)`	Parse the WebPage using ParseComponent
`abstract Unit`	`persist(WebPage page)`	Persist the webpage into the storage immediately.
`abstract Unit`	`delete(String url)`	Delete the webpage from the backend storage
`abstract Unit`	`delete(WebPage page)`	Delete the webpage from the backend storage
`abstract Unit`	`flush()`	Flush the backend storage
`abstract Unit`	`await()`	Wait until all tasks are done.
`abstract Unit`	`registerShutdownHook()`	Register shutdown hook
`abstract Integer`	`getId()`	The context id
`abstract Boolean`	`isActive()`	The context id
`abstract ImmutableConfig`	`getUnmodifiedConfig()`	An immutable config loaded from the config file at startup, and never changes
`abstract GlobalCache`	`getGlobalCache()`	The global cache
`abstract UrlPool`	`getCrawlPool()`	The url pool to fetch
`abstract CrawlLoops`	`getCrawlLoops()`	The main loops

Methods inherited from class java.lang.AutoCloseable
close
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Constructor Detail
- Method Detail
  - getBean
```
 abstract <T extends Any> T getBean(KClass<T> requiredType)
```
    Get a bean with the specified class, throws BeansException if the bean doesn't exist
  - getBeanOrNull
```
 abstract <T extends Any> T getBeanOrNull(KClass<T> requiredType)
```
    Get a bean with the specified class, returns null if the bean doesn't exist
  - createSession
```
 abstract PulsarSession createSession()
```
    Create a pulsar session
  - closeSession
```
 abstract Unit closeSession(PulsarSession session)
```
    Close a pulsar session
  - registerClosable
```
 abstract Unit registerClosable(AutoCloseable closable)
```
    Register close objects, the objects will be closed when the context closes
  - normalize
```
 abstract NormUrl normalize(String url, LoadOptions options, Boolean toItemOption)
```
    Normalize an url, the url can be in one of the following forms:
    a normal url
    a configured url
    a base64 encoded url
    a base64 encoded configured url
    An url can be configured by appending arguments to the url, and it also can be used with a LoadOptions, If both tailing arguments and LoadOptions are present, the LoadOptions overrides the tailing arguments, but default values in LoadOptions are ignored.
  - normalize
```
 abstract List<NormUrl> normalize(Iterable<String> urls, LoadOptions options, Boolean toItemOption)
```
    Normalize urls, remove invalid ones
    
    Parameters:
    
    urls - The urls to normalize
    
    options - The LoadOptions applied to each url
    
    toItemOption - If the LoadOptions is converted to item load options
  - normalize
```
 abstract NormUrl normalize(UrlAware url, LoadOptions options, Boolean toItemOption)
```
    Normalize an url.
    If both url arguments and LoadOptions are present, the LoadOptions overrides the tailing arguments, but default values in LoadOptions are ignored.
  - normalize
```
 abstract List<NormUrl> normalize(Collection<UrlAware> urls, LoadOptions options, Boolean toItemOption)
```
    Normalize urls, remove invalid ones
    
    Parameters:
    
    urls - The urls to normalize
    
    options - The LoadOptions applied to each url
    
    toItemOption - If the LoadOptions is converted to item load options
  - normalizeOrNull
```
 abstract NormUrl normalizeOrNull(String url, LoadOptions options, Boolean toItemOption)
```
  - normalizeOrNull
```
 abstract NormUrl normalizeOrNull(UrlAware url, LoadOptions options, Boolean toItemOption)
```
  - inject
```
 abstract WebPage inject(String url)
```
    Inject an url
    
    Parameters:
    
    url - The url followed by config options
  - inject
```
 abstract WebPage inject(NormUrl url)
```
    Inject an url
    
    Parameters:
    
    url - The url followed by config options
  - get
```
 abstract WebPage get(String url)
```
    Get a webpage from the storage
    
    Parameters:
    
    url - The url of the page to retrieve
  - get
```
 abstract WebPage get(String url, String fields)
```
    Get a page from storage.
    
    Parameters:
    
    url - The url
    
    fields - The fields to load from local storage
  - getOrNull
```
 abstract WebPage getOrNull(String url)
```
    Get a webpage from the storage
    
    Parameters:
    
    url - The url of the page to retrieve
  - getOrNull
```
 abstract WebPage getOrNull(String url, String fields)
```
    Get a page from storage.
    
    Parameters:
    
    url - The url
    
    fields - The fields to load from local storage
  - getContent
```
 abstract ByteBuffer getContent(String url)
```
    Get the content of the page from the storage
    
    Parameters:
    
    url - The url of the page to retrieve
  - getContentAsString
```
@Beta() abstract String getContentAsString(String url)
```
    Get the content of the page from the storage
    
    Parameters:
    
    url - The url of the page to retrieve
  - exists
```
 abstract Boolean exists(String url)
```
    Get a webpage from the storage
    
    Parameters:
    
    url - The url of the page to check
  - fetchState
```
 abstract CheckState fetchState(WebPage page, LoadOptions options)
```
    Check the fetch state of a webpage
    
    Parameters:
    
    page - The webpage to check
  - scan
```
 abstract Iterator<WebPage> scan(String urlPrefix)
```
    Scan webpages in the storage whose url start with urlPrefix
    
    Parameters:
    
    urlPrefix - The url prefix
  - scan
```
 abstract Iterator<WebPage> scan(String urlPrefix, Iterable<GWebPage.Field> fields)
```
  - scan
```
 abstract Iterator<WebPage> scan(String urlPrefix, Array<String> fields)
```
  - load
```
 abstract WebPage load(String url, LoadOptions options)
```
    Load a url with specified options, see LoadOptions for all options
    
    Parameters:
    
    url - The url followed by options
    
    options - The options
  - load
```
 abstract WebPage load(URL url, LoadOptions options)
```
    Load a url with specified options, see LoadOptions for all options
    
    Parameters:
    
    url - The url followed by options
    
    options - The options
  - load
```
 abstract WebPage load(NormUrl url)
```
    Load a url, options can be specified following the url, see LoadOptions for all options
    
    Parameters:
    
    url - The url followed by options
  - loadDeferred
```
 abstract WebPage loadDeferred(NormUrl url)
```
    Load a url, options can be specified following the url, see LoadOptions for all options
    
    Parameters:
    
    url - The url followed by options
  - loadAll
```
 abstract Collection<WebPage> loadAll(Iterable<String> urls, LoadOptions options)
```
    Load a batch of urls with the specified options.
    If the option indicates prefer parallel, urls are fetched in a parallel manner whenever applicable. If the batch is too large, only a random part of the urls is fetched immediately, all the rest urls are put into a pending fetch list and will be fetched in background later.
    If a page exists neither in local storage nor at the given remote location, WebPage.NIL is returned
    
    Parameters:
    
    urls - The urls to load
    
    options - The options
  - loadAll
```
 abstract Collection<WebPage> loadAll(Iterable<NormUrl> urls)
```
    Load a batch of urls with the specified options.
    If the option indicates prefer parallel, urls are fetched in a parallel manner whenever applicable. If the batch is too large, only a random part of the urls is fetched immediately, all the rest urls are put into a pending fetch list and will be fetched in background later.
    If a page exists neither in local storage nor at the given remote location, WebPage.NIL is returned
    
    Parameters:
    
    urls - The urls to load
  - loadAsync
```
 abstract CompletableFuture<WebPage> loadAsync(NormUrl url)
```
    Load a url asynchronously, the url is added to the task queue, and will be executed asynchronously
  - loadAllAsync
```
 abstract List<CompletableFuture<WebPage>> loadAllAsync(Iterable<NormUrl> urls)
```
    Load a batch of urls asynchronously, the urls are added to the task queue, and will be executed asynchronously
  - submit
```
 abstract PulsarContext submit(UrlAware url)
```
    Submit a url, the url will be added to the task queue, and will be executed asynchronously.
    The url should be standard or degenerate, otherwise it will be discarded.
    
    Parameters:
    
    url - The url to submit, which will be added to the task queue
  - submitAll
```
 abstract PulsarContext submitAll(Iterable<UrlAware> urls)
```
    Submit a batch of urls, the urls are added to the task queue, and will be executed asynchronously.
    The urls should be standard or degenerate, otherwise they will be discarded.
    
    Parameters:
    
    urls - The urls to submit, which will be added to the task queue
  - parse
```
 abstract FeaturedDocument parse(WebPage page)
```
    Parse the WebPage using ParseComponent
  - persist
```
 abstract Unit persist(WebPage page)
```
    Persist the webpage into the storage immediately. By default, the backend storage is the local file system, if mongodb is detected, the mongodb will be used as the backend storage.
  - delete
```
 abstract Unit delete(String url)
```
    Delete the webpage from the backend storage
  - delete
```
 abstract Unit delete(WebPage page)
```
    Delete the webpage from the backend storage
  - flush
```
 abstract Unit flush()
```
    Flush the backend storage
  - await
```
 abstract Unit await()
```
    Wait until all tasks are done.
  - registerShutdownHook
```
 abstract Unit registerShutdownHook()
```
    Register shutdown hook
  - getId
```
 abstract Integer getId()
```
    The context id
  - isActive
```
 abstract Boolean isActive()
```
    The context id
  - getUnmodifiedConfig
```
 abstract ImmutableConfig getUnmodifiedConfig()
```
    An immutable config loaded from the config file at startup, and never changes
  - getGlobalCache
```
 abstract GlobalCache getGlobalCache()
```
    The global cache
  - getCrawlPool
```
 abstract UrlPool getCrawlPool()
```
    The url pool to fetch
  - getCrawlLoops
```
 abstract CrawlLoops getCrawlLoops()
```
    The main loops

Interface PulsarContext

Nested Class Summary

Field Summary

Constructor Summary

Enum Constant Summary

Method Summary

Methods inherited from class java.lang.AutoCloseable

Methods inherited from class java.lang.Object

Constructor Detail

Method Detail

getBean

getBeanOrNull

createSession

closeSession

registerClosable

normalize

normalize

normalize

normalize

normalizeOrNull

normalizeOrNull

inject

inject

get

get

getOrNull

getOrNull

getContent

getContentAsString

exists

fetchState

scan

scan

scan

load

load

load

loadDeferred

loadAll

loadAll

loadAsync

loadAllAsync

submit

submitAll

parse

persist

delete

delete

flush

await

registerShutdownHook

getId

isActive

getUnmodifiedConfig

getGlobalCache

getCrawlPool

getCrawlLoops