Constructor Detail

• AbstractPulsarSession

  AbstractPulsarSession(AbstractPulsarContext context, VolatileConfig sessionConfig, Integer id)

Method Detail

• getUnmodifiedConfig

   ImmutableConfig getUnmodifiedConfig()

  An immutable config which is loaded from the config file at process startup, and never changes

• getSessionBeanFactory

   BeanFactory getSessionBeanFactory()

  The scoped bean factory: for each volatileConfig object, there is a bean factory

• getDisplay

   String getDisplay()

  A short descriptive display text.

• isActive

   final Boolean isActive()

• getGlobalCacheFactory

   GlobalCacheFactory getGlobalCacheFactory()

• getGlobalCache

   GlobalCache getGlobalCache()

  The global cache

• getPageCache

   ConcurrentExpiringLRUCache<String, WebPage> getPageCache()

  The global page cache

• getDocumentCache

   ConcurrentExpiringLRUCache<String, FeaturedDocument> getDocumentCache()

  The global document cache

• getContext

   AbstractPulsarContext getContext()

  The pulsar context

• getSessionConfig

   VolatileConfig getSessionConfig()

  The session scope volatile config, every setting is supposed to be changed at any time and any place

• getId

   Integer getId()

  The session id. Session id is expected to be set by the container, e.g. the h2 database runtime

• registerClosable

   Boolean registerClosable(AutoCloseable closable)

  Close objects when the session closes

• disablePDCache

   Unit disablePDCache()

  Disable page cache and document cache

• options

   LoadOptions options(String args, PageEvent event)

  Create a new LoadOptions object with args, event, and sessionConfig.

• property

   String property(String name)

  Get a property.

• property

   Unit property(String name, String value)

  Set a session scope property.

• normalize

   NormUrl normalize(String url)

  Normalize a url.

• normalize

   NormUrl normalize(String url, String args, Boolean toItemOption)

  Normalize a url.

• normalize

   NormUrl normalize(String url, LoadOptions options, Boolean toItemOption)

  Normalize a url.

• normalize

   List<NormUrl> normalize(Iterable<String> urls)

  Normalize urls.

• normalize

   List<NormUrl> normalize(Iterable<String> urls, String args, Boolean toItemOption)

  Normalize urls.

• normalize

   List<NormUrl> normalize(Iterable<String> urls, LoadOptions options, Boolean toItemOption)

  Normalize urls.

• normalize

   NormUrl normalize(UrlAware url)

  Normalize a url.

• normalize

   NormUrl normalize(UrlAware url, String args, Boolean toItemOption)

  Normalize a url.

• normalize

   NormUrl normalize(UrlAware url, LoadOptions options, Boolean toItemOption)

  Normalize a url.

• normalize

   List<NormUrl> normalize(Collection<UrlAware> urls)

  Normalize urls.

• normalize

   List<NormUrl> normalize(Collection<UrlAware> urls, String args, Boolean toItemOption)

  Normalize urls, remove invalid ones

  Parameters:

  urls - The urls to normalize

  args - The arguments

  toItemOption - If the LoadOptions is converted to item load options

  Returns:

  All normalized urls, all invalid input urls are removed

• normalize

   List<NormUrl> normalize(Collection<UrlAware> urls, LoadOptions options, Boolean toItemOption)

  Normalize urls, remove invalid ones

  Parameters:

  urls - The urls to normalize

  options - The LoadOptions applied to each url

  toItemOption - If the LoadOptions is converted to item load options

  Returns:

  All normalized urls, all invalid input urls are removed

• normalizeOrNull

   NormUrl normalizeOrNull(String url, LoadOptions options, Boolean toItemOption)

  Normalize a url.

• normalizeOrNull

   NormUrl normalizeOrNull(UrlAware url, LoadOptions options, Boolean toItemOption)

  Normalize a url.

• inject

   WebPage inject(String url)

  Inject a url as a seed to fetch. Injection is usually used in Nutch style crawls where the execution flow likes the following:

  inject -> generate -> fetch -> parse -> index -> update ^ ^ | <- <- <- <- |

  Parameters:

  url - The url to inject, con be followed by arguments

  Returns:

  A newly created webpage record which is ready to be generated

• get

   WebPage get(String url)

  Get a page from storage.

  Parameters:

  url - The url

  Returns:

  The webpage in storage if exists, otherwise returns a NIL page

• get

   WebPage get(String url, String fields)

  Get a page from storage.

  Parameters:

  url - The url

  fields - The fields to load from local storage

  Returns:

  The webpage in storage if exists, otherwise returns a NIL page

• getOrNull

   WebPage getOrNull(String url)

  Get a page from storage.

  Parameters:

  url - The url

  Returns:

  The page in storage if exists, otherwise returns null

• getOrNull

   WebPage getOrNull(String url, String fields)

  Get a page from storage.

  Parameters:

  url - The url

  fields - The fields to load from local storage

  Returns:

  The page in storage if exists, otherwise returns null

• getContent

   ByteBuffer getContent(String url)

  Get the content of the page from the storage

  Parameters:

  url - The url of the page to retrieve

  Returns:

  The page content or null

• getContentAsString

   String getContentAsString(String url)

  Get the content of the page from the storage

  Parameters:

  url - The url of the page to retrieve

  Returns:

  The page content in string format or null

• exists

   Boolean exists(String url)

  Check if the page exists in the storage.

  Parameters:

  url - The url to check

  Returns:

  true if the page exists, false otherwise

• fetchState

   CheckState fetchState(WebPage page, LoadOptions options)

  Return the fetch state of the page.

  Parameters:

  page - The webpage

  options - The load options

  Returns:

  The fetch state of the page

• open

   WebPage open(String url)

  Open a url.

  This method opens the url immediately, regardless of the previous state of the page.

  Parameters:

  url - The url to open

  Returns:

  The webpage loaded or NIL

• load

   WebPage load(String url)

  Load a url with specified arguments.

  This method first checks the url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

  Other fetch conditions can be specified by load arguments:

  • expiration

  • page size requirement

  • fields requirement

  • other

  Parameters:

  url - The url to load

  Returns:

  The webpage loaded or NIL

• load

   WebPage load(String url, String args)

  Load a url with specified arguments.

  This method first checks the url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

  Other fetch conditions can be specified by load arguments:

  • expiration

  • page size requirement

  • fields requirement

  • other

  Parameters:

  url - The url to load

  args - The load arguments

  Returns:

  The webpage loaded or NIL

• load

   WebPage load(String url, LoadOptions options)

  Load a url with specified options.

  This method first checks the url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

  Other fetch conditions can be specified by load arguments:

  • expiration

  • page size requirement

  • fields requirement

  • other

  Parameters:

  url - The url to load

  options - The load options

  Returns:

  The webpage loaded or NIL

• load

   WebPage load(UrlAware url)

  Load a url with the specified arguments.

  This method first checks the url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

  Parameters:

  url - The url to load

  Returns:

  The webpage loaded or NIL

• load

   WebPage load(UrlAware url, String args)

  Load a url with the specified arguments.

  This method first checks the url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

  Parameters:

  url - The url to load

  args - The load arguments

  Returns:

  The webpage loaded or NIL

• load

   WebPage load(UrlAware url, LoadOptions options)

  Load a url with options.

  This method first checks the url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

  Parameters:

  url - The url to load

  options - The load options

  Returns:

  The webpage loaded or NIL

• load

   WebPage load(NormUrl normUrl)

  Load a normal url.

  This method first checks the url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

  Parameters:

  normUrl - The normal url

  Returns:

  The webpage loaded or NIL

• loadDeferred

   WebPage loadDeferred(String url, String args)

  Load a url with specified options.

  This method first checks the url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

  This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.

  Parameters:

  url - The url to load

  args - The load options

  Returns:

  The webpage loaded or NIL

• loadDeferred

   WebPage loadDeferred(String url, LoadOptions options)

  Load a url with specified options.

  This method first checks the url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

  This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.

  Parameters:

  url - The url to load

  options - The load options

  Returns:

  The webpage loaded or NIL

• loadDeferred

   WebPage loadDeferred(UrlAware url, String args)

  Load a url with specified arguments.

  This method first checks the url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

  This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.

  Parameters:

  url - The url to load

  args - The load args

  Returns:

  The webpage loaded or NIL

• loadDeferred

   WebPage loadDeferred(UrlAware url, LoadOptions options)

  Load a url with specified options.

  This method first checks the url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

  This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.

  Parameters:

  url - The url to load

  options - The load options

  Returns:

  The webpage loaded or NIL

• loadDeferred

   WebPage loadDeferred(NormUrl url)

  Load a url with specified options

  This method first checks the url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

  This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.

  Parameters:

  url - The normal url

  Returns:

  The webpage loaded or NIL

• loadAll

   List<WebPage> loadAll(Iterable<String> urls)

  Load all urls with specified options

  This method first checks each url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

  Parameters:

  urls - The urls to load

  Returns:

  The webpage loaded or NIL

• loadAll

   List<WebPage> loadAll(Iterable<String> urls, String args)

  Load all urls with specified options

  This method first checks each url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

  Parameters:

  urls - The urls to load

  args - The load arguments

  Returns:

  The webpage loaded or NIL

• loadAll

   List<WebPage> loadAll(Iterable<String> urls, LoadOptions options)

  Load all urls with specified options

  This method first checks each url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

  Parameters:

  urls - The urls to load

  options - The load options

  Returns:

  The webpage loaded or NIL

• loadAll

   List<WebPage> loadAll(Collection<UrlAware> urls)

  Load all urls with specified options

  This method first checks each url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

  Parameters:

  urls - The urls to load

  Returns:

  The webpage loaded or NIL

• loadAll

   List<WebPage> loadAll(Collection<UrlAware> urls, String args)

  Load all urls with specified options

  This method first checks each url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

  Parameters:

  urls - The urls to load

  args - The load arguments

  Returns:

  The webpage loaded or NIL

• loadAll

   List<WebPage> loadAll(Collection<UrlAware> urls, LoadOptions options)

  Load all urls with specified options

  This method first checks each url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

  Parameters:

  urls - The urls to load

  options - The load options

  Returns:

  The webpage loaded or NIL

• loadAll

   List<WebPage> loadAll(List<NormUrl> normUrls)

  Load all normal urls with specified options

  This method first checks each url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

  Parameters:

  normUrls - The normal urls to load

  Returns:

  The loaded webpages

• loadAsync

   CompletableFuture<WebPage> loadAsync(String url)

  Load a normal url in java async style

  Parameters:

  url - The url to load

  Returns:

  A completable future of webpage

• loadAsync

   CompletableFuture<WebPage> loadAsync(String url, String args)

  Load a normal url in java async style

  Parameters:

  url - The url to load

  Returns:

  A completable future of webpage

• loadAsync

   CompletableFuture<WebPage> loadAsync(String url, LoadOptions options)

  Load a normal url in java async style

  Parameters:

  url - The url to load

  Returns:

  A completable future of webpage

• loadAsync

   CompletableFuture<WebPage> loadAsync(UrlAware url)

  Load a normal url in java async style

  Parameters:

  url - The url to load

  Returns:

  A completable future of webpage

• loadAsync

   CompletableFuture<WebPage> loadAsync(UrlAware url, String args)

  Load a normal url in java async style

  Parameters:

  url - The url to load

  Returns:

  A completable future of webpage

• loadAsync

   CompletableFuture<WebPage> loadAsync(UrlAware url, LoadOptions options)

  Load a normal url in java async style

  Parameters:

  url - The url to load

  Returns:

  A completable future of webpage

• loadAsync

   CompletableFuture<WebPage> loadAsync(NormUrl url)

  Load a normal url in java async style

  Parameters:

  url - The normal url to load

  Returns:

  A completable future of webpage

• loadAllAsync

   List<CompletableFuture<WebPage>> loadAllAsync(Iterable<String> urls)

  Load all normal urls in java async style

  This method first checks each url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

  Parameters:

  urls - The normal urls to load

  Returns:

  The completable futures of webpages

• loadAllAsync

   List<CompletableFuture<WebPage>> loadAllAsync(Iterable<String> urls, String args)

  Load all normal urls in java async style.

  This method first checks each url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

  Parameters:

  urls - The normal urls to load

  Returns:

  The completable futures of webpages

• loadAllAsync

   List<CompletableFuture<WebPage>> loadAllAsync(Iterable<String> urls, LoadOptions options)

  Load all normal urls in java async style.

  This method first checks each url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

  Parameters:

  urls - The normal urls to load

  Returns:

  The completable futures of webpages

• loadAllAsync

   List<CompletableFuture<WebPage>> loadAllAsync(Collection<UrlAware> urls)

  Load all normal urls in java async style.

  This method first checks each url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

  Parameters:

  urls - The normal urls to load

  Returns:

  The completable futures of webpages

• loadAllAsync

   List<CompletableFuture<WebPage>> loadAllAsync(Collection<UrlAware> urls, String args)

  Load all normal urls in java async style.

  This method first checks each url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

  Parameters:

  urls - The normal urls to load

  Returns:

  The completable futures of webpages

• loadAllAsync

   List<CompletableFuture<WebPage>> loadAllAsync(Collection<UrlAware> urls, LoadOptions options)

  Load all normal urls in java async style.

  This method first checks each url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

  Parameters:

  urls - The normal urls to load

  Returns:

  The completable futures of webpages

• loadAllAsync

   List<CompletableFuture<WebPage>> loadAllAsync(List<NormUrl> urls)

  Load all normal urls in java async style

  This method first checks each url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

  Parameters:

  urls - The normal urls to load

  Returns:

  The completable futures of webpages

• submit

   AbstractPulsarSession submit(String url)

  Submit a url to the URL pool, the url will be processed in the crawl loop later

  Parameters:

  url - The url to submit

  Returns:

  The PulsarSession itself to enabled chained operations

• submit

   AbstractPulsarSession submit(String url, String args)

  Submit a url to the URL pool, and it will be processed in a crawl loop

  Parameters:

  url - The url to submit

  args - The load arguments

  Returns:

  The PulsarSession itself to enabled chained operations

• submit

   AbstractPulsarSession submit(String url, LoadOptions options)

  Submit a url to the URL pool, and it will be processed in a crawl loop

  Parameters:

  url - The url to submit

  options - The load options

  Returns:

  The PulsarSession itself to enabled chained operations

• submit

   AbstractPulsarSession submit(UrlAware url)

  Submit a url to the URL pool, and it will be processed in a crawl loop

  Parameters:

  url - The url to submit

  Returns:

  The PulsarSession itself to enabled chained operations

• submit

   AbstractPulsarSession submit(UrlAware url, String args)

  Submit a url to the URL pool, and it will be processed in a crawl loop

  Parameters:

  url - The url to submit

  Returns:

  The PulsarSession itself to enabled chained operations

• submitAll

   AbstractPulsarSession submitAll(Iterable<String> urls)

  Submit the urls to the URL pool, the submitted urls will be processed in a crawl loop

  Parameters:

  urls - The urls to submit

  Returns:

  The PulsarSession itself to enabled chained operations

• submitAll

   AbstractPulsarSession submitAll(Iterable<String> urls, String args)

  Submit the urls to the URL pool, the submitted urls will be processed in a crawl loop

  Parameters:

  urls - The urls to submit

  args - The load arguments

  Returns:

  The PulsarSession itself to enabled chained operations

• submitAll

   AbstractPulsarSession submitAll(Iterable<String> urls, LoadOptions options)

  Submit the urls to the URL pool, the submitted urls will be processed in a crawl loop

  Parameters:

  urls - The urls to submit

  options - The load options

  Returns:

  The PulsarSession itself to enabled chained operations

• submitAll

   AbstractPulsarSession submitAll(Collection<UrlAware> urls)

  Submit the urls to the URL pool, the submitted urls will be processed in a crawl loop

  Parameters:

  urls - The urls to submit

  Returns:

  The PulsarSession itself to enabled chained operations

• submitAll

   AbstractPulsarSession submitAll(Collection<UrlAware> urls, String args)

  Submit the urls to the URL pool, the submitted urls will be processed in a crawl loop

  Parameters:

  urls - The urls to submit

  Returns:

  The PulsarSession itself to enabled chained operations

• loadOutPages

   List<WebPage> loadOutPages(String portalUrl, String args)

  Load or fetch the portal page, and then load or fetch the out links selected by -outLink option.

  Option -outLink specifies the cssSelector for links in the portal page to load.

  Parameters:

  portalUrl - The portal url from where to load pages

  args - The load arguments

  Returns:

  The loaded out pages

• loadOutPages

   List<WebPage> loadOutPages(String portalUrl, LoadOptions options)

  Load or fetch the portal page, and then load or fetch the out links selected by -outLink option.

  Option -outLink specifies the cssSelector for links in the portal page to load.

  Parameters:

  portalUrl - The portal url from where to load pages

  options - The load options

  Returns:

  The loaded out pages

• loadOutPages

   List<WebPage> loadOutPages(UrlAware portalUrl, String args)

  Load or fetch the portal page, and then load or fetch the out links selected by -outLink option.

  Option -outLink specifies the cssSelector for links in the portal page to load.

  Parameters:

  portalUrl - The portal url from where to load pages

  args - The load arguments

  Returns:

  The loaded out pages

• loadOutPages

   List<WebPage> loadOutPages(UrlAware portalUrl, LoadOptions options)

  Load or fetch the portal page, and then load or fetch the out links selected by -outLink option.

  Option -outLink specifies the cssSelector for links in the portal page to load.

  Parameters:

  portalUrl - The portal url from where to load pages

  options - The load options

  Returns:

  The loaded out pages

• submitForOutPages

   AbstractPulsarSession submitForOutPages(String portalUrl, String args)

  Load the portal page and submit the out links specified by the -outLink option to the URL pool.

  Option -outLink specifies the cssSelector for links in the portal page to load.

  The submitted urls will be processed in a crawl loop later.

  Parameters:

  portalUrl - The portal url from where to load pages

  args - The load arguments

  Returns:

  The PulsarSession itself to enable chained operation

• submitForOutPages

   AbstractPulsarSession submitForOutPages(String portalUrl, LoadOptions options)

  Load the portal page and submit the out links specified by the -outLink option to the URL pool.

  Option -outLink specifies the cssSelector for links in the portal page to load.

  The submitted urls will be processed in a crawl loop later.

  Parameters:

  portalUrl - The portal url from where to load pages

  options - The load options

  Returns:

  The PulsarSession itself to enable chained operation

• submitForOutPages

   AbstractPulsarSession submitForOutPages(UrlAware portalUrl, String args)

  Load the portal page and submit the out links specified by the -outLink option to the URL pool.

  Option -outLink specifies the cssSelector for links in the portal page to load.

  The submitted urls will be processed in a crawl loop later.

  Parameters:

  portalUrl - The portal url from where to load pages

  args - The load arguments

  Returns:

  The PulsarSession itself to enable chained operation

• submitForOutPages

   AbstractPulsarSession submitForOutPages(UrlAware portalUrl, LoadOptions options)

  Load the portal page and submit the out links specified by the -outLink option to the URL pool.

  Option -outLink specifies the cssSelector for links in the portal page to load.

  The submitted urls will be processed in a crawl loop later.

  Parameters:

  portalUrl - The portal url from where to load pages

  options - The load options

  Returns:

  The PulsarSession itself to enable chained operation

• loadOutPagesAsync

   List<CompletableFuture<WebPage>> loadOutPagesAsync(String portalUrl, String args)

  Load or fetch the portal page, and then load or fetch the out links selected by -outLink option asynchronously.

  Option -outLink specifies the cssSelector for links in the portal page to load.

  Parameters:

  portalUrl - The portal url from where to load pages

  args - The load arguments

  Returns:

  The loaded out pages

• loadOutPagesAsync

   List<CompletableFuture<WebPage>> loadOutPagesAsync(String portalUrl, LoadOptions options)

  Load or fetch the portal page, and then load or fetch the out links selected by -outLink option asynchronously.

  Option -outLink specifies the cssSelector for links in the portal page to load.

  Parameters:

  portalUrl - The portal url from where to load pages

  options - The load options

  Returns:

  The loaded out pages

• loadResource

   WebPage loadResource(String url, String referrer)

  Load a url as a resource without browser rendering.

  Referrer will be opened by a browser first to obtain browsing environment, such as headers and cookies, and then the browsing environment will be applied to the later resource fetching.

  Parameters:

  url - The url to load

  referrer - The referrer URL

  Returns:

  The webpage containing the resource

• loadResource

   WebPage loadResource(String url, String referrer, String args)

  Load a url as a resource without browser rendering.

  Referrer will be opened by a browser first to obtain browsing environment, such as headers and cookies, and then the browsing environment will be applied to the later resource fetching.

  Parameters:

  url - The url to load

  referrer - The referrer URL

  args - The load arguments

  Returns:

  The webpage containing the resource

• loadResource

   WebPage loadResource(String url, String referrer, LoadOptions options)

  Load a url as a resource without browser rendering.

  Referrer will be opened by a browser first to obtain browsing environment, such as headers and cookies, and then the browsing environment will be applied to the later resource fetching.

  Parameters:

  url - The url to load

  referrer - The referrer URL

  options - The load options

  Returns:

  The webpage containing the resource

• loadResourceDeferred

   WebPage loadResourceDeferred(String url, String referrer)

  Load a url as a resource without browser rendering.

  Referrer will be opened by a browser first to obtain browsing environment, such as headers and cookies, and then the browsing environment will be applied to the later resource fetching.

  This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.

  Parameters:

  url - The url to load

  referrer - The referrer URL

  Returns:

  The webpage containing the resource

• loadResourceDeferred

   WebPage loadResourceDeferred(String url, String referrer, String args)

  Load a url as a resource without browser rendering.

  Referrer will be opened by a browser first to obtain browsing environment, such as headers and cookies, and then the browsing environment will be applied to the later resource fetching.

  This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.

  Parameters:

  url - The url to load

  referrer - The referrer URL

  args - The load arguments

  Returns:

  The webpage containing the resource

• loadResourceDeferred

   WebPage loadResourceDeferred(String url, String referrer, LoadOptions options)

  Load a url as a resource without browser rendering.

  Referrer will be opened by a browser first to obtain browsing environment, such as headers and cookies, and then the browsing environment will be applied to the later resource fetching.

  This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.

  Parameters:

  url - The url to load

  referrer - The referrer URL

  options - The load options

  Returns:

  The webpage containing the resource

• parse

   FeaturedDocument parse(WebPage page)

  Parse a webpage into an HTML document.

• parse

   FeaturedDocument parse(WebPage page, Boolean noCache)

  Parse a webpage into an HTML document.

• loadDocument

   FeaturedDocument loadDocument(String url)

  Load or fetch a webpage and parse it into an HTML document

• loadDocument

   FeaturedDocument loadDocument(String url, String args)

  Load or fetch a webpage and parse it into an HTML document

• loadDocument

   FeaturedDocument loadDocument(String url, LoadOptions options)

  Load or fetch a webpage and parse it into an HTML document

• loadDocument

   FeaturedDocument loadDocument(UrlAware url)

  Load or fetch a webpage and parse it into an HTML document

• loadDocument

   FeaturedDocument loadDocument(UrlAware url, String args)

  Load or fetch a webpage and parse it into an HTML document

• loadDocument

   FeaturedDocument loadDocument(UrlAware url, LoadOptions options)

  Load or fetch a webpage and parse it into an HTML document

• loadDocument

   FeaturedDocument loadDocument(NormUrl url)

  Load or fetch a webpage and then parse it into an HTML document.

• scrape

   Map<String, String> scrape(String url, String args, Iterable<String> fieldSelectors)

  Load or fetch a webpage located by the given url, and then extract fields specified by field selectors.

  Parameters:

  url - The url to scrape

  args - The load arguments

  fieldSelectors - The selectors to extract fields

  Returns:

  All the extracted fields and their selectors

• scrape

   Map<String, String> scrape(String url, LoadOptions options, Iterable<String> fieldSelectors)

  Load or fetch a webpage located by the given url, and then extract fields specified by field selectors.

  Parameters:

  url - The url to scrape

  options - The load options

  fieldSelectors - The selectors to extract fields

  Returns:

  All the extracted fields and their selectors

• scrape

   Map<String, String> scrape(String url, String args, Map<String, String> fieldSelectors)

  Load or fetch a webpage located by the given url, and then extract fields specified by field selectors.

  Parameters:

  url - The url to scrape

  args - The load arguments

  fieldSelectors - The selectors to extract fields

  Returns:

  All the extracted fields and their names

• scrape

   Map<String, String> scrape(String url, LoadOptions options, Map<String, String> fieldSelectors)

  Load or fetch a webpage located by the given url, and then extract fields specified by field selectors.

  Parameters:

  url - The url to scrape

  options - The load options

  fieldSelectors - The selectors to extract fields

  Returns:

  All the extracted fields and their names

• scrape

   List<Map<String, String>> scrape(String url, String args, String restrictSelector, Iterable<String> fieldSelectors)

  Load or fetch a webpage located by the given url, and then extract fields specified by field selectors.

  Parameters:

  url - The url to scrape

  args - The load arguments

  restrictSelector - A CSS selector to locate a DOM where all fields are restricted to

  fieldSelectors - The selectors to extract fields

  Returns:

  All the extracted fields and their names

• scrape

   List<Map<String, String>> scrape(String url, LoadOptions options, String restrictSelector, Iterable<String> fieldSelectors)

  Load or fetch a webpage located by the given url, and then extract fields specified by field selectors.

  Parameters:

  url - The url to scrape

  options - The load options

  restrictSelector - A CSS selector to locate a DOM where all fields are restricted to

  fieldSelectors - The selectors to extract fields

  Returns:

  All the extracted fields and their selectors

• scrape

   List<Map<String, String>> scrape(String url, String args, String restrictSelector, Map<String, String> fieldSelectors)

  Load or fetch a webpage located by the given url, and then extract fields specified by field selectors.

  Parameters:

  url - The url to scrape

  args - The load arguments

  restrictSelector - A CSS selector to locate a DOM where all fields are restricted to

  fieldSelectors - The selectors to extract fields

  Returns:

  All the extracted fields and their names

• scrape

   List<Map<String, String>> scrape(String url, LoadOptions options, String restrictSelector, Map<String, String> fieldSelectors)

  Load or fetch a webpage located by the given url, and then extract fields specified by field selectors.

  Parameters:

  url - The url to scrape

  options - The load options

  restrictSelector - A CSS selector to locate a DOM where all fields are restricted to

  fieldSelectors - The selectors to extract fields

  Returns:

  All the extracted fields and their names

• scrapeOutPages

  @ExperimentalApi() List<Map<String, String>> scrapeOutPages(String portalUrl, String args, Iterable<String> fieldSelectors)

  Load or fetch out pages specified by out link selector, and then extract fields specified by field selectors from each out page.

  Parameters:

  portalUrl - The portal url to start scraping

  args - Load arguments for both the portal page and out pages

  fieldSelectors - CSS selectors to extract fields from out pages

  Returns:

  All extracted fields. For each out page, fields extracted with their selectors are saved in a map.

• scrapeOutPages

  @ExperimentalApi() List<Map<String, String>> scrapeOutPages(String portalUrl, LoadOptions options, Iterable<String> fieldSelectors)

  Load or fetch out pages specified by out link selector, and then extract fields specified by field selectors from each out page.

  Parameters:

  portalUrl - The portal url to start scraping

  options - Load options for both the portal page and out pages

  fieldSelectors - CSS selectors to extract fields from out pages

  Returns:

  All extracted fields. For each out page, fields extracted with their selectors are saved in a map.

• scrapeOutPages

  @ExperimentalApi() List<Map<String, String>> scrapeOutPages(String portalUrl, String args, String restrictSelector, Iterable<String> fieldSelectors)

  Load or fetch out pages specified by out link selector, and then extract fields specified by field selectors from each out page.

  Parameters:

  portalUrl - The portal url to start scraping

  args - Load arguments for both the portal page and out pages

  restrictSelector - A CSS selector to locate a DOM where all fields are restricted to

  fieldSelectors - CSS selectors to extract fields from out pages

  Returns:

  All extracted fields. For each out page, fields extracted with their selectors are saved in a map.

• scrapeOutPages

  @ExperimentalApi() List<Map<String, String>> scrapeOutPages(String portalUrl, LoadOptions options, String restrictSelector, Iterable<String> fieldSelectors)

  Load or fetch out pages specified by out link selector, and then extract fields specified by field selectors from each out page.

  Parameters:

  portalUrl - The portal url to start scraping

  options - Load options for both the portal page and out pages

  restrictSelector - A CSS selector to locate a DOM where all fields are restricted to

  fieldSelectors - CSS selectors to extract fields from out pages

  Returns:

  All extracted fields. For each out page, fields extracted with their selectors are saved in a map.

• scrapeOutPages

  @ExperimentalApi() List<Map<String, String>> scrapeOutPages(String portalUrl, String args, Map<String, String> fieldSelectors)

  Load or fetch out pages specified by out link selector, and then extract fields specified by field selectors from each out page.

  Parameters:

  portalUrl - The portal url to start scraping

  args - Load arguments for both the portal page and out pages

  fieldSelectors - CSS selectors to extract fields from out pages

  Returns:

  All extracted fields. For each out page, fields extracted with their names are saved in a map.

• scrapeOutPages

  @ExperimentalApi() List<Map<String, String>> scrapeOutPages(String portalUrl, LoadOptions options, Map<String, String> fieldSelectors)

  Load or fetch out pages specified by out link selector, and then extract fields specified by field selectors from each out page.

  Parameters:

  portalUrl - The portal url to start scraping

  options - Load options for both the portal page and out pages

  fieldSelectors - CSS selectors to extract fields from out pages

  Returns:

  All extracted fields. For each out page, fields extracted with their names are saved in a map.

• scrapeOutPages

  @ExperimentalApi() List<Map<String, String>> scrapeOutPages(String portalUrl, String args, String restrictSelector, Map<String, String> fieldSelectors)

  Load or fetch out pages specified by out link selector, and then extract fields specified by field selectors from each out page.

  Parameters:

  portalUrl - The portal url to start scraping

  args - Load arguments for both the portal page and out pages

  restrictSelector - A CSS selector to locate a DOM where all fields are restricted to

  fieldSelectors - CSS selectors to extract fields from out pages

  Returns:

  All extracted fields. For each out page, fields extracted with their names are saved in a map.

• scrapeOutPages

  @ExperimentalApi() List<Map<String, String>> scrapeOutPages(String portalUrl, LoadOptions options, String restrictSelector, Map<String, String> fieldSelectors)

  Load or fetch out pages specified by out link selector, and then extract fields specified by field selectors from each out page.

  Parameters:

  portalUrl - The portal url to start scraping

  options - Load options for both the portal page and out pages

  restrictSelector - A CSS selector to locate a DOM where all fields are restricted to

  fieldSelectors - CSS selectors to extract fields from out pages

  Returns:

  All extracted fields. For each out page, fields extracted with their names are saved in a map.

• getVariable

   Object getVariable(String name)

  Get a variable from this session

• setVariable

   Unit setVariable(String name, Object value)

  Set a variable into this session

• putSessionBean

  @Deprecated(message = "Not used any more") Unit putSessionBean(Object obj)

  Put session scope bean

• getSessionBean

   final <T extends Any> T getSessionBean()

• delete

   Unit delete(String url)

  Delete a webpage from the storage

• flush

   Unit flush()

  Flush to the storage

• persist

   Boolean persist(WebPage page)

  Persist to the storage

• export

   Path export(WebPage page)

  Export the content of a webpage.

  Parameters:

  page - Page to export

  Returns:

  The path of the exported page

• export

   Path export(WebPage page, String ident)

  Export the content of a webpage.

  Parameters:

  page - Page to export

  ident - File name identifier used to distinguish from other names

  Returns:

  The path of the exported page

• export

   Path export(FeaturedDocument doc)

  Export the outer HTML of the document.

  Parameters:

  doc - Document to export

  Returns:

  The path of the exported document

• export

   Path export(FeaturedDocument doc, String ident)

  Export the outer HTML of the document.

  Parameters:

  doc - Document to export

  ident - File name identifier used to distinguish from other names

  Returns:

  The path of the exported document

• exportTo

   Path exportTo(FeaturedDocument doc, Path path)

  Export the whole HTML of the document to the given path.

  Parameters:

  doc - Document to export

  path - Path to save the exported content

  Returns:

  The path of the exported document

• equals

   Boolean equals(Object other)

• hashCode

   Integer hashCode()

• toString

   String toString()

• close

   Unit close()