Constructor Detail

• AbstractPulsarSession

  AbstractPulsarSession(AbstractPulsarContext context, VolatileConfig sessionConfig, Integer id)

Method Detail

• getUnmodifiedConfig

   ImmutableConfig getUnmodifiedConfig()

• getSessionBeanFactory

   BeanFactory getSessionBeanFactory()

  The scoped bean factory: for each volatileConfig object, there is a bean factory

• getDisplay

   String getDisplay()

• getIsActive

   final Boolean getIsActive()

• getGlobalCacheFactory

   GlobalCacheFactory getGlobalCacheFactory()

• getGlobalCache

   GlobalCache getGlobalCache()

• getPageCache

   ConcurrentExpiringLRUCache<String, WebPage> getPageCache()

• getDocumentCache

   ConcurrentExpiringLRUCache<String, FeaturedDocument> getDocumentCache()

• getContext

   AbstractPulsarContext getContext()

  The pulsar context

• getSessionConfig

   VolatileConfig getSessionConfig()

  The session scope volatile config, every setting is supposed to be changed at any time and any place

• getId

   Integer getId()

  The session id. Session id is expected to be set by the container, e.g. the h2 database runtime

• registerClosable

   Boolean registerClosable(AutoCloseable closable)

  Close objects when the session closes

• disablePDCache

   Unit disablePDCache()

  Disable page cache and document cache

• options

   LoadOptions options(String args, PageEvent event)

  Create a new LoadOptions object with arguments args and event.

• property

   String property(String name)

  Get a property.

• property

   Unit property(String name, String value)

  Set a session scope property.

• normalize

   NormUrl normalize(String url, String args)

  Normalize a url.

• normalize

   NormUrl normalize(String url, LoadOptions options, Boolean toItemOption)

  Normalize a url.

• normalize

   List<NormUrl> normalize(Iterable<String> urls, LoadOptions options, Boolean toItemOption)

  Normalize urls.

• normalize

   NormUrl normalize(UrlAware url, LoadOptions options, Boolean toItemOption)

  Normalize a url.

• normalize

   List<NormUrl> normalize(Collection<UrlAware> urls, LoadOptions options, Boolean toItemOption)

  Normalize urls.

• normalizeOrNull

   NormUrl normalizeOrNull(String url, LoadOptions options, Boolean toItemOption)

  Normalize a url.

• normalizeOrNull

   NormUrl normalizeOrNull(UrlAware url, LoadOptions options, Boolean toItemOption)

  Normalize a url.

• inject

   WebPage inject(String url)

  Inject a url as a seed to fetch. Injection is usually used in Nutch style crawls where the execution flow likes the following:

  inject -> generate -> fetch -> parse -> index -> update ^ ^ | <- <- <- <- |

  Parameters:

  url - The url to inject, con be followed by arguments

• get

   WebPage get(String url)

  Get a page from storage.

  Parameters:

  url - The url

• getOrNull

   WebPage getOrNull(String url)

  Get a page from storage.

  Parameters:

  url - The url

• exists

   Boolean exists(String url)

  Check if the page exists in the storage.

  Parameters:

  url - The url to check

• fetchState

   CheckState fetchState(WebPage page, LoadOptions options)

  Return the fetch state of the page.

  Parameters:

  page - The webpage

  options - The load options

• open

   WebPage open(String url)

  Open a url.

  This method opens the url immediately, regardless of the previous state of the page.

  Parameters:

  url - The url to open

• load

   WebPage load(String url, String args)

  Load a url with specified arguments.

  This method first checks the url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

  Other fetch conditions can be specified by load arguments:

  • expiration

  • page size requirement

  • fields requirement

  • other

  Parameters:

  url - The url to load

  args - The load arguments

• load

   WebPage load(String url, LoadOptions options)

  Load a url with specified options.

  This method first checks the url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

  Other fetch conditions can be specified by load arguments:

  • expiration

  • page size requirement

  • fields requirement

  • other

  Parameters:

  url - The url to load

  options - The load options

• load

   WebPage load(UrlAware url, String args)

  Load a url with the specified arguments.

  This method first checks the url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

  Parameters:

  url - The url to load

  args - The load arguments

• load

   WebPage load(UrlAware url, LoadOptions options)

  Load a url with options.

  This method first checks the url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

  Parameters:

  url - The url to load

  options - The load options

• load

   WebPage load(NormUrl normUrl)

  Load a normal url.

  This method first checks the url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

  Parameters:

  normUrl - The normal url

• loadDeferred

   WebPage loadDeferred(String url, LoadOptions options)

  Load a url with specified options.

  This method first checks the url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

  This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.

  Parameters:

  url - The url to load

  options - The load options

• loadDeferred

   WebPage loadDeferred(UrlAware url, String args)

  Load a url with specified arguments.

  This method first checks the url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

  This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.

  Parameters:

  url - The url to load

  args - The load args

• loadDeferred

   WebPage loadDeferred(UrlAware url, LoadOptions options)

  Load a url with specified options.

  This method first checks the url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

  This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.

  Parameters:

  url - The url to load

  options - The load options

• loadDeferred

   WebPage loadDeferred(NormUrl normUrl)

  Load a url with specified options

  This method first checks the url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

  This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.

  Parameters:

  normUrl - The normal url

• loadAll

   List<WebPage> loadAll(Iterable<String> urls, LoadOptions options)

  Load all urls with specified options

  This method first checks each url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

  Parameters:

  urls - The urls to load

  options - The load options

• loadAll

   List<WebPage> loadAll(Iterable<NormUrl> normUrls)

  Load all normal urls with specified options

  This method first checks each url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

  Parameters:

  normUrls - The normal urls to load

• loadAsync

   CompletableFuture<WebPage> loadAsync(NormUrl url)

  Load a normal url in java async style

  Parameters:

  url - The normal url to load

• loadAllAsync

   List<CompletableFuture<WebPage>> loadAllAsync(Iterable<NormUrl> urls)

  Load all normal urls in java async style

  This method first checks each url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

  Parameters:

  urls - The normal urls to load

• submit

   AbstractPulsarSession submit(String url, String args)

  Submit a url to the URL pool, the url will be processed in the crawl loop later

  Parameters:

  url - The url to submit

  args - The load arguments

• submit

   AbstractPulsarSession submit(UrlAware url)

  Submit a url to the URL pool, the url will be processed in a crawl loop later

  Parameters:

  url - The url to submit

• submitAll

   AbstractPulsarSession submitAll(Iterable<String> urls)

  Submit the urls to the URL pool, the submitted urls will be processed in a crawl loop later

  Parameters:

  urls - The urls to submit

• submitAll

   AbstractPulsarSession submitAll(Iterable<String> urls, String args)

  Submit the urls to the URL pool, the submitted urls will be processed in a crawl loop later

  Parameters:

  urls - The urls to submit

  args - The load arguments

• submitAll

   AbstractPulsarSession submitAll(Collection<UrlAware> urls)

  Submit the urls to the URL pool, the submitted urls will be processed in a crawl loop later

  Parameters:

  urls - The urls to submit

• loadOutPages

   List<WebPage> loadOutPages(String portalUrl, String args)

  Load or fetch the portal page, and then load or fetch the out links selected by -outLink option.

  Parameters:

  portalUrl - The portal url from where to load pages

  args - The load arguments

• loadOutPages

   List<WebPage> loadOutPages(String portalUrl, LoadOptions options)

  Load or fetch the portal page, and then load or fetch the out links selected by -outLink option.

  Parameters:

  portalUrl - The portal url from where to load pages

  options - The load options

• submitOutPages

   AbstractPulsarSession submitOutPages(String portalUrl, String args)

  Load the portal page, and then submit the specified out links to the URL pool, the out links are selected by -outLink option.

  The submitted urls will be processed in a crawl loop later.

  Parameters:

  portalUrl - The portal url from where to load pages

  args - The load arguments

• submitOutPages

   AbstractPulsarSession submitOutPages(String portalUrl, LoadOptions options)

  Load the portal page, and then submit the specified out links to the URL pool, the out links are selected by -outLink option.

  The submitted urls will be processed in a crawl loop later.

  Parameters:

  portalUrl - The portal url from where to load pages

  options - The load options

• loadOutPagesAsync

   List<CompletableFuture<WebPage>> loadOutPagesAsync(String portalUrl, LoadOptions options)

  Load or fetch the portal page, and then load or fetch the out links selected by -outLink option asynchronously.

  Parameters:

  portalUrl - The portal url from where to load pages

  options - The load options

• loadResource

   WebPage loadResource(String url, String referrer, String args)

  Load a url as a resource without browser rendering.

  Parameters:

  url - The url to load

  referrer - The referrer URL

  args - The load arguments

• loadResource

   WebPage loadResource(String url, String referrer, LoadOptions options)

  Load a url as a resource without browser rendering.

  Parameters:

  url - The url to load

  referrer - The referrer URL

  options - The load options

• loadResourceDeferred

   WebPage loadResourceDeferred(String url, String referrer, String args)

  Load a url as a resource without browser rendering.

  This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.

  Parameters:

  url - The url to load

  referrer - The referrer URL

  args - The load arguments

• loadResourceDeferred

   WebPage loadResourceDeferred(String url, String referrer, LoadOptions options)

  Load a url as a resource without browser rendering.

  This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.

  Parameters:

  url - The url to load

  referrer - The referrer URL

  options - The load options

• parse

   FeaturedDocument parse(WebPage page, Boolean noCache)

  Parse a webpage into an HTML document.

• loadDocument

   FeaturedDocument loadDocument(String url, String args)

  Load or fetch a webpage and parse it into an HTML document

• loadDocument

   FeaturedDocument loadDocument(String url, LoadOptions options)

  Load or fetch a webpage and parse it into an HTML document

• loadDocument

   FeaturedDocument loadDocument(NormUrl normUrl)

  Load or fetch a webpage and then parse it into an HTML document.

• scrape

   Map<String, String> scrape(String url, String args, Iterable<String> fieldSelectors)

  Load or fetch a webpage located by the given url, and then extract fields specified by field selectors.

  Parameters:

  url - The url to scrape

  args - The load arguments

  fieldSelectors - The selectors to extract fields

• scrape

   Map<String, String> scrape(String url, LoadOptions options, Iterable<String> fieldSelectors)

  Load or fetch a webpage located by the given url, and then extract fields specified by field selectors.

  Parameters:

  url - The url to scrape

  options - The load options

  fieldSelectors - The selectors to extract fields

• scrape

   Map<String, String> scrape(String url, String args, Map<String, String> fieldSelectors)

  Load or fetch a webpage located by the given url, and then extract fields specified by field selectors.

  Parameters:

  url - The url to scrape

  args - The load arguments

  fieldSelectors - The selectors to extract fields

• scrape

   Map<String, String> scrape(String url, LoadOptions options, Map<String, String> fieldSelectors)

  Load or fetch a webpage located by the given url, and then extract fields specified by field selectors.

  Parameters:

  url - The url to scrape

  options - The load options

  fieldSelectors - The selectors to extract fields

• scrape

   List<Map<String, String>> scrape(String url, String args, String restrictSelector, Iterable<String> fieldSelectors)

  Load or fetch a webpage located by the given url, and then extract fields specified by field selectors.

  Parameters:

  url - The url to scrape

  args - The load arguments

  restrictSelector - A CSS selector to locate a DOM where all fields are restricted to

  fieldSelectors - The selectors to extract fields

• scrape

   List<Map<String, String>> scrape(String url, LoadOptions options, String restrictSelector, Iterable<String> fieldSelectors)

  Load or fetch a webpage located by the given url, and then extract fields specified by field selectors.

  Parameters:

  url - The url to scrape

  options - The load options

  restrictSelector - A CSS selector to locate a DOM where all fields are restricted to

  fieldSelectors - The selectors to extract fields

• scrape

   List<Map<String, String>> scrape(String url, String args, String restrictSelector, Map<String, String> fieldSelectors)

  Load or fetch a webpage located by the given url, and then extract fields specified by field selectors.

  Parameters:

  url - The url to scrape

  args - The load arguments

  restrictSelector - A CSS selector to locate a DOM where all fields are restricted to

  fieldSelectors - The selectors to extract fields

• scrape

   List<Map<String, String>> scrape(String url, LoadOptions options, String restrictSelector, Map<String, String> fieldSelectors)

  Load or fetch a webpage located by the given url, and then extract fields specified by field selectors.

  Parameters:

  url - The url to scrape

  options - The load options

  restrictSelector - A CSS selector to locate a DOM where all fields are restricted to

  fieldSelectors - The selectors to extract fields

• scrapeOutPages

  @ExperimentalApi() List<Map<String, String>> scrapeOutPages(String portalUrl, String args, Iterable<String> fieldSelectors)

  Load or fetch out pages specified by out link selector, and then extract fields specified by field selectors from each out page.

  Parameters:

  portalUrl - The portal url to start scraping

  args - Load arguments for both the portal page and out pages

  fieldSelectors - CSS selectors to extract fields from out pages

• scrapeOutPages

  @ExperimentalApi() List<Map<String, String>> scrapeOutPages(String portalUrl, LoadOptions options, Iterable<String> fieldSelectors)

  Load or fetch out pages specified by out link selector, and then extract fields specified by field selectors from each out page.

  Parameters:

  portalUrl - The portal url to start scraping

  options - Load options for both the portal page and out pages

  fieldSelectors - CSS selectors to extract fields from out pages

• scrapeOutPages

  @ExperimentalApi() List<Map<String, String>> scrapeOutPages(String portalUrl, String args, String restrictSelector, Iterable<String> fieldSelectors)

  Load or fetch out pages specified by out link selector, and then extract fields specified by field selectors from each out page.

  Parameters:

  portalUrl - The portal url to start scraping

  args - Load arguments for both the portal page and out pages

  restrictSelector - A CSS selector to locate a DOM where all fields are restricted to

  fieldSelectors - CSS selectors to extract fields from out pages

• scrapeOutPages

  @ExperimentalApi() List<Map<String, String>> scrapeOutPages(String portalUrl, LoadOptions options, String restrictSelector, Iterable<String> fieldSelectors)

  Load or fetch out pages specified by out link selector, and then extract fields specified by field selectors from each out page.

  Parameters:

  portalUrl - The portal url to start scraping

  options - Load options for both the portal page and out pages

  restrictSelector - A CSS selector to locate a DOM where all fields are restricted to

  fieldSelectors - CSS selectors to extract fields from out pages

• scrapeOutPages

  @ExperimentalApi() List<Map<String, String>> scrapeOutPages(String portalUrl, String args, Map<String, String> fieldSelectors)

  Load or fetch out pages specified by out link selector, and then extract fields specified by field selectors from each out page.

  Parameters:

  portalUrl - The portal url to start scraping

  args - Load arguments for both the portal page and out pages

  fieldSelectors - CSS selectors to extract fields from out pages

• scrapeOutPages

  @ExperimentalApi() List<Map<String, String>> scrapeOutPages(String portalUrl, LoadOptions options, Map<String, String> fieldSelectors)

  Load or fetch out pages specified by out link selector, and then extract fields specified by field selectors from each out page.

  Parameters:

  portalUrl - The portal url to start scraping

  options - Load options for both the portal page and out pages

  fieldSelectors - CSS selectors to extract fields from out pages

• scrapeOutPages

  @ExperimentalApi() List<Map<String, String>> scrapeOutPages(String portalUrl, String args, String restrictSelector, Map<String, String> fieldSelectors)

  Load or fetch out pages specified by out link selector, and then extract fields specified by field selectors from each out page.

  Parameters:

  portalUrl - The portal url to start scraping

  args - Load arguments for both the portal page and out pages

  restrictSelector - A CSS selector to locate a DOM where all fields are restricted to

  fieldSelectors - CSS selectors to extract fields from out pages

• scrapeOutPages

  @ExperimentalApi() List<Map<String, String>> scrapeOutPages(String portalUrl, LoadOptions options, String restrictSelector, Map<String, String> fieldSelectors)

  Load or fetch out pages specified by out link selector, and then extract fields specified by field selectors from each out page.

  Parameters:

  portalUrl - The portal url to start scraping

  options - Load options for both the portal page and out pages

  restrictSelector - A CSS selector to locate a DOM where all fields are restricted to

  fieldSelectors - CSS selectors to extract fields from out pages

• getVariable

   Object getVariable(String name)

  Get a variable from this session

• setVariable

   Unit setVariable(String name, Object value)

  Set a variable into this session

• putSessionBean

   Unit putSessionBean(Object obj)

  Put session scope bean

• getSessionBean

   final <T extends Any> T getSessionBean()

• delete

   Unit delete(String url)

  Delete a webpage from the storage

• flush

   Unit flush()

  Flush to the storage

• persist

   Boolean persist(WebPage page)

  Persist to the storage

• export

   Path export(WebPage page, String ident)

  Export the content of a webpage.

  Parameters:

  page - Page to export

  ident - File name identifier used to distinguish from other names

• export

   Path export(FeaturedDocument doc, String ident)

  Export the outer HTML of the document.

  Parameters:

  doc - Document to export

  ident - File name identifier used to distinguish from other names

• exportTo

   Path exportTo(FeaturedDocument doc, Path path)

  Export the whole HTML of the document to the given path.

  Parameters:

  doc - Document to export

  path - Path to save the exported content

• equals

   Boolean equals(Object other)

• hashCode

   Integer hashCode()

• toString

   String toString()

• close

   Unit close()