Constructor Detail

• AbstractPulsarSession

  AbstractPulsarSession(AbstractPulsarContext context, VolatileConfig sessionConfig, Integer id)

Method Detail

• getUnmodifiedConfig

   ImmutableConfig getUnmodifiedConfig()

• getSessionBeanFactory

   BeanFactory getSessionBeanFactory()

  The session scoped bean factory

• getDisplay

   String getDisplay()

• getIsActive

   final Boolean getIsActive()

• getGlobalCacheFactory

   GlobalCacheFactory getGlobalCacheFactory()

• getPageCache

   ConcurrentExpiringLRUCache<String, WebPage> getPageCache()

• getDocumentCache

   ConcurrentExpiringLRUCache<String, FeaturedDocument> getDocumentCache()

• getContext

   AbstractPulsarContext getContext()

  The pulsar context

• getSessionConfig

   VolatileConfig getSessionConfig()

  The session scope volatile config, every setting is supposed to be changed at any time and any place

• getId

   Integer getId()

  The session id. Session id is expected to be set by the container, e.g. the h2 database runtime

• registerClosable

   Boolean registerClosable(AutoCloseable closable)

  Close objects when the session is closing

• disablePDCache

   Unit disablePDCache()

  Disable page cache and document cache

• options

   LoadOptions options(String args, PulsarEventHandler eventHandler)

  Create a new options, with a new volatile config

• property

   String property(String name)

  Get a property

• property

   Unit property(String name, String value)

  Set a session scope property

• normalize

   NormUrl normalize(String url, String args)

  Normalize a url

• normalize

   NormUrl normalize(String url, LoadOptions options, Boolean toItemOption)

  Normalize a url

• normalize

   List<NormUrl> normalize(Iterable<String> urls, LoadOptions options, Boolean toItemOption)

  Normalize urls

• normalize

   NormUrl normalize(UrlAware url, LoadOptions options, Boolean toItemOption)

  Normalize a url

• normalize

   List<NormUrl> normalize(Collection<UrlAware> urls, LoadOptions options, Boolean toItemOption)

  Normalize urls

• normalizeOrNull

   NormUrl normalizeOrNull(String url, LoadOptions options, Boolean toItemOption)

  Normalize a url

• normalizeOrNull

   NormUrl normalizeOrNull(UrlAware url, LoadOptions options, Boolean toItemOption)

  Normalize a url

• inject

   WebPage inject(String url)

  Inject a url

  Parameters:

  url - The url followed by options

• get

   WebPage get(String url)

  Get a page from database if exists

  Parameters:

  url - The url

• getOrNull

   WebPage getOrNull(String url)

  Get a page from database if exists

  Parameters:

  url - The url

• exists

   Boolean exists(String url)

  Check if a page exists in the database

  Parameters:

  url - The url

• fetchState

   CheckState fetchState(WebPage page, LoadOptions options)

  Return the fetch state of the page

  Parameters:

  page - The webpage

  options - The load options

• open

   WebPage open(String url)

  Open a page with url

  Parameters:

  url - The url of the page to open

• load

   WebPage load(String url, String args)

  Load a url with specified options

  Parameters:

  url - The url to load

  args - The load args

• load

   WebPage load(String url, LoadOptions options)

  Load a url with specified options

  Parameters:

  url - The url to load

  options - The load options

• load

   WebPage load(UrlAware url, String args)

  Load a url with specified options

  Parameters:

  url - The url to load

  args - The load args

• load

   WebPage load(UrlAware url, LoadOptions options)

  Load a url with specified options

  Parameters:

  url - The url to load

  options - The load options

• load

   WebPage load(NormUrl normUrl)

  Load a url with specified options

  Parameters:

  normUrl - The normalized url

• loadDeferred

   WebPage loadDeferred(String url, LoadOptions options)

  Load a url with specified options

  Parameters:

  url - The url to load

  options - The load options

• loadDeferred

   WebPage loadDeferred(UrlAware url, String args)

  Load a url with specified options

  Parameters:

  url - The url to load

  args - The load args

• loadDeferred

   WebPage loadDeferred(UrlAware url, LoadOptions options)

  Parameters:

  url - The url to load

  options - The load options

• loadDeferred

   WebPage loadDeferred(NormUrl normUrl)

  Load a url with specified options

  Parameters:

  normUrl - The normalized url

• loadAll

   List<WebPage> loadAll(Iterable<String> urls, LoadOptions options, Boolean toItemOption)

  Load all urls with specified options, this may cause a parallel fetching if required

  Parameters:

  urls - The urls to load

  options - The load options for all urls

• loadAll

   List<WebPage> loadAll(Iterable<NormUrl> normUrls)

  Load all urls with specified options

  Parameters:

  normUrls - The urls to load

• loadAsync

   CompletableFuture<WebPage> loadAsync(NormUrl url)

  Load a url with java async style

  Parameters:

  url - The url to load

• loadAllAsync

   List<CompletableFuture<WebPage>> loadAllAsync(Iterable<NormUrl> urls)

  Load all urls with specified options with java async style

  Parameters:

  urls - The urls to load

• submit

   AbstractPulsarSession submit(UrlAware url)

  Submit a url to the url pool, the url will be processed in the main crawl loop later

  Parameters:

  url - The url to submit

• submitAll

   AbstractPulsarSession submitAll(Iterable<UrlAware> urls)

  Submit the urls to the url pool, the submitted urls will be processed in the main crawl loop later

  Parameters:

  urls - The urls to submit

• loadOutPages

   List<WebPage> loadOutPages(String portalUrl, String args)

  Load all out pages in a portal page

  Parameters:

  portalUrl - The portal url from where to load pages

  args - The load args

• loadOutPages

   List<WebPage> loadOutPages(String portalUrl, LoadOptions options)

  Load all out pages in a portal page

  Parameters:

  portalUrl - The portal url from where to load pages

  options - The load options

• submitOutPages

   AbstractPulsarSession submitOutPages(String portalUrl, String args)

  Submit the urls of out pages in the portal page, the submitted urls will be processed in the main crawl loop later

  Parameters:

  portalUrl - The portal url from where to load pages

  args - The load arguments

• submitOutPages

   AbstractPulsarSession submitOutPages(String portalUrl, LoadOptions options)

  Submit the urls of out pages in the portal page, the submitted urls will be processed in the main crawl loop later

  Parameters:

  portalUrl - The portal url from where to load pages

  options - The load options

• loadOutPagesAsync

   List<CompletableFuture<WebPage>> loadOutPagesAsync(String portalUrl, LoadOptions options)

  Load out pages linked from the portal page

  Parameters:

  portalUrl - The portal url from where to load pages

  options - The load options

• loadResource

   WebPage loadResource(String url, String referer, String args)

  Load a url as a resource without browser rendering in the browser context

  Parameters:

  url - The url to load

  args - The load arguments

• loadResource

   WebPage loadResource(String url, String referer, LoadOptions opts)

  Load a url as a resource without browser rendering in the browser context

  Parameters:

  url - The url to load

  opts - The load options

• parse

   FeaturedDocument parse(WebPage page, Boolean noCache)

  Parse the Web page into DOM. If the Web page is not changed since last parse, use the last result if available

• loadDocument

   FeaturedDocument loadDocument(String url, String args)

  Load or fetch a webpage and parse it into a document

• loadDocument

   FeaturedDocument loadDocument(String url, LoadOptions options)

  Load or fetch a webpage and parse it into a document

• loadDocument

   FeaturedDocument loadDocument(NormUrl normUrl)

  Load or fetch a webpage and parse it into a document

• scrape

   Map<String, String> scrape(String url, String args, Iterable<String> fieldSelectors)

  Scrape a webpage

• scrape

   Map<String, String> scrape(String url, String args, Map<String, String> fieldSelectors)

  Scrape a webpage

• scrape

   List<Map<String, String>> scrape(String url, String args, String restrictSelector, Iterable<String> fieldSelectors)

  Scrape a webpage

• scrape

   List<Map<String, String>> scrape(String url, String args, String restrictSelector, Map<String, String> fieldSelectors)

  Scrape a webpage

• scrapeOutPages

  @ExperimentalApi() List<Map<String, String>> scrapeOutPages(String portalUrl, String args, Iterable<String> fieldSelectors)

  Scrape out pages

• scrapeOutPages

  @ExperimentalApi() List<Map<String, String>> scrapeOutPages(String portalUrl, String args, String restrictSelector, Iterable<String> fieldSelectors)

  Scrape out pages

• scrapeOutPages

  @ExperimentalApi() List<Map<String, String>> scrapeOutPages(String portalUrl, String args, Map<String, String> fieldSelectors)

  Scrape out pages

• scrapeOutPages

  @ExperimentalApi() List<Map<String, String>> scrapeOutPages(String portalUrl, String args, String restrictSelector, Map<String, String> fieldSelectors)

  Scrape out pages

• getVariable

   Object getVariable(String name)

  Get a variable associated with this session

• setVariable

   Unit setVariable(String name, Object value)

  Set a variable associated with this session

• putSessionBean

   Unit putSessionBean(Object obj)

  Put session scope bean

• getSessionBean

   final <T extends Any> T getSessionBean()

• delete

   Unit delete(String url)

  Delete a webpage from the backend storage

• flush

   Unit flush()

  Flush to the backend storage

• persist

   Boolean persist(WebPage page)

  Persist to the backend storage

• export

   Path export(WebPage page, String ident)

  Export a webpage

• export

   Path export(FeaturedDocument doc, String ident)

  Export a document

• exportTo

   Path exportTo(FeaturedDocument doc, Path path)

  Export a document to the given path

• equals

   Boolean equals(Object other)

• hashCode

   Integer hashCode()

• toString

   String toString()

• close

   Unit close()