Method Detail

• registerClosable

   abstract Boolean registerClosable(AutoCloseable closable)

  Close objects when sessions close

• disablePDCache

   abstract Unit disablePDCache()

  Disable page cache and document cache

• options

   abstract LoadOptions options(String args, PulsarEventHandler eventHandler)

  Create a new options, with a new volatile config

• property

   abstract String property(String name)

  Get a property

• property

   abstract Unit property(String name, String value)

  Set a session scope property

• normalize

   abstract NormUrl normalize(String url, String args)

  Normalize a url

• normalize

   abstract NormUrl normalize(String url, LoadOptions options, Boolean toItemOption)

  Normalize a url

• normalize

   abstract List<NormUrl> normalize(Iterable<String> urls, LoadOptions options, Boolean toItemOption)

  Normalize urls

• normalize

   abstract NormUrl normalize(UrlAware url, LoadOptions options, Boolean toItemOption)

  Normalize a url

• normalize

   abstract List<NormUrl> normalize(Collection<UrlAware> urls, LoadOptions options, Boolean toItemOption)

  Normalize urls

• normalizeOrNull

   abstract NormUrl normalizeOrNull(String url, LoadOptions options, Boolean toItemOption)

  Normalize a url

• normalizeOrNull

   abstract NormUrl normalizeOrNull(UrlAware url, LoadOptions options, Boolean toItemOption)

  Normalize a url

• inject

   abstract WebPage inject(String url)

  Inject a url to fetch later

  Parameters:

  url - The url followed by options

• get

   abstract WebPage get(String url)

  Get a page from database if exists

  Parameters:

  url - The url

• getOrNull

   abstract WebPage getOrNull(String url)

  Get a page from database if exists

  Parameters:

  url - The url

• exists

   abstract Boolean exists(String url)

  Check if a page exists in the database

  Parameters:

  url - The url

• fetchState

   abstract CheckState fetchState(WebPage page, LoadOptions options)

  Return the fetch state of the page

  Parameters:

  page - The webpage

  options - The load options

• open

   abstract WebPage open(String url)

  Open a page with url

  Parameters:

  url - The url of the page to open

• load

   abstract WebPage load(String url, String args)

  Load a url with specified options

  Parameters:

  url - The url to load

  args - The load args

• load

   abstract WebPage load(String url, LoadOptions options)

  Load a url with specified options

  Parameters:

  url - The url to load

  options - The load options

• load

   abstract WebPage load(UrlAware url, String args)

  Load a url with specified options

  Parameters:

  url - The url to load

  args - The load args

• load

   abstract WebPage load(UrlAware url, LoadOptions options)

  Load a url with specified options

  Parameters:

  url - The url to load

  options - The load options

• load

   abstract WebPage load(NormUrl normUrl)

  Load a url with specified options

  Parameters:

  normUrl - The normalized url

• loadDeferred

   abstract WebPage loadDeferred(String url, LoadOptions options)

  Load a url with specified options

  Parameters:

  url - The url to load

  options - The load options

• loadDeferred

   abstract WebPage loadDeferred(UrlAware url, String args)

  Load a url with specified options

  Parameters:

  url - The url to load

  args - The load args

• loadDeferred

   abstract WebPage loadDeferred(UrlAware url, LoadOptions options)

  Load a url with specified options

  Parameters:

  url - The url to load

  options - The load options

• loadDeferred

   abstract WebPage loadDeferred(NormUrl normUrl)

  Load a url with specified options

  Parameters:

  normUrl - The normalized url

• loadAll

   abstract List<WebPage> loadAll(Iterable<String> urls, LoadOptions options)

  Load all urls with specified options, this causes a parallel fetching whenever applicable

  Parameters:

  urls - The urls to load

  options - The load options

• loadAll

   abstract List<WebPage> loadAll(Iterable<NormUrl> normUrls)

  Load all urls with specified options

  Parameters:

  normUrls - The urls to load

• loadAsync

   abstract CompletableFuture<WebPage> loadAsync(NormUrl url)

  Load a url with java async style

  Parameters:

  url - The url to load

• loadAllAsync

   abstract List<CompletableFuture<WebPage>> loadAllAsync(Iterable<NormUrl> urls)

  Load all urls with specified options with java async style

  Parameters:

  urls - The urls to load

• submit

   abstract PulsarSession submit(UrlAware url)

  Submit a url to the url pool, the url will be processed in the main crawl loop later

  Parameters:

  url - The url to submit

• submitAll

   abstract PulsarSession submitAll(Iterable<UrlAware> urls)

  Submit the urls to the url pool, the submitted urls will be processed in the main crawl loop later

  Parameters:

  urls - The urls to submit

• loadOutPages

   abstract List<WebPage> loadOutPages(String portalUrl, String args)

  Load out pages linked from the portal page

  Parameters:

  portalUrl - The portal url from where to load pages

  args - The load args

• loadOutPages

   abstract List<WebPage> loadOutPages(String portalUrl, LoadOptions options)

  Load out pages linked from the portal page

  Parameters:

  portalUrl - The portal url from where to load pages

  options - The load options

• loadOutPagesAsync

   abstract List<CompletableFuture<WebPage>> loadOutPagesAsync(String portalUrl, LoadOptions options)

  Load out pages linked from the portal page

  Parameters:

  portalUrl - The portal url from where to load pages

  options - The load options

• submitOutPages

   abstract AbstractPulsarSession submitOutPages(String portalUrl, String args)

  Submit the urls of out pages in the portal page, the submitted urls will be processed in the main crawl loop later

  Parameters:

  portalUrl - The portal url from where to load pages

  args - The load arguments

• submitOutPages

   abstract PulsarSession submitOutPages(String portalUrl, LoadOptions options)

  Submit the urls of out pages in the portal page, the submitted urls will be processed in the main crawl loop later

  Parameters:

  portalUrl - The portal url from where to load pages

  options - The load options

• loadResource

   abstract WebPage loadResource(String url, String referer, String args)

  Load a url as a resource without browser rendering in the browser context

  Parameters:

  url - The url to load

  args - The load arguments

• loadResource

   abstract WebPage loadResource(String url, String referer, LoadOptions opts)

  Load a url as a resource without browser rendering in the browser context

  Parameters:

  url - The url to load

  opts - The load options

• parse

   abstract FeaturedDocument parse(WebPage page, Boolean noCache)

  Parse the Web page into DOM. If the Web page is not changed since last parse, use the last result if available

• loadDocument

   abstract FeaturedDocument loadDocument(String url, String args)

  Load or fetch a webpage and parse it into a document

• loadDocument

   abstract FeaturedDocument loadDocument(String url, LoadOptions options)

  Load or fetch a webpage and parse it into a document

• loadDocument

   abstract FeaturedDocument loadDocument(NormUrl normUrl)

  Load or fetch a webpage and parse it into a document

• scrape

   abstract Map<String, String> scrape(String url, String args, Iterable<String> fieldSelectors)

  Scrape a webpage

• scrape

   abstract Map<String, String> scrape(String url, LoadOptions options, Iterable<String> fieldSelectors)

  Scrape a webpage

• scrape

   abstract Map<String, String> scrape(String url, String args, Map<String, String> fieldSelectors)

  Scrape a webpage

• scrape

   abstract Map<String, String> scrape(String url, LoadOptions options, Map<String, String> fieldSelectors)

  Scrape a webpage

• scrape

   abstract List<Map<String, String>> scrape(String url, String args, String restrictSelector, Iterable<String> fieldSelectors)

  Scrape a webpage

• scrape

   abstract List<Map<String, String>> scrape(String url, LoadOptions options, String restrictSelector, Iterable<String> fieldSelectors)

  Scrape a webpage

• scrape

   abstract List<Map<String, String>> scrape(String url, String args, String restrictSelector, Map<String, String> fieldSelectors)

  Scrape a webpage

• scrape

   abstract List<Map<String, String>> scrape(String url, LoadOptions options, String restrictSelector, Map<String, String> fieldSelectors)

  Scrape a webpage

• scrapeOutPages

  @ExperimentalApi() abstract List<Map<String, String>> scrapeOutPages(String portalUrl, String args, Iterable<String> fieldSelectors)

  Scrape out pages

• scrapeOutPages

  @ExperimentalApi() abstract List<Map<String, String>> scrapeOutPages(String portalUrl, LoadOptions options, Iterable<String> fieldSelectors)

  Scrape out pages

• scrapeOutPages

  @ExperimentalApi() abstract List<Map<String, String>> scrapeOutPages(String portalUrl, String args, String restrictSelector, Iterable<String> fieldSelectors)

  Scrape out pages

• scrapeOutPages

  @ExperimentalApi() abstract List<Map<String, String>> scrapeOutPages(String portalUrl, LoadOptions options, String restrictSelector, Iterable<String> fieldSelectors)

  Scrape out pages

• scrapeOutPages

  @ExperimentalApi() abstract List<Map<String, String>> scrapeOutPages(String portalUrl, String args, Map<String, String> fieldSelectors)

  Scrape out pages

• scrapeOutPages

  @ExperimentalApi() abstract List<Map<String, String>> scrapeOutPages(String portalUrl, LoadOptions options, Map<String, String> fieldSelectors)

  Scrape out pages

• scrapeOutPages

  @ExperimentalApi() abstract List<Map<String, String>> scrapeOutPages(String portalUrl, String args, String restrictSelector, Map<String, String> fieldSelectors)

  Scrape out pages

• scrapeOutPages

  @ExperimentalApi() abstract List<Map<String, String>> scrapeOutPages(String portalUrl, LoadOptions options, String restrictSelector, Map<String, String> fieldSelectors)

  Scrape out pages

• getVariable

   abstract Object getVariable(String name)

  Get a variable associated with this session

• setVariable

   abstract Unit setVariable(String name, Object value)

  Set a variable associated with this session

• putSessionBean

   abstract Unit putSessionBean(Object obj)

  Put session scope bean

• delete

   abstract Unit delete(String url)

  Delete a webpage from the backend storage

• flush

   abstract Unit flush()

  Flush to the backend storage

• persist

   abstract Boolean persist(WebPage page)

  Persist to the backend storage

• export

   abstract Path export(WebPage page, String ident)

  Export a webpage

• export

   abstract Path export(FeaturedDocument doc, String ident)

  Export a document

• exportTo

   abstract Path exportTo(FeaturedDocument doc, Path path)

  Export a document to the given path

• getId

   abstract Integer getId()

  The session id. Session id is expected to be set by the container, e.g. the h2 database runtime

• getContext

   abstract PulsarContext getContext()

  The pulsar context

• getSessionConfig

   abstract VolatileConfig getSessionConfig()

  The session scope volatile config, every setting is supposed to be changed at any time and any place

• getUnmodifiedConfig

   abstract ImmutableConfig getUnmodifiedConfig()

• getSessionBeanFactory

   abstract BeanFactory getSessionBeanFactory()

  The scoped bean factory: for each volatileConfig object, there is a bean factory TODO: session scoped?

• getDisplay

   abstract String getDisplay()

• getPageCache

   abstract ConcurrentExpiringLRUCache<String, WebPage> getPageCache()

• getDocumentCache

   abstract ConcurrentExpiringLRUCache<String, FeaturedDocument> getDocumentCache()

• getGlobalCacheFactory

   abstract GlobalCacheFactory getGlobalCacheFactory()