-
- All Implemented Interfaces:
-
ai.platon.pulsar.session.PulsarSession,java.lang.AutoCloseable
public abstract class AbstractPulsarSession implements PulsarSession
Created by vincent on 18-1-17. Copyright @ 2013-2017 Platon AI. All rights reserved
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description public classAbstractPulsarSession.Companion
-
Field Summary
Fields Modifier and Type Field Description private final ImmutableConfigunmodifiedConfigprivate final BeanFactorysessionBeanFactoryprivate final Stringdisplayprivate final BooleanisActiveprivate final GlobalCacheFactoryglobalCacheFactoryprivate final GlobalCacheglobalCacheprivate final ConcurrentExpiringLRUCache<String, WebPage>pageCacheprivate final ConcurrentExpiringLRUCache<String, FeaturedDocument>documentCacheprivate final AbstractPulsarContextcontextprivate final VolatileConfigsessionConfigprivate final Integerid
-
Constructor Summary
Constructors Constructor Description AbstractPulsarSession(AbstractPulsarContext context, VolatileConfig sessionConfig, Integer id)
-
Method Summary
Modifier and Type Method Description ImmutableConfiggetUnmodifiedConfig()BeanFactorygetSessionBeanFactory()The session scoped bean factory StringgetDisplay()final BooleangetIsActive()GlobalCacheFactorygetGlobalCacheFactory()GlobalCachegetGlobalCache()ConcurrentExpiringLRUCache<String, WebPage>getPageCache()ConcurrentExpiringLRUCache<String, FeaturedDocument>getDocumentCache()AbstractPulsarContextgetContext()The pulsar context VolatileConfiggetSessionConfig()The session scope volatile config, every setting is supposed to be changed at any time and any place IntegergetId()The session id. BooleanregisterClosable(AutoCloseable closable)Close objects when the session is closing UnitdisablePDCache()Disable page cache and document cache LoadOptionsoptions(String args, PageEvent event)Create a new options, with a new volatile config Stringproperty(String name)Get a property Unitproperty(String name, String value)Set a session scope property NormUrlnormalize(String url, String args)Normalize a url NormUrlnormalize(String url, LoadOptions options, Boolean toItemOption)Normalize a url List<NormUrl>normalize(Iterable<String> urls, LoadOptions options, Boolean toItemOption)Normalize urls NormUrlnormalize(UrlAware url, LoadOptions options, Boolean toItemOption)Normalize a url List<NormUrl>normalize(Collection<UrlAware> urls, LoadOptions options, Boolean toItemOption)Normalize urls NormUrlnormalizeOrNull(String url, LoadOptions options, Boolean toItemOption)Normalize a url NormUrlnormalizeOrNull(UrlAware url, LoadOptions options, Boolean toItemOption)Normalize a url WebPageinject(String url)Inject a url WebPageget(String url)Get a page from database if exists WebPagegetOrNull(String url)Get a page from database if exists Booleanexists(String url)Check if a page exists in the database CheckStatefetchState(WebPage page, LoadOptions options)Return the fetch state of the page WebPageopen(String url)Open a page with url WebPageload(String url, String args)Load a url with specified options WebPageload(String url, LoadOptions options)Load a url with specified options WebPageload(UrlAware url, String args)Load a url with specified options WebPageload(UrlAware url, LoadOptions options)Load a url with options. WebPageload(NormUrl normUrl)Load a url in normalized form. WebPageloadDeferred(String url, LoadOptions options)Load a url with specified options. WebPageloadDeferred(UrlAware url, String args)Load a url with specified options. WebPageloadDeferred(UrlAware url, LoadOptions options)WebPageloadDeferred(NormUrl normUrl)Load a url with specified optionsThis function is a kotlin suspend function, which could be started, paused, and resume. List<WebPage>loadAll(Iterable<String> urls, LoadOptions options)Load all urls with specified options, this may cause a parallel fetching if required List<WebPage>loadAll(Iterable<NormUrl> normUrls)Load all urls with specified options CompletableFuture<WebPage>loadAsync(NormUrl url)Load a url with java async style List<CompletableFuture<WebPage>>loadAllAsync(Iterable<NormUrl> urls)Load all urls with specified options with java async style AbstractPulsarSessionsubmit(UrlAware url)Submit a url to the url pool, the url will be processed in the main crawl loop later AbstractPulsarSessionsubmitAll(Iterable<UrlAware> urls)Submit the urls to the url pool, the submitted urls will be processed in the main crawl loop later List<WebPage>loadOutPages(String portalUrl, String args)Load all out pages in a portal page List<WebPage>loadOutPages(String portalUrl, LoadOptions options)Load out pages from a portal page AbstractPulsarSessionsubmitOutPages(String portalUrl, String args)Submit the urls of out pages in the portal page, the submitted urls will be processed in the main crawl loop later AbstractPulsarSessionsubmitOutPages(String portalUrl, LoadOptions options)Submit the urls of out pages in the portal page, the submitted urls will be processed in the main crawl loop later List<CompletableFuture<WebPage>>loadOutPagesAsync(String portalUrl, LoadOptions options)Load out pages linked from the portal page WebPageloadResource(String url, String referer, String args)Load a url as a resource without browser rendering in the browser contextThis function is a kotlin suspend function, which could be started, paused, and resume. WebPageloadResource(String url, String referer, LoadOptions opts)Load a url as a resource without browser rendering in the browser contextThis function is a kotlin suspend function, which could be started, paused, and resume. FeaturedDocumentparse(WebPage page, Boolean noCache)Parse the Web page into DOM. FeaturedDocumentloadDocument(String url, String args)Load or fetch a webpage and parse it into a document FeaturedDocumentloadDocument(String url, LoadOptions options)Load or fetch a webpage and parse it into a document FeaturedDocumentloadDocument(NormUrl normUrl)Load or fetch a webpage and parse it into a document Map<String, String>scrape(String url, String args, Iterable<String> fieldSelectors)Scrape a webpage Map<String, String>scrape(String url, LoadOptions options, Iterable<String> fieldSelectors)Scrape a webpage Map<String, String>scrape(String url, String args, Map<String, String> fieldSelectors)Scrape a webpage Map<String, String>scrape(String url, LoadOptions options, Map<String, String> fieldSelectors)Scrape a webpage List<Map<String, String>>scrape(String url, String args, String restrictSelector, Iterable<String> fieldSelectors)Scrape a webpage List<Map<String, String>>scrape(String url, LoadOptions options, String restrictSelector, Iterable<String> fieldSelectors)Scrape a webpage List<Map<String, String>>scrape(String url, String args, String restrictSelector, Map<String, String> fieldSelectors)Scrape a webpage List<Map<String, String>>scrape(String url, LoadOptions options, String restrictSelector, Map<String, String> fieldSelectors)Scrape a webpage List<Map<String, String>>scrapeOutPages(String portalUrl, String args, Iterable<String> fieldSelectors)Scrape out pages using given selectors. List<Map<String, String>>scrapeOutPages(String portalUrl, LoadOptions options, Iterable<String> fieldSelectors)Scrape out pages using given selectors. List<Map<String, String>>scrapeOutPages(String portalUrl, String args, String restrictSelector, Iterable<String> fieldSelectors)Scrape out pages using given selectors. List<Map<String, String>>scrapeOutPages(String portalUrl, LoadOptions options, String restrictSelector, Iterable<String> fieldSelectors)Scrape out pages using given selectors. List<Map<String, String>>scrapeOutPages(String portalUrl, String args, Map<String, String> fieldSelectors)Scrape out pages using given selectors. List<Map<String, String>>scrapeOutPages(String portalUrl, LoadOptions options, Map<String, String> fieldSelectors)Scrape out pages using given selectors. List<Map<String, String>>scrapeOutPages(String portalUrl, String args, String restrictSelector, Map<String, String> fieldSelectors)Scrape out pages using given selectors. List<Map<String, String>>scrapeOutPages(String portalUrl, LoadOptions options, String restrictSelector, Map<String, String> fieldSelectors)Scrape out pages using given selectors. ObjectgetVariable(String name)Get a variable associated with this session UnitsetVariable(String name, Object value)Set a variable associated with this session UnitputSessionBean(Object obj)Put session scope bean final <T extends Any> TgetSessionBean()Unitdelete(String url)Delete a webpage from the backend storage Unitflush()Flush to the storage Booleanpersist(WebPage page)Persist to the storage Pathexport(WebPage page, String ident)Export the content of a webpage. Pathexport(FeaturedDocument doc, String ident)Export the outer HTML of the document. PathexportTo(FeaturedDocument doc, Path path)Export the whole HTML of the document to the given path. Booleanequals(Object other)IntegerhashCode()StringtoString()Unitclose()-
-
Constructor Detail
-
AbstractPulsarSession
AbstractPulsarSession(AbstractPulsarContext context, VolatileConfig sessionConfig, Integer id)
-
-
Method Detail
-
getUnmodifiedConfig
ImmutableConfig getUnmodifiedConfig()
-
getSessionBeanFactory
BeanFactory getSessionBeanFactory()
The session scoped bean factory
-
getDisplay
String getDisplay()
-
getIsActive
final Boolean getIsActive()
-
getGlobalCacheFactory
GlobalCacheFactory getGlobalCacheFactory()
-
getGlobalCache
GlobalCache getGlobalCache()
-
getPageCache
ConcurrentExpiringLRUCache<String, WebPage> getPageCache()
-
getDocumentCache
ConcurrentExpiringLRUCache<String, FeaturedDocument> getDocumentCache()
-
getContext
AbstractPulsarContext getContext()
The pulsar context
-
getSessionConfig
VolatileConfig getSessionConfig()
The session scope volatile config, every setting is supposed to be changed at any time and any place
-
getId
Integer getId()
The session id. Session id is expected to be set by the container, e.g. the h2 database runtime
-
registerClosable
Boolean registerClosable(AutoCloseable closable)
Close objects when the session is closing
-
disablePDCache
Unit disablePDCache()
Disable page cache and document cache
-
options
LoadOptions options(String args, PageEvent event)
Create a new options, with a new volatile config
-
normalize
NormUrl normalize(String url, LoadOptions options, Boolean toItemOption)
Normalize a url
-
normalize
List<NormUrl> normalize(Iterable<String> urls, LoadOptions options, Boolean toItemOption)
Normalize urls
-
normalize
NormUrl normalize(UrlAware url, LoadOptions options, Boolean toItemOption)
Normalize a url
-
normalize
List<NormUrl> normalize(Collection<UrlAware> urls, LoadOptions options, Boolean toItemOption)
Normalize urls
-
normalizeOrNull
NormUrl normalizeOrNull(String url, LoadOptions options, Boolean toItemOption)
Normalize a url
-
normalizeOrNull
NormUrl normalizeOrNull(UrlAware url, LoadOptions options, Boolean toItemOption)
Normalize a url
-
getOrNull
WebPage getOrNull(String url)
Get a page from database if exists
- Parameters:
url- The url
-
fetchState
CheckState fetchState(WebPage page, LoadOptions options)
Return the fetch state of the page
- Parameters:
page- The webpageoptions- The load options
-
load
WebPage load(String url, String args)
Load a url with specified options
- Parameters:
url- The url to loadargs- The load args
-
load
WebPage load(String url, LoadOptions options)
Load a url with specified options
- Parameters:
url- The url to loadoptions- The load options
-
load
WebPage load(UrlAware url, String args)
Load a url with specified options
- Parameters:
url- The url to loadargs- The load args
-
load
WebPage load(UrlAware url, LoadOptions options)
Load a url with options.
- Parameters:
url- The url to loadoptions- The load options
-
load
WebPage load(NormUrl normUrl)
Load a url in normalized form.
- Parameters:
normUrl- The normalized url
-
loadDeferred
WebPage loadDeferred(String url, LoadOptions options)
Load a url with specified options.
This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.
- Parameters:
url- The url to loadoptions- The load options
-
loadDeferred
WebPage loadDeferred(UrlAware url, String args)
Load a url with specified options.
This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.
- Parameters:
url- The url to loadargs- The load args
-
loadDeferred
WebPage loadDeferred(UrlAware url, LoadOptions options)
- Parameters:
url- The url to loadoptions- The load options
-
loadDeferred
WebPage loadDeferred(NormUrl normUrl)
Load a url with specified options
This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.
- Parameters:
normUrl- The normalized url
-
loadAll
List<WebPage> loadAll(Iterable<String> urls, LoadOptions options)
Load all urls with specified options, this may cause a parallel fetching if required
- Parameters:
urls- The urls to loadoptions- The load options for all urls
-
loadAll
List<WebPage> loadAll(Iterable<NormUrl> normUrls)
Load all urls with specified options
- Parameters:
normUrls- The urls to load
-
loadAsync
CompletableFuture<WebPage> loadAsync(NormUrl url)
Load a url with java async style
- Parameters:
url- The url to load
-
loadAllAsync
List<CompletableFuture<WebPage>> loadAllAsync(Iterable<NormUrl> urls)
Load all urls with specified options with java async style
- Parameters:
urls- The urls to load
-
submit
AbstractPulsarSession submit(UrlAware url)
Submit a url to the url pool, the url will be processed in the main crawl loop later
- Parameters:
url- The url to submit
-
submitAll
AbstractPulsarSession submitAll(Iterable<UrlAware> urls)
Submit the urls to the url pool, the submitted urls will be processed in the main crawl loop later
- Parameters:
urls- The urls to submit
-
loadOutPages
List<WebPage> loadOutPages(String portalUrl, String args)
Load all out pages in a portal page
- Parameters:
portalUrl- The portal url from where to load pagesargs- The load args
-
loadOutPages
List<WebPage> loadOutPages(String portalUrl, LoadOptions options)
Load out pages from a portal page
- Parameters:
portalUrl- The portal url from where to load pagesoptions- The load options
-
submitOutPages
AbstractPulsarSession submitOutPages(String portalUrl, String args)
Submit the urls of out pages in the portal page, the submitted urls will be processed in the main crawl loop later
- Parameters:
portalUrl- The portal url from where to load pagesargs- The load arguments
-
submitOutPages
AbstractPulsarSession submitOutPages(String portalUrl, LoadOptions options)
Submit the urls of out pages in the portal page, the submitted urls will be processed in the main crawl loop later
- Parameters:
portalUrl- The portal url from where to load pagesoptions- The load options
-
loadOutPagesAsync
List<CompletableFuture<WebPage>> loadOutPagesAsync(String portalUrl, LoadOptions options)
Load out pages linked from the portal page
- Parameters:
portalUrl- The portal url from where to load pagesoptions- The load options
-
loadResource
WebPage loadResource(String url, String referer, String args)
Load a url as a resource without browser rendering in the browser context
This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.
- Parameters:
url- The url to loadargs- The load arguments
-
loadResource
WebPage loadResource(String url, String referer, LoadOptions opts)
Load a url as a resource without browser rendering in the browser context
This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.
- Parameters:
url- The url to loadopts- The load options
-
parse
FeaturedDocument parse(WebPage page, Boolean noCache)
Parse the Web page into DOM. If the Web page is not changed since last parse, use the last result if available
-
loadDocument
FeaturedDocument loadDocument(String url, String args)
Load or fetch a webpage and parse it into a document
-
loadDocument
FeaturedDocument loadDocument(String url, LoadOptions options)
Load or fetch a webpage and parse it into a document
-
loadDocument
FeaturedDocument loadDocument(NormUrl normUrl)
Load or fetch a webpage and parse it into a document
-
scrape
Map<String, String> scrape(String url, String args, Iterable<String> fieldSelectors)
Scrape a webpage
-
scrape
Map<String, String> scrape(String url, LoadOptions options, Iterable<String> fieldSelectors)
Scrape a webpage
-
scrape
Map<String, String> scrape(String url, String args, Map<String, String> fieldSelectors)
Scrape a webpage
-
scrape
Map<String, String> scrape(String url, LoadOptions options, Map<String, String> fieldSelectors)
Scrape a webpage
-
scrape
List<Map<String, String>> scrape(String url, String args, String restrictSelector, Iterable<String> fieldSelectors)
Scrape a webpage
-
scrape
List<Map<String, String>> scrape(String url, LoadOptions options, String restrictSelector, Iterable<String> fieldSelectors)
Scrape a webpage
-
scrape
List<Map<String, String>> scrape(String url, String args, String restrictSelector, Map<String, String> fieldSelectors)
Scrape a webpage
-
scrape
List<Map<String, String>> scrape(String url, LoadOptions options, String restrictSelector, Map<String, String> fieldSelectors)
Scrape a webpage
-
scrapeOutPages
@ExperimentalApi() List<Map<String, String>> scrapeOutPages(String portalUrl, String args, Iterable<String> fieldSelectors)
Scrape out pages using given selectors.
- Parameters:
portalUrl- The portal url the scraping start fromargs- The load argumentsfieldSelectors- The CSS selectors to extract fields from out pages
-
scrapeOutPages
@ExperimentalApi() List<Map<String, String>> scrapeOutPages(String portalUrl, LoadOptions options, Iterable<String> fieldSelectors)
Scrape out pages using given selectors.
- Parameters:
portalUrl- The portal url the scraping start fromoptions- The load optionsfieldSelectors- The CSS selectors to extract fields from out pages
-
scrapeOutPages
@ExperimentalApi() List<Map<String, String>> scrapeOutPages(String portalUrl, String args, String restrictSelector, Iterable<String> fieldSelectors)
Scrape out pages using given selectors.
- Parameters:
portalUrl- The portal url the scraping start fromargs- The load argumentsrestrictSelector- The selector used to restrict all fields to be inside the DOMfieldSelectors- The CSS selectors to extract fields from out pages
-
scrapeOutPages
@ExperimentalApi() List<Map<String, String>> scrapeOutPages(String portalUrl, LoadOptions options, String restrictSelector, Iterable<String> fieldSelectors)
Scrape out pages using given selectors.
- Parameters:
portalUrl- The portal url the scraping start fromoptions- The load optionsrestrictSelector- The selector used to restrict all fields to be inside the DOMfieldSelectors- The CSS selectors to extract fields from out pages
-
scrapeOutPages
@ExperimentalApi() List<Map<String, String>> scrapeOutPages(String portalUrl, String args, Map<String, String> fieldSelectors)
Scrape out pages using given selectors.
- Parameters:
portalUrl- The portal url the scraping start fromargs- The load argumentsfieldSelectors- The CSS selectors to extract fields from out pages
-
scrapeOutPages
@ExperimentalApi() List<Map<String, String>> scrapeOutPages(String portalUrl, LoadOptions options, Map<String, String> fieldSelectors)
Scrape out pages using given selectors.
- Parameters:
portalUrl- The portal url the scraping start fromoptions- The load optionsfieldSelectors- The CSS selectors to extract fields from out pages
-
scrapeOutPages
@ExperimentalApi() List<Map<String, String>> scrapeOutPages(String portalUrl, String args, String restrictSelector, Map<String, String> fieldSelectors)
Scrape out pages using given selectors.
- Parameters:
portalUrl- The portal url the scraping start fromargs- The load argumentsrestrictSelector- The selector used to restrict all fields to be inside the DOMfieldSelectors- The CSS selectors to extract fields from out pages
-
scrapeOutPages
@ExperimentalApi() List<Map<String, String>> scrapeOutPages(String portalUrl, LoadOptions options, String restrictSelector, Map<String, String> fieldSelectors)
Scrape out pages using given selectors.
- Parameters:
portalUrl- The portal url the scraping start fromoptions- The load optionsrestrictSelector- The selector used to restrict all fields to be inside the DOMfieldSelectors- The CSS selectors to extract fields from out pages
-
getVariable
Object getVariable(String name)
Get a variable associated with this session
-
setVariable
Unit setVariable(String name, Object value)
Set a variable associated with this session
-
putSessionBean
Unit putSessionBean(Object obj)
Put session scope bean
-
getSessionBean
final <T extends Any> T getSessionBean()
-
export
Path export(WebPage page, String ident)
Export the content of a webpage.
- Parameters:
page- Page to exportident- File name identifier used to distinguish from other names
-
export
Path export(FeaturedDocument doc, String ident)
Export the outer HTML of the document.
- Parameters:
doc- Document to exportident- File name identifier used to distinguish from other names
-
exportTo
Path exportTo(FeaturedDocument doc, Path path)
Export the whole HTML of the document to the given path.
- Parameters:
doc- Document to exportpath- Path to save the exported content
-
-
-
-