Package 

Interface PulsarSession

  • All Implemented Interfaces:
    java.lang.AutoCloseable 
    
public interface PulsarSession
 implements AutoCloseable

    PulsarSession defines a concise interface to fetch webpages, from the local storage or the Internet, and methods to parse, extract, persist, index, export them.

    Key methods:

    PulsarSession.load, load a webpage from local storage, or fetch it from the Internet. PulsarSession.parse, parse a webpage into a document. PulsarSession.scrape, load a webpage, parse it into a document and then extract fields from the document.

    And also the batch versions:

    PulsarSession.loadOutPages, load the portal page and out pages. PulsarSession.scrapeOutPages, load the portal page and out pages, extract fields from out pages.

    The first method to keep in mind is how to load a page.

    A load method checks the local storage first, if it exists and is good, return the persisted version, otherwise, fetch it from the Internet.

    Other fetch condition can be specified in load arguments, or load options:

    • expiration

    • page size requirements

    • field requirements

    Once a webpage is loaded from local storage, or fetched from the Internet, we come to the next process steps:

    • parse the web content into a HTML document

    • extract fields from the HTML document

    • write the fields into a destination, such as

    There are many ways to fetch the content of a page from the Internet:

    • http protocol

    • through a real browser

    Since the webpages are becoming more and more complex, fetching webpages through real browsers is the primer way nowadays.

    When we fetch webpages using a real browser, we need to interact with pages to ensure the required fields are loaded correctly and completely. Enable PageEvent and use WebDriver to archive such purpose.

    val options = session.options(args)
options.event.browseEvent.onDidDOMStateCheck.addLast { page, driver ->
  driver.scrollDown()
}
session.load(url, options)

    Pulsar WebDriver provides a complete method set for RPA, just like selenium, playwright and puppeteer, all actions and behaviors are optimized to mimic real people as closely as possible.

    • Method Detail

      • inject

         abstract WebPage inject(String url)

        Inject a url to fetch later

        Parameters:
        url - The url followed by options

      • get

         abstract WebPage get(String url)

        Get a page from database if exists

        Parameters:
        url - The url

      • getOrNull

         abstract WebPage getOrNull(String url)

        Get a page from database if exists

        Parameters:
        url - The url

      • exists

         abstract Boolean exists(String url)

        Check if a page exists in the database

        Parameters:
        url - The url

      • fetchState

         abstract CheckState fetchState(WebPage page, LoadOptions options)

        Return the fetch state of the page

        Parameters:
        page - The webpage
        options - The load options

      • open

         abstract WebPage open(String url)

        Open a url.

        This method opens the url immediately, regardless of the previous state of the page.

        Parameters:
        url - The url to open

      • load

         abstract WebPage load(String url, String args)

        Load a url with arguments.

        This method checks the local storage first, if it exists and is good, return the persisted version, otherwise, fetch it from the Internet.

        Other fetch condition can be specified in load arguments:

        • expiration

        • page size requirement

        • fields requirement

        Parameters:
        url - The url to load
        args - The load arguments

      • load

         abstract WebPage load(String url, LoadOptions options)

        Load a url with options.

        This method checks the local storage first, if it exists and is good, return the persisted version, otherwise, fetch it from the Internet.

        Other fetch condition can be specified in load arguments:

        • expiration

        • page size requirement

        • fields requirement

        Parameters:
        url - The url to load
        options - The load options

      • load

         abstract WebPage load(UrlAware url, String args)

        Load a url with specified options

        Parameters:
        url - The url to load
        args - The load args

      • load

         abstract WebPage load(UrlAware url, LoadOptions options)

        Load a url with options.

        Parameters:
        url - The url to load
        options - The load options

      • load

         abstract WebPage load(NormUrl normUrl)

        Load a url in normalized form.

        Parameters:
        normUrl - The normalized url

      • loadDeferred

         abstract WebPage loadDeferred(String url, LoadOptions options)

        Load a url with specified options.

        This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.

        Parameters:
        url - The url to load
        options - The load options

      • loadDeferred

         abstract WebPage loadDeferred(UrlAware url, String args)

        Load a url with specified options.

        This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.

        Parameters:
        url - The url to load
        args - The load args

      • loadDeferred

         abstract WebPage loadDeferred(UrlAware url, LoadOptions options)

        Load a url with specified options.

        This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.

        Parameters:
        url - The url to load
        options - The load options

      • loadDeferred

         abstract WebPage loadDeferred(NormUrl normUrl)

        Load a url with specified options

        This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.

        Parameters:
        normUrl - The normalized url

      • loadAll

         abstract List<WebPage> loadAll(Iterable<String> urls, LoadOptions options)

        Load all urls with specified options, this causes a parallel fetching whenever applicable

        Parameters:
        urls - The urls to load
        options - The load options

      • loadAll

         abstract List<WebPage> loadAll(Iterable<NormUrl> normUrls)

        Load all urls with specified options

        Parameters:
        normUrls - The urls to load

      • submit

         abstract PulsarSession submit(UrlAware url)

        Submit a url to the url pool, the url will be processed in the main crawl loop later

        Parameters:
        url - The url to submit

      • submitAll

         abstract PulsarSession submitAll(Iterable<UrlAware> urls)

        Submit the urls to the url pool, the submitted urls will be processed in the main crawl loop later

        Parameters:
        urls - The urls to submit

      • loadOutPages

         abstract List<WebPage> loadOutPages(String portalUrl, String args)

        Load out pages linked from the portal page

        Parameters:
        portalUrl - The portal url from where to load pages
        args - The load args

      • loadOutPages

         abstract List<WebPage> loadOutPages(String portalUrl, LoadOptions options)

        Load out pages linked from the portal page

        Parameters:
        portalUrl - The portal url from where to load pages
        options - The load options

      • submitOutPages

         abstract AbstractPulsarSession submitOutPages(String portalUrl, String args)

        Submit the urls of out pages in the portal page, the submitted urls will be processed in the main crawl loop later

        Parameters:
        portalUrl - The portal url from where to load pages
        args - The load arguments

      • submitOutPages

         abstract PulsarSession submitOutPages(String portalUrl, LoadOptions options)

        Submit the urls of out pages in the portal page, the submitted urls will be processed in the main crawl loop later

        Parameters:
        portalUrl - The portal url from where to load pages
        options - The load options

      • loadResource

         abstract WebPage loadResource(String url, String referer, String args)

        Load a url as a resource without browser rendering in the browser context

        This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.

        Parameters:
        url - The url to load
        args - The load arguments

      • loadResource

         abstract WebPage loadResource(String url, String referer, LoadOptions opts)

        Load a url as a resource without browser rendering in the browser context

        This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.

        Parameters:
        url - The url to load
        opts - The load options

      • parse

         abstract FeaturedDocument parse(WebPage page, Boolean noCache)

        Parse the Web page into DOM. If the Web page is not changed since last parse, use the last result if available

      • loadDocument

         abstract FeaturedDocument loadDocument(String url, String args)

        Load or fetch a webpage and parse it into a document

      • loadDocument

         abstract FeaturedDocument loadDocument(NormUrl normUrl)

        Load or fetch a webpage and parse it into a document

      • scrapeOutPages

        @ExperimentalApi() abstract List<Map<String, String>> scrapeOutPages(String portalUrl, String args, Iterable<String> fieldSelectors)

        Scrape out pages using given selectors.

        Parameters:
        portalUrl - The portal url the scraping start from
        args - The load arguments
        fieldSelectors - The CSS selectors to extract fields from out pages

      • scrapeOutPages

        @ExperimentalApi() abstract List<Map<String, String>> scrapeOutPages(String portalUrl, LoadOptions options, Iterable<String> fieldSelectors)

        Scrape out pages using given selectors.

        Parameters:
        portalUrl - The portal url the scraping start from
        options - The load options
        fieldSelectors - The CSS selectors to extract fields from out pages

      • scrapeOutPages

        @ExperimentalApi() abstract List<Map<String, String>> scrapeOutPages(String portalUrl, String args, String restrictSelector, Iterable<String> fieldSelectors)

        Scrape out pages using given selectors.

        Parameters:
        portalUrl - The portal url the scraping start from
        args - The load arguments
        restrictSelector - The selector used to restrict all fields to be inside the DOM
        fieldSelectors - The CSS selectors to extract fields from out pages

      • scrapeOutPages

        @ExperimentalApi() abstract List<Map<String, String>> scrapeOutPages(String portalUrl, LoadOptions options, String restrictSelector, Iterable<String> fieldSelectors)

        Scrape out pages using given selectors.

        Parameters:
        portalUrl - The portal url the scraping start from
        options - The load options
        restrictSelector - The selector used to restrict all fields to be inside the DOM
        fieldSelectors - The CSS selectors to extract fields from out pages

      • scrapeOutPages

        @ExperimentalApi() abstract List<Map<String, String>> scrapeOutPages(String portalUrl, String args, Map<String, String> fieldSelectors)

        Scrape out pages using given selectors.

        Parameters:
        portalUrl - The portal url the scraping start from
        args - The load arguments
        fieldSelectors - The CSS selectors to extract fields from out pages

      • scrapeOutPages

        @ExperimentalApi() abstract List<Map<String, String>> scrapeOutPages(String portalUrl, LoadOptions options, Map<String, String> fieldSelectors)

        Scrape out pages using given selectors.

        Parameters:
        portalUrl - The portal url the scraping start from
        options - The load options
        fieldSelectors - The CSS selectors to extract fields from out pages

      • scrapeOutPages

        @ExperimentalApi() abstract List<Map<String, String>> scrapeOutPages(String portalUrl, String args, String restrictSelector, Map<String, String> fieldSelectors)

        Scrape out pages using given selectors.

        Parameters:
        portalUrl - The portal url the scraping start from
        args - The load arguments
        restrictSelector - The selector used to restrict all fields to be inside the DOM
        fieldSelectors - The CSS selectors to extract fields from out pages

      • scrapeOutPages

        @ExperimentalApi() abstract List<Map<String, String>> scrapeOutPages(String portalUrl, LoadOptions options, String restrictSelector, Map<String, String> fieldSelectors)

        Scrape out pages using given selectors.

        Parameters:
        portalUrl - The portal url the scraping start from
        options - The load options
        restrictSelector - The selector used to restrict all fields to be inside the DOM
        fieldSelectors - The CSS selectors to extract fields from out pages

      • delete

         abstract Unit delete(String url)

        Delete a webpage from the backend storage

      • flush

         abstract Unit flush()

        Flush to the storage

      • persist

         abstract Boolean persist(WebPage page)

        Persist to the storage

      • export

         abstract Path export(WebPage page, String ident)

        Export the content of a webpage.

        Parameters:
        page - Page to export
        ident - File name identifier used to distinguish from other names

      • export

         abstract Path export(FeaturedDocument doc, String ident)

        Export the outer HTML of the document.

        Parameters:
        doc - Document to export
        ident - File name identifier used to distinguish from other names

      • exportTo

         abstract Path exportTo(FeaturedDocument doc, Path path)

        Export the whole HTML of the document to the given path.

        Parameters:
        doc - Document to export
        path - Path to save the exported content

      • getId

         abstract Integer getId()

        The session id. Session id is expected to be set by the container, e.g. the h2 database runtime

      • getSessionConfig

         abstract VolatileConfig getSessionConfig()

        The session scope volatile config, every setting is supposed to be changed at any time and any place

      • getSessionBeanFactory

         abstract BeanFactory getSessionBeanFactory()

        The scoped bean factory: for each volatileConfig object, there is a bean factory

      • getPageCache

         abstract ConcurrentExpiringLRUCache<String, WebPage> getPageCache()

        The global page cache

      • getDocumentCache

         abstract ConcurrentExpiringLRUCache<String, FeaturedDocument> getDocumentCache()

        The global document cache