Package ai.platon.pulsar.session

Interface PulsarSession

  • All Implemented Interfaces:
    java.lang.AutoCloseable 
    
public interface PulsarSession
 implements AutoCloseable

    PulsarSession defines an interface to load webpages from local storage or fetch from the Internet, as well as methods for parsing, extracting, saving and exporting webpages.

    Key methods:

    • load: load a webpage from local storage, or fetch it from the Internet.

    • parse: parse a webpage into a document.

    • scrape: load a webpage, parse it into a document and then extract fields from the document.

    • submit: submit a url to the URL pool, the url will be processed in the main loop later.

    And also the batch versions:

    • loadOutPages: load the portal page and out pages.

    • scrapeOutPages: load the portal page and out pages, extract fields from out pages.

    The first thing to understand is how to load a page. Load methods like load first check the local storage and return the local version if the required page exists and meets the requirements, otherwise it will be fetched from the Internet.

    Load parameters or load options can be used to specify when the system will fetch a webpage from the Internet:

    • Expiration

    • Force refresh

    • Page size

    • Required fields

    • Other conditions

    Once a page is loaded from local storage, or fetched from the Internet, we come to the next process steps:

    • parse the page content into an HTML document

    • extract fields from the HTML document

    • write the extraction results to a destination, such as

    There are many ways to fetch the content of a page from the Internet:

    • through HTTP protocol

    • through a real browser

    Since the webpages are becoming more and more complex, fetching webpages through real browsers is the primary way nowadays.

    When we fetch webpages using a real browser, we need to interact with pages to ensure the required fields are loaded correctly and completely. Enable PageEvent and use WebDriver to archive such purpose.

    val options = session.options(args)
options.event.browseEvent.onDidDOMStateCheck.addLast { page, driver ->
  driver.scrollDown()
}
session.load(url, options)

    WebDriver provides a complete method set for RPA, just like selenium, playwright and puppeteer does, all actions and behaviors are optimized to mimic real people as closely as possible.

    • Constructor Detail

    • Method Detail

      • normalize

         abstract List<NormUrl> normalize(Collection<UrlAware> urls, String args, Boolean toItemOption)

        Normalize urls, remove invalid ones

        Parameters:
        urls - The urls to normalize
        args - The arguments
        toItemOption - If the LoadOptions is converted to item load options

      • normalize

         abstract List<NormUrl> normalize(Collection<UrlAware> urls, LoadOptions options, Boolean toItemOption)

        Normalize urls, remove invalid ones

        Parameters:
        urls - The urls to normalize
        options - The LoadOptions applied to each url
        toItemOption - If the LoadOptions is converted to item load options

      • inject

         abstract WebPage inject(String url)

        Inject a url as a seed to fetch. Injection is usually used in Nutch style crawls where the execution flow likes the following:

        inject -> generate -> fetch -> parse -> index -> update ^ ^ | <- <- <- <- |

        Parameters:
        url - The url to inject, con be followed by arguments

      • get

         abstract WebPage get(String url)

        Get a page from storage.

        Parameters:
        url - The url

      • get

         abstract WebPage get(String url, String fields)

        Get a page from storage.

        Parameters:
        url - The url
        fields - The fields to load from local storage

      • getOrNull

         abstract WebPage getOrNull(String url)

        Get a page from storage.

        Parameters:
        url - The url

      • getOrNull

         abstract WebPage getOrNull(String url, String fields)

        Get a page from storage.

        Parameters:
        url - The url
        fields - The fields to load from local storage

      • getContent

         abstract ByteBuffer getContent(String url)

        Get the content of the page from the storage

        Parameters:
        url - The url of the page to retrieve

      • getContentAsString

        @Beta() abstract String getContentAsString(String url)

        Get the content of the page from the storage

        Parameters:
        url - The url of the page to retrieve

      • exists

         abstract Boolean exists(String url)

        Check if the page exists in the storage.

        Parameters:
        url - The url to check

      • fetchState

         abstract CheckState fetchState(WebPage page, LoadOptions options)

        Return the fetch state of the page.

        Parameters:
        page - The webpage
        options - The load options

      • open

         abstract WebPage open(String url)

        Open a url.

        This method opens the url immediately, regardless of the previous state of the page.

        Parameters:
        url - The url to open

      • load

         abstract WebPage load(String url)

        Load a url with specified arguments.

        This method first checks the url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

        Other fetch conditions can be specified by load arguments:

        • expiration

        • page size requirement

        • fields requirement

        • other

        Parameters:
        url - The url to load

      • load

         abstract WebPage load(String url, String args)

        Load a url with specified arguments.

        This method first checks the url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

        Other fetch conditions can be specified by load arguments:

        • expiration

        • page size requirement

        • fields requirement

        • other

        Parameters:
        url - The url to load
        args - The load arguments

      • load

         abstract WebPage load(String url, LoadOptions options)

        Load a url with specified options.

        This method first checks the url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

        Other fetch conditions can be specified by load arguments:

        • expiration

        • page size requirement

        • fields requirement

        • other

        Parameters:
        url - The url to load
        options - The load options

      • load

         abstract WebPage load(UrlAware url)

        Load a url with the specified arguments.

        This method first checks the url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

        Parameters:
        url - The url to load

      • load

         abstract WebPage load(UrlAware url, String args)

        Load a url with the specified arguments.

        This method first checks the url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

        Parameters:
        url - The url to load
        args - The load arguments

      • load

         abstract WebPage load(UrlAware url, LoadOptions options)

        Load a url with options.

        This method first checks the url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

        Parameters:
        url - The url to load
        options - The load options

      • load

         abstract WebPage load(NormUrl url)

        Load a normal url.

        This method first checks the url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

        Parameters:
        url - The normal url

      • loadDeferred

         abstract WebPage loadDeferred(String url, String args)

        Load a url with specified options.

        This method first checks the url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

        This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.

        Parameters:
        url - The url to load
        args - The load options

      • loadDeferred

         abstract WebPage loadDeferred(String url, LoadOptions options)

        Load a url with specified options.

        This method first checks the url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

        This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.

        Parameters:
        url - The url to load
        options - The load options

      • loadDeferred

         abstract WebPage loadDeferred(UrlAware url, String args)

        Load a url with specified arguments.

        This method first checks the url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

        This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.

        Parameters:
        url - The url to load
        args - The load args

      • loadDeferred

         abstract WebPage loadDeferred(UrlAware url, LoadOptions options)

        Load a url with specified options.

        This method first checks the url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

        This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.

        Parameters:
        url - The url to load
        options - The load options

      • loadDeferred

         abstract WebPage loadDeferred(NormUrl url)

        Load a url with specified options

        This method first checks the url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

        This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.

        Parameters:
        url - The normal url

      • loadAll

         abstract List<WebPage> loadAll(Iterable<String> urls)

        Load all urls with specified options

        This method first checks each url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

        Parameters:
        urls - The urls to load

      • loadAll

         abstract List<WebPage> loadAll(Iterable<String> urls, String args)

        Load all urls with specified options

        This method first checks each url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

        Parameters:
        urls - The urls to load
        args - The load arguments

      • loadAll

         abstract List<WebPage> loadAll(Iterable<String> urls, LoadOptions options)

        Load all urls with specified options

        This method first checks each url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

        Parameters:
        urls - The urls to load
        options - The load options

      • loadAll

         abstract List<WebPage> loadAll(Collection<UrlAware> urls)

        Load all urls with specified options

        This method first checks each url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

        Parameters:
        urls - The urls to load

      • loadAll

         abstract List<WebPage> loadAll(Collection<UrlAware> urls, String args)

        Load all urls with specified options

        This method first checks each url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

        Parameters:
        urls - The urls to load
        args - The load arguments

      • loadAll

         abstract List<WebPage> loadAll(Collection<UrlAware> urls, LoadOptions options)

        Load all urls with specified options

        This method first checks each url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

        Parameters:
        urls - The urls to load
        options - The load options

      • loadAll

         abstract List<WebPage> loadAll(List<NormUrl> normUrls)

        Load all normal urls with specified options

        This method first checks each url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

        Parameters:
        normUrls - The normal urls to load

      • loadAsync

         abstract CompletableFuture<WebPage> loadAsync(UrlAware url)

        Load a normal url in java async style

        Parameters:
        url - The url to load

      • loadAllAsync

         abstract List<CompletableFuture<WebPage>> loadAllAsync(Iterable<String> urls)

        Load all normal urls in java async style

        This method first checks each url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

        Parameters:
        urls - The normal urls to load

      • loadAllAsync

         abstract List<CompletableFuture<WebPage>> loadAllAsync(Iterable<String> urls, String args)

        Load all normal urls in java async style.

        This method first checks each url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

        Parameters:
        urls - The normal urls to load

      • loadAllAsync

         abstract List<CompletableFuture<WebPage>> loadAllAsync(Iterable<String> urls, LoadOptions options)

        Load all normal urls in java async style.

        This method first checks each url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

        Parameters:
        urls - The normal urls to load

      • loadAllAsync

         abstract List<CompletableFuture<WebPage>> loadAllAsync(Collection<UrlAware> urls)

        Load all normal urls in java async style.

        This method first checks each url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

        Parameters:
        urls - The normal urls to load

      • loadAllAsync

         abstract List<CompletableFuture<WebPage>> loadAllAsync(Collection<UrlAware> urls, String args)

        Load all normal urls in java async style.

        This method first checks each url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

        Parameters:
        urls - The normal urls to load

      • loadAllAsync

         abstract List<CompletableFuture<WebPage>> loadAllAsync(Collection<UrlAware> urls, LoadOptions options)

        Load all normal urls in java async style.

        This method first checks each url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

        Parameters:
        urls - The normal urls to load

      • loadAllAsync

         abstract List<CompletableFuture<WebPage>> loadAllAsync(List<NormUrl> urls)

        Load all normal urls in java async style

        This method first checks each url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

        Parameters:
        urls - The normal urls to load

      • submit

         abstract PulsarSession submit(String url)

        Submit a url to the URL pool, the url will be processed in the crawl loop later

        Parameters:
        url - The url to submit

      • submit

         abstract PulsarSession submit(String url, String args)

        Submit a url to the URL pool, and it will be processed in a crawl loop

        Parameters:
        url - The url to submit
        args - The load arguments

      • submit

         abstract PulsarSession submit(String url, LoadOptions options)

        Submit a url to the URL pool, and it will be processed in a crawl loop

        Parameters:
        url - The url to submit
        options - The load options

      • submit

         abstract PulsarSession submit(UrlAware url)

        Submit a url to the URL pool, and it will be processed in a crawl loop

        Parameters:
        url - The url to submit

      • submit

         abstract PulsarSession submit(UrlAware url, String args)

        Submit a url to the URL pool, and it will be processed in a crawl loop

        Parameters:
        url - The url to submit

      • submitAll

         abstract PulsarSession submitAll(Iterable<String> urls)

        Submit the urls to the URL pool, the submitted urls will be processed in a crawl loop

        Parameters:
        urls - The urls to submit

      • submitAll

         abstract PulsarSession submitAll(Iterable<String> urls, String args)

        Submit the urls to the URL pool, the submitted urls will be processed in a crawl loop

        Parameters:
        urls - The urls to submit
        args - The load arguments

      • submitAll

         abstract PulsarSession submitAll(Iterable<String> urls, LoadOptions options)

        Submit the urls to the URL pool, the submitted urls will be processed in a crawl loop

        Parameters:
        urls - The urls to submit
        options - The load options

      • submitAll

         abstract PulsarSession submitAll(Collection<UrlAware> urls)

        Submit the urls to the URL pool, the submitted urls will be processed in a crawl loop

        Parameters:
        urls - The urls to submit

      • submitAll

         abstract PulsarSession submitAll(Collection<UrlAware> urls, String args)

        Submit the urls to the URL pool, the submitted urls will be processed in a crawl loop

        Parameters:
        urls - The urls to submit

      • loadOutPages

         abstract List<WebPage> loadOutPages(String portalUrl, String args)

        Load or fetch the portal page, and then load or fetch the out links selected by -outLink option.

        Option -outLink specifies the cssSelector for links in the portal page to load.

        Parameters:
        portalUrl - The portal url from where to load pages
        args - The load arguments

      • loadOutPages

         abstract List<WebPage> loadOutPages(String portalUrl, LoadOptions options)

        Load or fetch the portal page, and then load or fetch the out links selected by -outLink option.

        Option -outLink specifies the cssSelector for links in the portal page to load.

        Parameters:
        portalUrl - The portal url from where to load pages
        options - The load options

      • loadOutPages

         abstract List<WebPage> loadOutPages(UrlAware portalUrl, String args)

        Load or fetch the portal page, and then load or fetch the out links selected by -outLink option.

        Option -outLink specifies the cssSelector for links in the portal page to load.

        Parameters:
        portalUrl - The portal url from where to load pages
        args - The load arguments

      • loadOutPages

         abstract List<WebPage> loadOutPages(UrlAware portalUrl, LoadOptions options)

        Load or fetch the portal page, and then load or fetch the out links selected by -outLink option.

        Option -outLink specifies the cssSelector for links in the portal page to load.

        Parameters:
        portalUrl - The portal url from where to load pages
        options - The load options

      • loadOutPagesAsync

         abstract List<CompletableFuture<WebPage>> loadOutPagesAsync(String portalUrl, String args)

        Load or fetch the portal page, and then load or fetch the out links selected by -outLink option asynchronously.

        Option -outLink specifies the cssSelector for links in the portal page to load.

        Parameters:
        portalUrl - The portal url from where to load pages
        args - The load arguments

      • loadOutPagesAsync

         abstract List<CompletableFuture<WebPage>> loadOutPagesAsync(String portalUrl, LoadOptions options)

        Load or fetch the portal page, and then load or fetch the out links selected by -outLink option asynchronously.

        Option -outLink specifies the cssSelector for links in the portal page to load.

        Parameters:
        portalUrl - The portal url from where to load pages
        options - The load options

      • submitForOutPages

         abstract PulsarSession submitForOutPages(String portalUrl, String args)

        Load the portal page and submit the out links specified by the -outLink option to the URL pool.

        Option -outLink specifies the cssSelector for links in the portal page to load.

        The submitted urls will be processed in a crawl loop later.

        Parameters:
        portalUrl - The portal url from where to load pages
        args - The load arguments

      • submitForOutPages

         abstract PulsarSession submitForOutPages(String portalUrl, LoadOptions options)

        Load the portal page and submit the out links specified by the -outLink option to the URL pool.

        Option -outLink specifies the cssSelector for links in the portal page to load.

        The submitted urls will be processed in a crawl loop later.

        Parameters:
        portalUrl - The portal url from where to load pages
        options - The load options

      • submitForOutPages

         abstract PulsarSession submitForOutPages(UrlAware portalUrl, String args)

        Load the portal page and submit the out links specified by the -outLink option to the URL pool.

        Option -outLink specifies the cssSelector for links in the portal page to load.

        The submitted urls will be processed in a crawl loop later.

        Parameters:
        portalUrl - The portal url from where to load pages
        args - The load arguments

      • submitForOutPages

         abstract PulsarSession submitForOutPages(UrlAware portalUrl, LoadOptions options)

        Load the portal page and submit the out links specified by the -outLink option to the URL pool.

        Option -outLink specifies the cssSelector for links in the portal page to load.

        The submitted urls will be processed in a crawl loop later.

        Parameters:
        portalUrl - The portal url from where to load pages
        options - The load options

      • loadResource

         abstract WebPage loadResource(String url, String referrer)

        Load a url as a resource without browser rendering.

        Referrer will be opened by a browser first to obtain browsing environment, such as headers and cookies, and then the browsing environment will be applied to the later resource fetching.

        Parameters:
        url - The url to load
        referrer - The referrer URL

      • loadResource

         abstract WebPage loadResource(String url, String referrer, String args)

        Load a url as a resource without browser rendering.

        Referrer will be opened by a browser first to obtain browsing environment, such as headers and cookies, and then the browsing environment will be applied to the later resource fetching.

        Parameters:
        url - The url to load
        referrer - The referrer URL
        args - The load arguments

      • loadResource

         abstract WebPage loadResource(String url, String referrer, LoadOptions options)

        Load a url as a resource without browser rendering.

        Referrer will be opened by a browser first to obtain browsing environment, such as headers and cookies, and then the browsing environment will be applied to the later resource fetching.

        Parameters:
        url - The url to load
        referrer - The referrer URL
        options - The load options

      • loadResourceDeferred

         abstract WebPage loadResourceDeferred(String url, String referrer)

        Load a url as a resource without browser rendering.

        Referrer will be opened by a browser first to obtain browsing environment, such as headers and cookies, and then the browsing environment will be applied to the later resource fetching.

        This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.

        Parameters:
        url - The url to load
        referrer - The referrer URL

      • loadResourceDeferred

         abstract WebPage loadResourceDeferred(String url, String referrer, String args)

        Load a url as a resource without browser rendering.

        Referrer will be opened by a browser first to obtain browsing environment, such as headers and cookies, and then the browsing environment will be applied to the later resource fetching.

        This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.

        Parameters:
        url - The url to load
        referrer - The referrer URL
        args - The load arguments

      • loadResourceDeferred

         abstract WebPage loadResourceDeferred(String url, String referrer, LoadOptions options)

        Load a url as a resource without browser rendering.

        Referrer will be opened by a browser first to obtain browsing environment, such as headers and cookies, and then the browsing environment will be applied to the later resource fetching.

        This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.

        Parameters:
        url - The url to load
        referrer - The referrer URL
        options - The load options

      • parse

         abstract FeaturedDocument parse(WebPage page)

        Parse a webpage into an HTML document.

      • parse

         abstract FeaturedDocument parse(WebPage page, Boolean noCache)

        Parse a webpage into an HTML document.

      • loadDocument

         abstract FeaturedDocument loadDocument(String url)

        Load or fetch a webpage and parse it into an HTML document

      • loadDocument

         abstract FeaturedDocument loadDocument(String url, String args)

        Load or fetch a webpage and parse it into an HTML document

      • loadDocument

         abstract FeaturedDocument loadDocument(String url, LoadOptions options)

        Load or fetch a webpage and parse it into an HTML document

      • loadDocument

         abstract FeaturedDocument loadDocument(UrlAware url)

        Load or fetch a webpage and parse it into an HTML document

      • loadDocument

         abstract FeaturedDocument loadDocument(UrlAware url, String args)

        Load or fetch a webpage and parse it into an HTML document

      • loadDocument

         abstract FeaturedDocument loadDocument(UrlAware url, LoadOptions options)

        Load or fetch a webpage and parse it into an HTML document

      • loadDocument

         abstract FeaturedDocument loadDocument(NormUrl url)

        Load or fetch a webpage and then parse it into an HTML document.

      • scrape

         abstract Map<String, String> scrape(String url, String args, Iterable<String> fieldSelectors)

        Load or fetch a webpage located by the given url, and then extract fields specified by field selectors.

        Parameters:
        url - The url to scrape
        args - The load arguments
        fieldSelectors - The selectors to extract fields

      • scrape

         abstract Map<String, String> scrape(String url, LoadOptions options, Iterable<String> fieldSelectors)

        Load or fetch a webpage located by the given url, and then extract fields specified by field selectors.

        Parameters:
        url - The url to scrape
        options - The load options
        fieldSelectors - The selectors to extract fields

      • scrape

         abstract Map<String, String> scrape(String url, String args, Map<String, String> fieldSelectors)

        Load or fetch a webpage located by the given url, and then extract fields specified by field selectors.

        Parameters:
        url - The url to scrape
        args - The load arguments
        fieldSelectors - The selectors to extract fields

      • scrape

         abstract Map<String, String> scrape(String url, LoadOptions options, Map<String, String> fieldSelectors)

        Load or fetch a webpage located by the given url, and then extract fields specified by field selectors.

        Parameters:
        url - The url to scrape
        options - The load options
        fieldSelectors - The selectors to extract fields

      • scrape

         abstract List<Map<String, String>> scrape(String url, String args, String restrictSelector, Iterable<String> fieldSelectors)

        Load or fetch a webpage located by the given url, and then extract fields specified by field selectors.

        Parameters:
        url - The url to scrape
        args - The load arguments
        restrictSelector - A CSS selector to locate a DOM where all fields are restricted to
        fieldSelectors - The selectors to extract fields

      • scrape

         abstract List<Map<String, String>> scrape(String url, LoadOptions options, String restrictSelector, Iterable<String> fieldSelectors)

        Load or fetch a webpage located by the given url, and then extract fields specified by field selectors.

        Parameters:
        url - The url to scrape
        options - The load options
        restrictSelector - A CSS selector to locate a DOM where all fields are restricted to
        fieldSelectors - The selectors to extract fields

      • scrape

         abstract List<Map<String, String>> scrape(String url, String args, String restrictSelector, Map<String, String> fieldSelectors)

        Load or fetch a webpage located by the given url, and then extract fields specified by field selectors.

        Parameters:
        url - The url to scrape
        args - The load arguments
        restrictSelector - A CSS selector to locate a DOM where all fields are restricted to
        fieldSelectors - The selectors to extract fields

      • scrape

         abstract List<Map<String, String>> scrape(String url, LoadOptions options, String restrictSelector, Map<String, String> fieldSelectors)

        Load or fetch a webpage located by the given url, and then extract fields specified by field selectors.

        Parameters:
        url - The url to scrape
        options - The load options
        restrictSelector - A CSS selector to locate a DOM where all fields are restricted to
        fieldSelectors - The selectors to extract fields

      • scrapeOutPages

        @ExperimentalApi() abstract List<Map<String, String>> scrapeOutPages(String portalUrl, String args, Iterable<String> fieldSelectors)

        Load or fetch out pages specified by out link selector, and then extract fields specified by field selectors from each out page.

        Parameters:
        portalUrl - The portal url to start scraping
        args - Load arguments for both the portal page and out pages
        fieldSelectors - CSS selectors to extract fields from out pages

      • scrapeOutPages

        @ExperimentalApi() abstract List<Map<String, String>> scrapeOutPages(String portalUrl, LoadOptions options, Iterable<String> fieldSelectors)

        Load or fetch out pages specified by out link selector, and then extract fields specified by field selectors from each out page.

        Parameters:
        portalUrl - The portal url to start scraping
        options - Load options for both the portal page and out pages
        fieldSelectors - CSS selectors to extract fields from out pages

      • scrapeOutPages

        @ExperimentalApi() abstract List<Map<String, String>> scrapeOutPages(String portalUrl, String args, String restrictSelector, Iterable<String> fieldSelectors)

        Load or fetch out pages specified by out link selector, and then extract fields specified by field selectors from each out page.

        Parameters:
        portalUrl - The portal url to start scraping
        args - Load arguments for both the portal page and out pages
        restrictSelector - A CSS selector to locate a DOM where all fields are restricted to
        fieldSelectors - CSS selectors to extract fields from out pages

      • scrapeOutPages

        @ExperimentalApi() abstract List<Map<String, String>> scrapeOutPages(String portalUrl, LoadOptions options, String restrictSelector, Iterable<String> fieldSelectors)

        Load or fetch out pages specified by out link selector, and then extract fields specified by field selectors from each out page.

        Parameters:
        portalUrl - The portal url to start scraping
        options - Load options for both the portal page and out pages
        restrictSelector - A CSS selector to locate a DOM where all fields are restricted to
        fieldSelectors - CSS selectors to extract fields from out pages

      • scrapeOutPages

        @ExperimentalApi() abstract List<Map<String, String>> scrapeOutPages(String portalUrl, String args, Map<String, String> fieldSelectors)

        Load or fetch out pages specified by out link selector, and then extract fields specified by field selectors from each out page.

        Parameters:
        portalUrl - The portal url to start scraping
        args - Load arguments for both the portal page and out pages
        fieldSelectors - CSS selectors to extract fields from out pages

      • scrapeOutPages

        @ExperimentalApi() abstract List<Map<String, String>> scrapeOutPages(String portalUrl, LoadOptions options, Map<String, String> fieldSelectors)

        Load or fetch out pages specified by out link selector, and then extract fields specified by field selectors from each out page.

        Parameters:
        portalUrl - The portal url to start scraping
        options - Load options for both the portal page and out pages
        fieldSelectors - CSS selectors to extract fields from out pages

      • scrapeOutPages

        @ExperimentalApi() abstract List<Map<String, String>> scrapeOutPages(String portalUrl, String args, String restrictSelector, Map<String, String> fieldSelectors)

        Load or fetch out pages specified by out link selector, and then extract fields specified by field selectors from each out page.

        Parameters:
        portalUrl - The portal url to start scraping
        args - Load arguments for both the portal page and out pages
        restrictSelector - A CSS selector to locate a DOM where all fields are restricted to
        fieldSelectors - CSS selectors to extract fields from out pages

      • scrapeOutPages

        @ExperimentalApi() abstract List<Map<String, String>> scrapeOutPages(String portalUrl, LoadOptions options, String restrictSelector, Map<String, String> fieldSelectors)

        Load or fetch out pages specified by out link selector, and then extract fields specified by field selectors from each out page.

        Parameters:
        portalUrl - The portal url to start scraping
        options - Load options for both the portal page and out pages
        restrictSelector - A CSS selector to locate a DOM where all fields are restricted to
        fieldSelectors - CSS selectors to extract fields from out pages

      • flush

         abstract Unit flush()

        Flush to the storage

      • persist

         abstract Boolean persist(WebPage page)

        Persist to the storage

      • export

         abstract Path export(WebPage page)

        Export the content of a webpage.

        Parameters:
        page - Page to export

      • export

         abstract Path export(WebPage page, String ident)

        Export the content of a webpage.

        Parameters:
        page - Page to export
        ident - File name identifier used to distinguish from other names

      • export

         abstract Path export(FeaturedDocument doc)

        Export the outer HTML of the document.

        Parameters:
        doc - Document to export

      • export

         abstract Path export(FeaturedDocument doc, String ident)

        Export the outer HTML of the document.

        Parameters:
        doc - Document to export
        ident - File name identifier used to distinguish from other names

      • exportTo

         abstract Path exportTo(FeaturedDocument doc, Path path)

        Export the whole HTML of the document to the given path.

        Parameters:
        doc - Document to export
        path - Path to save the exported content

      • getUnmodifiedConfig

         abstract ImmutableConfig getUnmodifiedConfig()

        An immutable config which is loaded from the config file at process startup, and never changes

      • getSessionConfig

         abstract VolatileConfig getSessionConfig()

        The session scope volatile config, every setting is supposed to be changed at any time and any place

      • getSessionBeanFactory

         abstract BeanFactory getSessionBeanFactory()

        The scoped bean factory: for each volatileConfig object, there is a bean factory

      • getPageCache

         abstract ConcurrentExpiringLRUCache<String, WebPage> getPageCache()

        The global page cache

      • getDocumentCache

         abstract ConcurrentExpiringLRUCache<String, FeaturedDocument> getDocumentCache()

        The global document cache