Package ai.platon.pulsar.session

Interface PulsarSession

  • All Implemented Interfaces:
    java.lang.AutoCloseable 
    
public interface PulsarSession
 implements AutoCloseable

    PulsarSession defines an interface to load webpages from local storage or fetch from the Internet, as well as methods for parsing, extracting, saving and exporting webpages.

    Key methods:

    • load: load a webpage from local storage, or fetch it from the Internet.

    • parse: parse a webpage into a document.

    • scrape: load a webpage, parse it into a document and then extract fields from the document.

    • submit: submit a url to the URL pool, the url will be processed in the main loop later.

    And also the batch versions:

    • loadOutPages: load the portal page and out pages.

    • scrapeOutPages: load the portal page and out pages, extract fields from out pages.

    The first thing to understand is how to load a page. Load methods like load first check the local storage and return the local version if the required page exists and meets the requirements, otherwise it will be fetched from the Internet.

    Load parameters or load options can be used to specify when the system will fetch a webpage from the Internet:

    • Expiration

    • Force refresh

    • Page size

    • Required fields

    • Other conditions

    Once a page is loaded from local storage, or fetched from the Internet, we come to the next process steps:

    • parse the page content into an HTML document

    • extract fields from the HTML document

    • write the extraction results to a destination, such as

    There are many ways to fetch the content of a page from the Internet:

    • through HTTP protocol

    • through a real browser

    Since the webpages are becoming more and more complex, fetching webpages through real browsers is the primary way nowadays.

    When we fetch webpages using a real browser, we need to interact with pages to ensure the required fields are loaded correctly and completely. Enable PageEvent and use WebDriver to archive such purpose.

    val options = session.options(args)
options.event.browseEvent.onDidDOMStateCheck.addLast { page, driver ->
  driver.scrollDown()
}
session.load(url, options)

    WebDriver provides a complete method set for RPA, just like selenium, playwright and puppeteer does, all actions and behaviors are optimized to mimic real people as closely as possible.

    • Constructor Detail

    • Method Detail

      • normalize

         abstract List<NormUrl> normalize(Collection<UrlAware> urls, String args, Boolean toItemOption)

        Normalize urls, remove invalid ones

        Parameters:
        urls - The urls to normalize
        args - The arguments
        toItemOption - If the LoadOptions is converted to item load options
        Returns:

        All normalized urls, all invalid input urls are removed

      • normalize

         abstract List<NormUrl> normalize(Collection<UrlAware> urls, LoadOptions options, Boolean toItemOption)

        Normalize urls, remove invalid ones

        Parameters:
        urls - The urls to normalize
        options - The LoadOptions applied to each url
        toItemOption - If the LoadOptions is converted to item load options
        Returns:

        All normalized urls, all invalid input urls are removed

      • inject

         abstract WebPage inject(String url)

        Inject a url as a seed to fetch. Injection is usually used in Nutch style crawls where the execution flow likes the following:

        inject -> generate -> fetch -> parse -> index -> update ^ ^ | <- <- <- <- |

        Parameters:
        url - The url to inject, con be followed by arguments
        Returns:

        A newly created webpage record which is ready to be generated

      • get

         abstract WebPage get(String url)

        Get a page from storage.

        Parameters:
        url - The url
        Returns:

        The webpage in storage if exists, otherwise returns a NIL page

      • get

         abstract WebPage get(String url, String fields)

        Get a page from storage.

        Parameters:
        url - The url
        fields - The fields to load from local storage
        Returns:

        The webpage in storage if exists, otherwise returns a NIL page

      • getOrNull

         abstract WebPage getOrNull(String url)

        Get a page from storage.

        Parameters:
        url - The url
        Returns:

        The page in storage if exists, otherwise returns null

      • getOrNull

         abstract WebPage getOrNull(String url, String fields)

        Get a page from storage.

        Parameters:
        url - The url
        fields - The fields to load from local storage
        Returns:

        The page in storage if exists, otherwise returns null

      • getContent

         abstract ByteBuffer getContent(String url)

        Get the content of the page from the storage

        Parameters:
        url - The url of the page to retrieve
        Returns:

        The page content or null

      • getContentAsString

        @Beta() abstract String getContentAsString(String url)

        Get the content of the page from the storage

        Parameters:
        url - The url of the page to retrieve
        Returns:

        The page content in string format or null

      • exists

         abstract Boolean exists(String url)

        Check if the page exists in the storage.

        Parameters:
        url - The url to check
        Returns:

        true if the page exists, false otherwise

      • fetchState

         abstract CheckState fetchState(WebPage page, LoadOptions options)

        Return the fetch state of the page.

        Parameters:
        page - The webpage
        options - The load options
        Returns:

        The fetch state of the page

      • open

         abstract WebPage open(String url)

        Open a url.

        This method opens the url immediately, regardless of the previous state of the page.

        Parameters:
        url - The url to open
        Returns:

        The webpage loaded or NIL

      • load

         abstract WebPage load(String url)

        Load a url with specified arguments.

        This method first checks the url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

        Other fetch conditions can be specified by load arguments:

        • expiration

        • page size requirement

        • fields requirement

        • other

        Parameters:
        url - The url to load
        Returns:

        The webpage loaded or NIL

      • load

         abstract WebPage load(String url, String args)

        Load a url with specified arguments.

        This method first checks the url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

        Other fetch conditions can be specified by load arguments:

        • expiration

        • page size requirement

        • fields requirement

        • other

        Parameters:
        url - The url to load
        args - The load arguments
        Returns:

        The webpage loaded or NIL

      • load

         abstract WebPage load(String url, LoadOptions options)

        Load a url with specified options.

        This method first checks the url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

        Other fetch conditions can be specified by load arguments:

        • expiration

        • page size requirement

        • fields requirement

        • other

        Parameters:
        url - The url to load
        options - The load options
        Returns:

        The webpage loaded or NIL

      • load

         abstract WebPage load(UrlAware url)

        Load a url with the specified arguments.

        This method first checks the url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

        Parameters:
        url - The url to load
        Returns:

        The webpage loaded or NIL

      • load

         abstract WebPage load(UrlAware url, String args)

        Load a url with the specified arguments.

        This method first checks the url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

        Parameters:
        url - The url to load
        args - The load arguments
        Returns:

        The webpage loaded or NIL

      • load

         abstract WebPage load(UrlAware url, LoadOptions options)

        Load a url with options.

        This method first checks the url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

        Parameters:
        url - The url to load
        options - The load options
        Returns:

        The webpage loaded or NIL

      • load

         abstract WebPage load(NormUrl url)

        Load a normal url.

        This method first checks the url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

        Parameters:
        url - The normal url
        Returns:

        The webpage loaded or NIL

      • loadDeferred

         abstract WebPage loadDeferred(String url, String args)

        Load a url with specified options.

        This method first checks the url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

        This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.

        Parameters:
        url - The url to load
        args - The load options
        Returns:

        The webpage loaded or NIL

      • loadDeferred

         abstract WebPage loadDeferred(String url, LoadOptions options)

        Load a url with specified options.

        This method first checks the url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

        This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.

        Parameters:
        url - The url to load
        options - The load options
        Returns:

        The webpage loaded or NIL

      • loadDeferred

         abstract WebPage loadDeferred(UrlAware url, String args)

        Load a url with specified arguments.

        This method first checks the url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

        This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.

        Parameters:
        url - The url to load
        args - The load args
        Returns:

        The webpage loaded or NIL

      • loadDeferred

         abstract WebPage loadDeferred(UrlAware url, LoadOptions options)

        Load a url with specified options.

        This method first checks the url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

        This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.

        Parameters:
        url - The url to load
        options - The load options
        Returns:

        The webpage loaded or NIL

      • loadDeferred

         abstract WebPage loadDeferred(NormUrl url)

        Load a url with specified options

        This method first checks the url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

        This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.

        Parameters:
        url - The normal url
        Returns:

        The webpage loaded or NIL

      • loadAll

         abstract List<WebPage> loadAll(Iterable<String> urls)

        Load all urls with specified options

        This method first checks each url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

        Parameters:
        urls - The urls to load
        Returns:

        The webpage loaded or NIL

      • loadAll

         abstract List<WebPage> loadAll(Iterable<String> urls, String args)

        Load all urls with specified options

        This method first checks each url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

        Parameters:
        urls - The urls to load
        args - The load arguments
        Returns:

        The webpage loaded or NIL

      • loadAll

         abstract List<WebPage> loadAll(Iterable<String> urls, LoadOptions options)

        Load all urls with specified options

        This method first checks each url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

        Parameters:
        urls - The urls to load
        options - The load options
        Returns:

        The webpage loaded or NIL

      • loadAll

         abstract List<WebPage> loadAll(Collection<UrlAware> urls)

        Load all urls with specified options

        This method first checks each url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

        Parameters:
        urls - The urls to load
        Returns:

        The webpage loaded or NIL

      • loadAll

         abstract List<WebPage> loadAll(Collection<UrlAware> urls, String args)

        Load all urls with specified options

        This method first checks each url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

        Parameters:
        urls - The urls to load
        args - The load arguments
        Returns:

        The webpage loaded or NIL

      • loadAll

         abstract List<WebPage> loadAll(Collection<UrlAware> urls, LoadOptions options)

        Load all urls with specified options

        This method first checks each url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

        Parameters:
        urls - The urls to load
        options - The load options
        Returns:

        The webpage loaded or NIL

      • loadAll

         abstract List<WebPage> loadAll(List<NormUrl> normUrls)

        Load all normal urls with specified options

        This method first checks each url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

        Parameters:
        normUrls - The normal urls to load
        Returns:

        The loaded webpages

      • loadAsync

         abstract CompletableFuture<WebPage> loadAsync(String url)

        Load a normal url in java async style

        Parameters:
        url - The url to load
        Returns:

        A completable future of webpage

      • loadAsync

         abstract CompletableFuture<WebPage> loadAsync(String url, String args)

        Load a normal url in java async style

        Parameters:
        url - The url to load
        Returns:

        A completable future of webpage

      • loadAsync

         abstract CompletableFuture<WebPage> loadAsync(UrlAware url)

        Load a normal url in java async style

        Parameters:
        url - The url to load
        Returns:

        A completable future of webpage

      • loadAsync

         abstract CompletableFuture<WebPage> loadAsync(UrlAware url, String args)

        Load a normal url in java async style

        Parameters:
        url - The url to load
        Returns:

        A completable future of webpage

      • loadAsync

         abstract CompletableFuture<WebPage> loadAsync(UrlAware url, LoadOptions options)

        Load a normal url in java async style

        Parameters:
        url - The url to load
        Returns:

        A completable future of webpage

      • loadAsync

         abstract CompletableFuture<WebPage> loadAsync(NormUrl url)

        Load a normal url in java async style

        Parameters:
        url - The normal url to load
        Returns:

        A completable future of webpage

      • loadAllAsync

         abstract List<CompletableFuture<WebPage>> loadAllAsync(Iterable<String> urls)

        Load all normal urls in java async style

        This method first checks each url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

        Parameters:
        urls - The normal urls to load
        Returns:

        The completable futures of webpages

      • loadAllAsync

         abstract List<CompletableFuture<WebPage>> loadAllAsync(Iterable<String> urls, String args)

        Load all normal urls in java async style.

        This method first checks each url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

        Parameters:
        urls - The normal urls to load
        Returns:

        The completable futures of webpages

      • loadAllAsync

         abstract List<CompletableFuture<WebPage>> loadAllAsync(Iterable<String> urls, LoadOptions options)

        Load all normal urls in java async style.

        This method first checks each url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

        Parameters:
        urls - The normal urls to load
        Returns:

        The completable futures of webpages

      • loadAllAsync

         abstract List<CompletableFuture<WebPage>> loadAllAsync(Collection<UrlAware> urls)

        Load all normal urls in java async style.

        This method first checks each url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

        Parameters:
        urls - The normal urls to load
        Returns:

        The completable futures of webpages

      • loadAllAsync

         abstract List<CompletableFuture<WebPage>> loadAllAsync(Collection<UrlAware> urls, String args)

        Load all normal urls in java async style.

        This method first checks each url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

        Parameters:
        urls - The normal urls to load
        Returns:

        The completable futures of webpages

      • loadAllAsync

         abstract List<CompletableFuture<WebPage>> loadAllAsync(Collection<UrlAware> urls, LoadOptions options)

        Load all normal urls in java async style.

        This method first checks each url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

        Parameters:
        urls - The normal urls to load
        Returns:

        The completable futures of webpages

      • loadAllAsync

         abstract List<CompletableFuture<WebPage>> loadAllAsync(List<NormUrl> urls)

        Load all normal urls in java async style

        This method first checks each url in the local store and return the local version if the page exists and matches the requirements, otherwise fetch it from the Internet.

        Parameters:
        urls - The normal urls to load
        Returns:

        The completable futures of webpages

      • submit

         abstract PulsarSession submit(String url)

        Submit a url to the URL pool, the url will be processed in the crawl loop later

        Parameters:
        url - The url to submit
        Returns:

        The PulsarSession itself to enabled chained operations

      • submit

         abstract PulsarSession submit(String url, String args)

        Submit a url to the URL pool, and it will be processed in a crawl loop

        Parameters:
        url - The url to submit
        args - The load arguments
        Returns:

        The PulsarSession itself to enabled chained operations

      • submit

         abstract PulsarSession submit(String url, LoadOptions options)

        Submit a url to the URL pool, and it will be processed in a crawl loop

        Parameters:
        url - The url to submit
        options - The load options
        Returns:

        The PulsarSession itself to enabled chained operations

      • submit

         abstract PulsarSession submit(UrlAware url)

        Submit a url to the URL pool, and it will be processed in a crawl loop

        Parameters:
        url - The url to submit
        Returns:

        The PulsarSession itself to enabled chained operations

      • submit

         abstract PulsarSession submit(UrlAware url, String args)

        Submit a url to the URL pool, and it will be processed in a crawl loop

        Parameters:
        url - The url to submit
        Returns:

        The PulsarSession itself to enabled chained operations

      • submitAll

         abstract PulsarSession submitAll(Iterable<String> urls)

        Submit the urls to the URL pool, the submitted urls will be processed in a crawl loop

        Parameters:
        urls - The urls to submit
        Returns:

        The PulsarSession itself to enabled chained operations

      • submitAll

         abstract PulsarSession submitAll(Iterable<String> urls, String args)

        Submit the urls to the URL pool, the submitted urls will be processed in a crawl loop

        Parameters:
        urls - The urls to submit
        args - The load arguments
        Returns:

        The PulsarSession itself to enabled chained operations

      • submitAll

         abstract PulsarSession submitAll(Iterable<String> urls, LoadOptions options)

        Submit the urls to the URL pool, the submitted urls will be processed in a crawl loop

        Parameters:
        urls - The urls to submit
        options - The load options
        Returns:

        The PulsarSession itself to enabled chained operations

      • submitAll

         abstract PulsarSession submitAll(Collection<UrlAware> urls)

        Submit the urls to the URL pool, the submitted urls will be processed in a crawl loop

        Parameters:
        urls - The urls to submit
        Returns:

        The PulsarSession itself to enabled chained operations

      • submitAll

         abstract PulsarSession submitAll(Collection<UrlAware> urls, String args)

        Submit the urls to the URL pool, the submitted urls will be processed in a crawl loop

        Parameters:
        urls - The urls to submit
        Returns:

        The PulsarSession itself to enabled chained operations

      • loadOutPages

         abstract List<WebPage> loadOutPages(String portalUrl, String args)

        Load or fetch the portal page, and then load or fetch the out links selected by -outLink option.

        Option -outLink specifies the cssSelector for links in the portal page to load.

        Parameters:
        portalUrl - The portal url from where to load pages
        args - The load arguments
        Returns:

        The loaded out pages

      • loadOutPages

         abstract List<WebPage> loadOutPages(String portalUrl, LoadOptions options)

        Load or fetch the portal page, and then load or fetch the out links selected by -outLink option.

        Option -outLink specifies the cssSelector for links in the portal page to load.

        Parameters:
        portalUrl - The portal url from where to load pages
        options - The load options
        Returns:

        The loaded out pages

      • loadOutPages

         abstract List<WebPage> loadOutPages(UrlAware portalUrl, String args)

        Load or fetch the portal page, and then load or fetch the out links selected by -outLink option.

        Option -outLink specifies the cssSelector for links in the portal page to load.

        Parameters:
        portalUrl - The portal url from where to load pages
        args - The load arguments
        Returns:

        The loaded out pages

      • loadOutPages

         abstract List<WebPage> loadOutPages(UrlAware portalUrl, LoadOptions options)

        Load or fetch the portal page, and then load or fetch the out links selected by -outLink option.

        Option -outLink specifies the cssSelector for links in the portal page to load.

        Parameters:
        portalUrl - The portal url from where to load pages
        options - The load options
        Returns:

        The loaded out pages

      • loadOutPagesAsync

         abstract List<CompletableFuture<WebPage>> loadOutPagesAsync(String portalUrl, String args)

        Load or fetch the portal page, and then load or fetch the out links selected by -outLink option asynchronously.

        Option -outLink specifies the cssSelector for links in the portal page to load.

        Parameters:
        portalUrl - The portal url from where to load pages
        args - The load arguments
        Returns:

        The loaded out pages

      • loadOutPagesAsync

         abstract List<CompletableFuture<WebPage>> loadOutPagesAsync(String portalUrl, LoadOptions options)

        Load or fetch the portal page, and then load or fetch the out links selected by -outLink option asynchronously.

        Option -outLink specifies the cssSelector for links in the portal page to load.

        Parameters:
        portalUrl - The portal url from where to load pages
        options - The load options
        Returns:

        The loaded out pages

      • submitForOutPages

         abstract PulsarSession submitForOutPages(String portalUrl, String args)

        Load the portal page and submit the out links specified by the -outLink option to the URL pool.

        Option -outLink specifies the cssSelector for links in the portal page to load.

        The submitted urls will be processed in a crawl loop later.

        Parameters:
        portalUrl - The portal url from where to load pages
        args - The load arguments
        Returns:

        The PulsarSession itself to enable chained operation

      • submitForOutPages

         abstract PulsarSession submitForOutPages(String portalUrl, LoadOptions options)

        Load the portal page and submit the out links specified by the -outLink option to the URL pool.

        Option -outLink specifies the cssSelector for links in the portal page to load.

        The submitted urls will be processed in a crawl loop later.

        Parameters:
        portalUrl - The portal url from where to load pages
        options - The load options
        Returns:

        The PulsarSession itself to enable chained operation

      • submitForOutPages

         abstract PulsarSession submitForOutPages(UrlAware portalUrl, String args)

        Load the portal page and submit the out links specified by the -outLink option to the URL pool.

        Option -outLink specifies the cssSelector for links in the portal page to load.

        The submitted urls will be processed in a crawl loop later.

        Parameters:
        portalUrl - The portal url from where to load pages
        args - The load arguments
        Returns:

        The PulsarSession itself to enable chained operation

      • submitForOutPages

         abstract PulsarSession submitForOutPages(UrlAware portalUrl, LoadOptions options)

        Load the portal page and submit the out links specified by the -outLink option to the URL pool.

        Option -outLink specifies the cssSelector for links in the portal page to load.

        The submitted urls will be processed in a crawl loop later.

        Parameters:
        portalUrl - The portal url from where to load pages
        options - The load options
        Returns:

        The PulsarSession itself to enable chained operation

      • loadResource

         abstract WebPage loadResource(String url, String referrer)

        Load a url as a resource without browser rendering.

        Referrer will be opened by a browser first to obtain browsing environment, such as headers and cookies, and then the browsing environment will be applied to the later resource fetching.

        Parameters:
        url - The url to load
        referrer - The referrer URL
        Returns:

        The webpage containing the resource

      • loadResource

         abstract WebPage loadResource(String url, String referrer, String args)

        Load a url as a resource without browser rendering.

        Referrer will be opened by a browser first to obtain browsing environment, such as headers and cookies, and then the browsing environment will be applied to the later resource fetching.

        Parameters:
        url - The url to load
        referrer - The referrer URL
        args - The load arguments
        Returns:

        The webpage containing the resource

      • loadResource

         abstract WebPage loadResource(String url, String referrer, LoadOptions options)

        Load a url as a resource without browser rendering.

        Referrer will be opened by a browser first to obtain browsing environment, such as headers and cookies, and then the browsing environment will be applied to the later resource fetching.

        Parameters:
        url - The url to load
        referrer - The referrer URL
        options - The load options
        Returns:

        The webpage containing the resource

      • loadResourceDeferred

         abstract WebPage loadResourceDeferred(String url, String referrer)

        Load a url as a resource without browser rendering.

        Referrer will be opened by a browser first to obtain browsing environment, such as headers and cookies, and then the browsing environment will be applied to the later resource fetching.

        This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.

        Parameters:
        url - The url to load
        referrer - The referrer URL
        Returns:

        The webpage containing the resource

      • loadResourceDeferred

         abstract WebPage loadResourceDeferred(String url, String referrer, String args)

        Load a url as a resource without browser rendering.

        Referrer will be opened by a browser first to obtain browsing environment, such as headers and cookies, and then the browsing environment will be applied to the later resource fetching.

        This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.

        Parameters:
        url - The url to load
        referrer - The referrer URL
        args - The load arguments
        Returns:

        The webpage containing the resource

      • loadResourceDeferred

         abstract WebPage loadResourceDeferred(String url, String referrer, LoadOptions options)

        Load a url as a resource without browser rendering.

        Referrer will be opened by a browser first to obtain browsing environment, such as headers and cookies, and then the browsing environment will be applied to the later resource fetching.

        This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.

        Parameters:
        url - The url to load
        referrer - The referrer URL
        options - The load options
        Returns:

        The webpage containing the resource

      • parse

         abstract FeaturedDocument parse(WebPage page)

        Parse a webpage into an HTML document.

      • parse

         abstract FeaturedDocument parse(WebPage page, Boolean noCache)

        Parse a webpage into an HTML document.

      • loadDocument

         abstract FeaturedDocument loadDocument(String url)

        Load or fetch a webpage and parse it into an HTML document

      • loadDocument

         abstract FeaturedDocument loadDocument(String url, String args)

        Load or fetch a webpage and parse it into an HTML document

      • loadDocument

         abstract FeaturedDocument loadDocument(String url, LoadOptions options)

        Load or fetch a webpage and parse it into an HTML document

      • loadDocument

         abstract FeaturedDocument loadDocument(UrlAware url)

        Load or fetch a webpage and parse it into an HTML document

      • loadDocument

         abstract FeaturedDocument loadDocument(UrlAware url, String args)

        Load or fetch a webpage and parse it into an HTML document

      • loadDocument

         abstract FeaturedDocument loadDocument(UrlAware url, LoadOptions options)

        Load or fetch a webpage and parse it into an HTML document

      • loadDocument

         abstract FeaturedDocument loadDocument(NormUrl url)

        Load or fetch a webpage and then parse it into an HTML document.

      • scrape

         abstract Map<String, String> scrape(String url, String args, Iterable<String> fieldSelectors)

        Load or fetch a webpage located by the given url, and then extract fields specified by field selectors.

        Parameters:
        url - The url to scrape
        args - The load arguments
        fieldSelectors - The selectors to extract fields
        Returns:

        All the extracted fields and their selectors

      • scrape

         abstract Map<String, String> scrape(String url, LoadOptions options, Iterable<String> fieldSelectors)

        Load or fetch a webpage located by the given url, and then extract fields specified by field selectors.

        Parameters:
        url - The url to scrape
        options - The load options
        fieldSelectors - The selectors to extract fields
        Returns:

        All the extracted fields and their selectors

      • scrape

         abstract Map<String, String> scrape(String url, String args, Map<String, String> fieldSelectors)

        Load or fetch a webpage located by the given url, and then extract fields specified by field selectors.

        Parameters:
        url - The url to scrape
        args - The load arguments
        fieldSelectors - The selectors to extract fields
        Returns:

        All the extracted fields and their names

      • scrape

         abstract Map<String, String> scrape(String url, LoadOptions options, Map<String, String> fieldSelectors)

        Load or fetch a webpage located by the given url, and then extract fields specified by field selectors.

        Parameters:
        url - The url to scrape
        options - The load options
        fieldSelectors - The selectors to extract fields
        Returns:

        All the extracted fields and their names

      • scrape

         abstract List<Map<String, String>> scrape(String url, String args, String restrictSelector, Iterable<String> fieldSelectors)

        Load or fetch a webpage located by the given url, and then extract fields specified by field selectors.

        Parameters:
        url - The url to scrape
        args - The load arguments
        restrictSelector - A CSS selector to locate a DOM where all fields are restricted to
        fieldSelectors - The selectors to extract fields
        Returns:

        All the extracted fields and their names

      • scrape

         abstract List<Map<String, String>> scrape(String url, LoadOptions options, String restrictSelector, Iterable<String> fieldSelectors)

        Load or fetch a webpage located by the given url, and then extract fields specified by field selectors.

        Parameters:
        url - The url to scrape
        options - The load options
        restrictSelector - A CSS selector to locate a DOM where all fields are restricted to
        fieldSelectors - The selectors to extract fields
        Returns:

        All the extracted fields and their selectors

      • scrape

         abstract List<Map<String, String>> scrape(String url, String args, String restrictSelector, Map<String, String> fieldSelectors)

        Load or fetch a webpage located by the given url, and then extract fields specified by field selectors.

        Parameters:
        url - The url to scrape
        args - The load arguments
        restrictSelector - A CSS selector to locate a DOM where all fields are restricted to
        fieldSelectors - The selectors to extract fields
        Returns:

        All the extracted fields and their names

      • scrape

         abstract List<Map<String, String>> scrape(String url, LoadOptions options, String restrictSelector, Map<String, String> fieldSelectors)

        Load or fetch a webpage located by the given url, and then extract fields specified by field selectors.

        Parameters:
        url - The url to scrape
        options - The load options
        restrictSelector - A CSS selector to locate a DOM where all fields are restricted to
        fieldSelectors - The selectors to extract fields
        Returns:

        All the extracted fields and their names

      • scrapeOutPages

        @ExperimentalApi() abstract List<Map<String, String>> scrapeOutPages(String portalUrl, String args, Iterable<String> fieldSelectors)

        Load or fetch out pages specified by out link selector, and then extract fields specified by field selectors from each out page.

        Parameters:
        portalUrl - The portal url to start scraping
        args - Load arguments for both the portal page and out pages
        fieldSelectors - CSS selectors to extract fields from out pages
        Returns:

        All extracted fields. For each out page, fields extracted with their selectors are saved in a map.

      • scrapeOutPages

        @ExperimentalApi() abstract List<Map<String, String>> scrapeOutPages(String portalUrl, LoadOptions options, Iterable<String> fieldSelectors)

        Load or fetch out pages specified by out link selector, and then extract fields specified by field selectors from each out page.

        Parameters:
        portalUrl - The portal url to start scraping
        options - Load options for both the portal page and out pages
        fieldSelectors - CSS selectors to extract fields from out pages
        Returns:

        All extracted fields. For each out page, fields extracted with their selectors are saved in a map.

      • scrapeOutPages

        @ExperimentalApi() abstract List<Map<String, String>> scrapeOutPages(String portalUrl, String args, String restrictSelector, Iterable<String> fieldSelectors)

        Load or fetch out pages specified by out link selector, and then extract fields specified by field selectors from each out page.

        Parameters:
        portalUrl - The portal url to start scraping
        args - Load arguments for both the portal page and out pages
        restrictSelector - A CSS selector to locate a DOM where all fields are restricted to
        fieldSelectors - CSS selectors to extract fields from out pages
        Returns:

        All extracted fields. For each out page, fields extracted with their selectors are saved in a map.

      • scrapeOutPages

        @ExperimentalApi() abstract List<Map<String, String>> scrapeOutPages(String portalUrl, LoadOptions options, String restrictSelector, Iterable<String> fieldSelectors)

        Load or fetch out pages specified by out link selector, and then extract fields specified by field selectors from each out page.

        Parameters:
        portalUrl - The portal url to start scraping
        options - Load options for both the portal page and out pages
        restrictSelector - A CSS selector to locate a DOM where all fields are restricted to
        fieldSelectors - CSS selectors to extract fields from out pages
        Returns:

        All extracted fields. For each out page, fields extracted with their selectors are saved in a map.

      • scrapeOutPages

        @ExperimentalApi() abstract List<Map<String, String>> scrapeOutPages(String portalUrl, String args, Map<String, String> fieldSelectors)

        Load or fetch out pages specified by out link selector, and then extract fields specified by field selectors from each out page.

        Parameters:
        portalUrl - The portal url to start scraping
        args - Load arguments for both the portal page and out pages
        fieldSelectors - CSS selectors to extract fields from out pages
        Returns:

        All extracted fields. For each out page, fields extracted with their names are saved in a map.

      • scrapeOutPages

        @ExperimentalApi() abstract List<Map<String, String>> scrapeOutPages(String portalUrl, LoadOptions options, Map<String, String> fieldSelectors)

        Load or fetch out pages specified by out link selector, and then extract fields specified by field selectors from each out page.

        Parameters:
        portalUrl - The portal url to start scraping
        options - Load options for both the portal page and out pages
        fieldSelectors - CSS selectors to extract fields from out pages
        Returns:

        All extracted fields. For each out page, fields extracted with their names are saved in a map.

      • scrapeOutPages

        @ExperimentalApi() abstract List<Map<String, String>> scrapeOutPages(String portalUrl, String args, String restrictSelector, Map<String, String> fieldSelectors)

        Load or fetch out pages specified by out link selector, and then extract fields specified by field selectors from each out page.

        Parameters:
        portalUrl - The portal url to start scraping
        args - Load arguments for both the portal page and out pages
        restrictSelector - A CSS selector to locate a DOM where all fields are restricted to
        fieldSelectors - CSS selectors to extract fields from out pages
        Returns:

        All extracted fields. For each out page, fields extracted with their names are saved in a map.

      • scrapeOutPages

        @ExperimentalApi() abstract List<Map<String, String>> scrapeOutPages(String portalUrl, LoadOptions options, String restrictSelector, Map<String, String> fieldSelectors)

        Load or fetch out pages specified by out link selector, and then extract fields specified by field selectors from each out page.

        Parameters:
        portalUrl - The portal url to start scraping
        options - Load options for both the portal page and out pages
        restrictSelector - A CSS selector to locate a DOM where all fields are restricted to
        fieldSelectors - CSS selectors to extract fields from out pages
        Returns:

        All extracted fields. For each out page, fields extracted with their names are saved in a map.

      • flush

         abstract Unit flush()

        Flush to the storage

      • persist

         abstract Boolean persist(WebPage page)

        Persist to the storage

      • export

         abstract Path export(WebPage page)

        Export the content of a webpage.

        Parameters:
        page - Page to export
        Returns:

        The path of the exported page

      • export

         abstract Path export(WebPage page, String ident)

        Export the content of a webpage.

        Parameters:
        page - Page to export
        ident - File name identifier used to distinguish from other names
        Returns:

        The path of the exported page

      • export

         abstract Path export(FeaturedDocument doc)

        Export the outer HTML of the document.

        Parameters:
        doc - Document to export
        Returns:

        The path of the exported document

      • export

         abstract Path export(FeaturedDocument doc, String ident)

        Export the outer HTML of the document.

        Parameters:
        doc - Document to export
        ident - File name identifier used to distinguish from other names
        Returns:

        The path of the exported document

      • exportTo

         abstract Path exportTo(FeaturedDocument doc, Path path)

        Export the whole HTML of the document to the given path.

        Parameters:
        doc - Document to export
        path - Path to save the exported content
        Returns:

        The path of the exported document

      • getUnmodifiedConfig

         abstract ImmutableConfig getUnmodifiedConfig()

        An immutable config which is loaded from the config file at process startup, and never changes

      • getSessionConfig

         abstract VolatileConfig getSessionConfig()

        The session scope volatile config, every setting is supposed to be changed at any time and any place

      • getSessionBeanFactory

         abstract BeanFactory getSessionBeanFactory()

        The scoped bean factory: for each volatileConfig object, there is a bean factory

      • getPageCache

         abstract ConcurrentExpiringLRUCache<String, WebPage> getPageCache()

        The global page cache

      • getDocumentCache

         abstract ConcurrentExpiringLRUCache<String, FeaturedDocument> getDocumentCache()

        The global document cache