Package ai.platon.pulsar.skeleton.session

Interface PulsarSession

  • All Implemented Interfaces:
    java.lang.AutoCloseable 
    
public interface PulsarSession
 implements AutoCloseable

    The PulsarSession interface is designed to provide methods for loading pages from either local storage or from the internet. Additionally, it offers functionality for parsing these pages, extracting relevant data, and saving or exporting the web content.

    Key methods:

    • load: load a page from local storage or the Internet.

    • parse: parse a page into a document.

    • submit: submit a url to the URL pool, the url will be processed in the main loop later.

    • scrape: load a page, parse it into a document and then extract fields from the document.

    Basic examples:

    val url = "http://example.com"
val page = session.load(url)
val document = session.parse(page)

val title = document.selectFirstTextOrNull(".title")
val content = document.selectFirstTextOrNull(".content")

val fields = session.scrape("http://example.com", "-expire 1d", listOf(".title", ".content"))

val document2 = session.loadDocument(url)

val url2 = "http://example.com/1"
session.submit(url2)

    And also the batch versions:

    val urls = listOf("http://example.com", "http://example.com/1")
val pages = session.loadAll(urls)

val pages2 = session.loadOutPages("http://example.com", "-outLink a[rel='next']")
val fields = session.scrapeOutPages("http://example.com", "-outLink a", listOf("title", "content"))

session.submitAll(urls)
session.submitForOutPages("http://example.com", "-outLink a[rel='next']")

    The primary consideration is understanding how to load a page efficiently. Load methods, such as load, initially check local storage for the required page. If the page exists locally and meets the necessary criteria, it is retrieved from the local storage; otherwise, the page is fetched from the internet.

    To load a page, PulsarRPA follows these steps:

    • Check the local storage for the required page.

    • If the page exists in local storage and meets the specified criteria (such as being up-to-date or valid), return the local version.

    • If the page does not exist in local storage or does not meet the criteria, fetch the page from the internet.

    • Handle any errors that occur during the process, such as issues with local storage access or problems with the network fetch.

    To optimize the retrieval of webpages, load parameters or load options are used to establish the conditions under which a page should be fetched from the internet.

    These conditions may encompass:

    • Expiration timestamps to determine if the local page is still valid.

    • A directive for a forced refresh to ensure the latest content is retrieved.

    • Specifications regarding the minimal acceptable page size.

    • Identifiers for essential fields that must be present in the page.

    • Additional custom criteria that align with the application’s needs.

    For example:

    val url = "http://example.com"
val page = session.load(url, "-expire 1d")

val url2 = "http://example.com/1"
val page2 = session.load(url2, "-refresh")

val url3 = "http://example.com/2"
val page3 = session.load(url3, "-requireSize 100000")

    Once a page has been retrieved from either local storage or the internet, the subsequent processing steps include:

    • Parsing the page content to construct an HTML document.

    • Extracting relevant data fields from the parsed HTML document.

    • Recording the extracted data to a designated destination.

    For example:

    val page = session.load("http://example.com")
val document = session.parse(page)

val title = document.selectFirstTextOrNull(".title")
val content = document.selectFirstTextOrNull(".content")

val path = session.exportTo(page, Paths.get("/tmp/example.html"))

    There are many ways to fetch the content of a page from the Internet, but the two primary methods are:

    • through HTTP protocol

    • through a real browser

    Since the webpages are becoming more and more complex, fetching webpages through real browsers is the primary way nowadays.

    When we fetch webpages using a real browser, sometimes we need to interact with pages to ensure the required fields are loaded correctly and completely. Enable PageEventHandlers and use WebDriver to archive such purpose.

    val options = session.options(args)
options.eventHandlers.browseEventHandlers.onDocumentSteady.addLast { page, driver ->
  driver.fill("input[name='search']", "geek")
  driver.click("button[type='submit']")
}
session.load(url, options)

    WebDriver offers a comprehensive method set for browser automation, meticulously designed to replicate real human actions and behaviors with precision.

    • Constructor Detail

    • Method Detail

      • data

         abstract Object data(String name)

        Get a variable which is stored in this session

        Parameters:
        name - The name of the variable
        Returns:

        The value of the variable

      • data

         abstract Unit data(String name, Object value)

        Store a variable in this session

        Parameters:
        name - The name of the variable
        value - The value of the variable

      • normalize

         abstract NormURL normalize(String url)

        Normalize a url.

        Parameters:
        url - The url to normalize
        Returns:

        The normalized url

      • normalize

         abstract NormURL normalize(String url, String args, Boolean toItemOption)

        Normalize a url.

        Parameters:
        url - The url to normalize
        args - The arguments
        toItemOption - If the LoadOptions is converted to item load options
        Returns:

        The normalized url

      • normalize

         abstract NormURL normalize(String url, LoadOptions options, Boolean toItemOption)

        Normalize a url.

        Parameters:
        url - The url to normalize
        options - The LoadOptions applied to the url
        toItemOption - If the LoadOptions is converted to item load options
        Returns:

        The normalized url

      • normalize

         abstract List<NormURL> normalize(Iterable<String> urls)

        Normalize urls, remove invalid ones.

        Parameters:
        urls - The urls to normalize
        Returns:

        All normalized urls

      • normalize

         abstract List<NormURL> normalize(Iterable<String> urls, String args, Boolean toItemOption)

        Normalize urls, remove invalid ones.

        Parameters:
        urls - The urls to normalize
        args - The arguments
        toItemOption - If the LoadOptions is converted to item load options
        Returns:

        All normalized urls

      • normalize

         abstract List<NormURL> normalize(Iterable<String> urls, LoadOptions options, Boolean toItemOption)

        Normalize urls, remove invalid ones.

        Parameters:
        urls - The urls to normalize
        options - The LoadOptions applied to each url
        toItemOption - If the LoadOptions is converted to item load options
        Returns:

        All normalized urls

      • normalize

         abstract NormURL normalize(UrlAware url)

        Normalize a url, return NormURL.NIL if the url is invalid.

        Parameters:
        url - The url to normalize
        Returns:

        The normalized url

      • normalize

         abstract NormURL normalize(UrlAware url, String args, Boolean toItemOption)

        Normalize a url, return NormURL.NIL if the url is invalid.

        Parameters:
        url - The url to normalize
        args - The arguments
        toItemOption - If the LoadOptions is converted to item load options
        Returns:

        The normalized url

      • normalize

         abstract NormURL normalize(UrlAware url, LoadOptions options, Boolean toItemOption)

        Normalize a url, return NormURL.NIL if the url is invalid.

        Parameters:
        url - The url to normalize
        options - The LoadOptions applied to the url
        toItemOption - If the LoadOptions is converted to item load options
        Returns:

        The normalized url

      • normalize

         abstract List<NormURL> normalize(Collection<UrlAware> urls)

        Normalize urls, remove invalid ones.

        Parameters:
        urls - The urls to normalize
        Returns:

        All normalized urls

      • normalize

         abstract List<NormURL> normalize(Collection<UrlAware> urls, String args, Boolean toItemOption)

        Normalize urls, remove invalid ones

        Parameters:
        urls - The urls to normalize
        args - The arguments
        toItemOption - If the LoadOptions is converted to item load options
        Returns:

        All normalized urls, all invalid input urls are removed

      • normalize

         abstract List<NormURL> normalize(Collection<UrlAware> urls, LoadOptions options, Boolean toItemOption)

        Normalize urls, remove invalid ones

        Parameters:
        urls - The urls to normalize
        options - The LoadOptions applied to each url
        toItemOption - If the LoadOptions is converted to item load options
        Returns:

        All normalized urls, all invalid input urls are removed

      • normalizeOrNull

         abstract NormURL normalizeOrNull(String url, LoadOptions options, Boolean toItemOption)

        Normalize a url, return null if the url is invalid.

        Parameters:
        url - The url to normalize
        options - The LoadOptions applied to the url
        toItemOption - If the LoadOptions is converted to item load options
        Returns:

        The normalized url or null

      • normalizeOrNull

         abstract NormURL normalizeOrNull(UrlAware url, LoadOptions options, Boolean toItemOption)

        Normalize a url, return null if the url is invalid.

        Parameters:
        url - The url to normalize
        options - The LoadOptions applied to the url
        toItemOption - If the LoadOptions is converted to item load options
        Returns:

        The normalized url or null

      • get

         abstract WebPage get(String url)

        Get a page from local storage.

        val url = "http://example.com"
val page = session.get(url)
        Parameters:
        url - The url
        Returns:

        The webpage in storage if exists, otherwise returns a NIL page

      • get

         abstract WebPage get(String url, String fields)

        Get a page from local storage.

        val url = "http://example.com"
val page = session.get(url, "content", "metadata")
        Parameters:
        url - The url
        fields - The fields to load from local storage
        Returns:

        The webpage in storage if exists, otherwise returns a NIL page

      • getOrNull

         abstract WebPage getOrNull(String url)

        Get a page from local storage.

        val url = "http://example.com"
val page = session.getOrNull(url)
        Parameters:
        url - The url
        Returns:

        The page in storage if exists, otherwise returns null

      • getOrNull

         abstract WebPage getOrNull(String url, String fields)

        Get a page from local storage.

        val url = "http://example.com"
val page = session.getOrNull(url, "content", "metadata")
        Parameters:
        url - The url
        fields - The fields to load from local storage
        Returns:

        The page in storage if exists, otherwise returns null

      • getContent

         abstract ByteBuffer getContent(String url)

        Get the content of the page from the storage.

        val url = "http://example.com"
val content = session.getContent(url)
        Parameters:
        url - The url of the page to retrieve
        Returns:

        The page content or null

      • getContentAsString

        @Beta() abstract String getContentAsString(String url)

        Get the content of the page from the storage

        val url = "http://example.com"
val content = session.getContentAsString(url)
        Parameters:
        url - The url of the page to retrieve
        Returns:

        The page content in string format or null

      • exists

         abstract Boolean exists(String url)

        Check if the page exists in the storage.

        val url = "http://example.com"
val exists = session.exists(url)
        Parameters:
        url - The url to check
        Returns:

        true if the page exists, false otherwise

      • fetchState

         abstract CheckState fetchState(WebPage page, LoadOptions options)

        Return the fetch state of the page.

        val url = "http://example.com"
val state = session.fetchState(url)
        Parameters:
        page - The webpage
        options - The load options
        Returns:

        The fetch state of the page

      • open

         abstract WebPage open(String url)

        Open a url.

        This method opens the url immediately, regardless of the previous state of the page.

        val url = "http://example.com"
val page = session.open(url)
        Parameters:
        url - The url to open
        Returns:

        The webpage loaded or NIL

      • open

         abstract WebPage open(String url, PageEventHandlers eventHandlers)

        Open a url with page events.

        This method opens the url immediately, regardless of the previous state of the page.

        val url = "http://example.com"
val event = PrintFlowEvent()
val page = session.open(url, event)
        Parameters:
        url - The url to open
        Returns:

        The webpage loaded or NIL

      • open

         abstract WebPage open(String url, WebDriver driver)

        Open a url with webdriver.

        This method opens the url immediately, regardless of the previous state of the page.

        val url = "http://example.com"
val page = session.open(url, driver)
        Parameters:
        url - The url to open
        Returns:

        The webpage loaded or NIL

      • open

         abstract WebPage open(String url, WebDriver driver, PageEventHandlers eventHandlers)

        Open a url with page events and webdriver.

        This method opens the url immediately, regardless of the previous state of the page.

        val url = "http://example.com"
val event = PrintFlowEvent()
val page = session.open(url, event)
        Parameters:
        url - The url to open
        Returns:

        The webpage loaded or NIL

      • connect

        @Beta() abstract Unit connect(WebDriver driver)

        Connect a webpage to a webdriver.

        val page = session.connect(driver)
        Returns:

        The webpage connected to the webdriver or NIL

      • load

         abstract WebPage load(String url)

        Load an url.

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        Other fetch conditions can be specified by load arguments:

        • expiration

        • page size requirement

        • fields requirement

        • other

        val url = "http://example.com"
val page = session.load(url)
        Parameters:
        url - The url to load
        Returns:

        The webpage loaded or NIL

      • load

         abstract WebPage load(String url, String args)

        Load a url with specified arguments.

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        Fetch conditions can be specified by load arguments:

        • expiration

        • page size requirement

        • fields requirement

        • other

        val url = "http://example.com"
val args = "-expire 1d"
val page = session.load(url, args)
        Parameters:
        url - The url to load
        args - The load arguments
        Returns:

        The webpage loaded or NIL

      • load

         abstract WebPage load(String url, LoadOptions options)

        Load a url with specified options.

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        Fetch conditions can be specified by load arguments:

        • expiration

        • page size requirement

        • fields requirement

        • other

        val url = "http://example.com"
val options = session.options("-expire 1d")
val page = session.load(url, options)
        Parameters:
        url - The url to load
        options - The load options
        Returns:

        The webpage loaded or NIL

      • load

         abstract WebPage load(UrlAware url)

        Load a url with the specified arguments.

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        val url = Hyperlink("http://example.com")
val page = session.load(url)
        Parameters:
        url - The url to load
        Returns:

        The webpage loaded or NIL

      • load

         abstract WebPage load(UrlAware url, String args)

        Load a url with the specified arguments.

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        val url = Hyperlink("http://example.com")
val page = session.load(url, "-expire 1d")
        Parameters:
        url - The url to load
        args - The load arguments
        Returns:

        The webpage loaded or NIL

      • load

         abstract WebPage load(UrlAware url, LoadOptions options)

        Load a url with options.

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        val url = Hyperlink("http://example.com")
val options = session.options("-expire 1d")
val page = session.load(url, options)
        Parameters:
        url - The url to load
        options - The load options
        Returns:

        The webpage loaded or NIL

      • load

         abstract WebPage load(NormURL url)

        Load a normal url.

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        val url = session.normalize("http://example.com")
val page = session.load(url)

        • @param url The normal url

        Returns:

        The webpage loaded or NIL

      • loadDeferred

         abstract WebPage loadDeferred(String url, String args)

        Load a url with specified options.

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.

        val url = "http://example.com"
val args = "-expire 1d"
val page = session.loadDeferred(url, args)
        Parameters:
        url - The url to load
        args - The load options
        Returns:

        The webpage loaded or NIL

      • loadDeferred

         abstract WebPage loadDeferred(String url, LoadOptions options)

        Load a url with specified options.

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.

        val url = "http://example.com"
val options = session.options("-expire 1d")
val page = session.loadDeferred(url, options)
        Parameters:
        url - The url to load
        options - The load options
        Returns:

        The webpage loaded or NIL

      • loadDeferred

         abstract WebPage loadDeferred(UrlAware url, String args)

        Load a url with specified arguments.

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.

        val url = Hyperlink("http://example.com")
val page = session.loadDeferred(url, "-expire 1d")
        Parameters:
        url - The url to load
        args - The load args
        Returns:

        The webpage loaded or NIL

      • loadDeferred

         abstract WebPage loadDeferred(UrlAware url, LoadOptions options)

        Load a url with specified options.

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.

        val url = Hyperlink("http://example.com")
val options = session.options("-expire 1d")
val page = session.loadDeferred(url, options)
        Parameters:
        url - The url to load
        options - The load options
        Returns:

        The webpage loaded or NIL

      • loadDeferred

         abstract WebPage loadDeferred(NormURL url)

        Load a url with specified options

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.

        val url = session.normalize("http://example.com")
val page = session.loadDeferred(url)
        Parameters:
        url - The normal url
        Returns:

        The webpage loaded or NIL

      • loadAll

         abstract List<WebPage> loadAll(Iterable<String> urls)

        Load all urls with specified options

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        val urls = listOf("http://example.com", "http://example.com/1")
val pages = session.loadAll(urls)
        Parameters:
        urls - The urls to load
        Returns:

        The successfully loaded webpages, all failed urls are ignored

      • loadAll

         abstract List<WebPage> loadAll(Iterable<String> urls, String args)

        Load all urls with specified options

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        val urls = listOf("http://example.com", "http://example.com/1")
val pages = session.loadAll(urls, "-expire 1d")
        Parameters:
        urls - The urls to load
        args - The load arguments
        Returns:

        The successfully loaded webpages, all failed urls are ignored

      • loadAll

         abstract List<WebPage> loadAll(Iterable<String> urls, LoadOptions options)

        Load all urls with specified options

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        val urls = listOf("http://example.com", "http://example.com/1")
val pages = session.loadAll(urls, session.options("-expire 1d"))
        Parameters:
        urls - The urls to load
        options - The load options
        Returns:

        The successfully loaded webpages, all failed urls are ignored

      • loadAll

         abstract List<WebPage> loadAll(Collection<UrlAware> urls)

        Load all urls with specified options

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        val urls = listOf(Hyperlink("http://example.com"), Hyperlink("http://example.com/1"))
val pages = session.loadAll(urls)
        Parameters:
        urls - The urls to load
        Returns:

        The successfully loaded webpages, all failed urls are ignored

      • loadAll

         abstract List<WebPage> loadAll(Collection<UrlAware> urls, String args)

        Load all urls with specified options

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        val urls = listOf(Hyperlink("http://example.com"), Hyperlink("http://example.com/1"))
val pages = session.loadAll(urls, "-expire 1d")
        Parameters:
        urls - The urls to load
        args - The load arguments
        Returns:

        The successfully loaded webpages, all failed urls are ignored

      • loadAll

         abstract List<WebPage> loadAll(Collection<UrlAware> urls, LoadOptions options)

        Load all urls with specified options

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        val urls = listOf(Hyperlink("http://example.com"), Hyperlink("http://example.com/1"))
val pages = session.loadAll(urls, session.options("-expire 1d"))
        Parameters:
        urls - The urls to load
        options - The load options
        Returns:

        The successfully loaded webpages, all failed urls are ignored

      • loadAll

         abstract List<WebPage> loadAll(List<NormURL> normUrls)

        Load all normal urls with specified options

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        val urls = listOf("http://example.com", "http://example.com/1").map { session.normalize(it) }
val pages = session.loadAll(urls)
        Parameters:
        normUrls - The normal urls to load
        Returns:

        The successfully loaded webpages, all failed urls are ignored

      • loadAsync

         abstract CompletableFuture<WebPage> loadAsync(String url)

        Load a url in java async style

        String url = "http://example.com";
WebPage page = session.loadAsync(url).join();
        Parameters:
        url - The url to load
        Returns:

        A completable future of webpage

      • loadAsync

         abstract CompletableFuture<WebPage> loadAsync(String url, String args)

        Load a url in java async style

        String url = "http://example.com";
WebPage page = session.loadAsync(url, "-expire 1d").join();
        Parameters:
        url - The url to load
        Returns:

        A completable future of webpage

      • loadAsync

         abstract CompletableFuture<WebPage> loadAsync(String url, LoadOptions options)

        Load a url in java async style

        String url = "http://example.com";
WebPage page = session.loadAsync(url, session.options("-expire 1d")).join();
        Parameters:
        url - The url to load
        Returns:

        A completable future of webpage

      • loadAsync

         abstract CompletableFuture<WebPage> loadAsync(UrlAware url)

        Load a url in java async style

        Hyperlink url = new Hyperlink("http://example.com");
WebPage page = session.loadAsync(url).join();
        Parameters:
        url - The url to load
        Returns:

        A completable future of webpage

      • loadAsync

         abstract CompletableFuture<WebPage> loadAsync(UrlAware url, String args)

        Load a url in java async style

        Hyperlink url = new Hyperlink("http://example.com");
WebPage page = session.loadAsync(url, "-expire 1d").join();
        Parameters:
        url - The url to load
        Returns:

        A completable future of webpage

      • loadAsync

         abstract CompletableFuture<WebPage> loadAsync(UrlAware url, LoadOptions options)

        Load a url in java async style

        Hyperlink url = new Hyperlink("http://example.com");
WebPage page = session.loadAsync(url, session.options("-expire 1d")).join();
        Parameters:
        url - The url to load
        Returns:

        A completable future of webpage

      • loadAsync

         abstract CompletableFuture<WebPage> loadAsync(NormURL url)

        Load a normal url in java async style

        NormURL url = session.normalize("http://example.com");
WebPage page = session.loadAsync(url).join();
        Parameters:
        url - The normal url to load
        Returns:

        A completable future of webpage

      • loadAllAsync

         abstract List<CompletableFuture<WebPage>> loadAllAsync(Iterable<String> urls)

        Load all urls in java async style

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        List<String> urls = Arrays.asList("http://example.com", "http://example.com/1");
CompletableFuture<?>[] pages = session.loadAllAsync(urls).toArray(CompletableFuture<?>[]::new);
CompletableFuture.allOf(futures).join();
        Parameters:
        urls - The normal urls to load
        Returns:

        The completable futures of webpages

      • loadAllAsync

         abstract List<CompletableFuture<WebPage>> loadAllAsync(Iterable<String> urls, String args)

        Load all urls in java async style.

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        List<String> urls = Arrays.asList("http://example.com", "http://example.com/1");
CompletableFuture<?>[] pages = session.loadAllAsync(urls, "-expire 1d").toArray(CompletableFuture<?>[]::new);
CompletableFuture.allOf(futures).join();
        Parameters:
        urls - The normal urls to load
        Returns:

        The completable futures of webpages

      • loadAllAsync

         abstract List<CompletableFuture<WebPage>> loadAllAsync(Iterable<String> urls, LoadOptions options)

        Load all normal urls in java async style.

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        List<String> urls = Arrays.asList("http://example.com", "http://example.com/1");
CompletableFuture<?>[] pages = session.loadAllAsync(urls, session.options("-expire 1d")).toArray(CompletableFuture<?>[]::new);
CompletableFuture.allOf(futures).join();
        Parameters:
        urls - The normal urls to load
        Returns:

        The completable futures of webpages

      • loadAllAsync

         abstract List<CompletableFuture<WebPage>> loadAllAsync(Collection<UrlAware> urls)

        Load all urls in java async style.

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        List<UrlAware> urls = Arrays.asList(new Hyperlink("http://example.com"), new Hyperlink("http://example.com/1"));
CompletableFuture<?>[] pages = session.loadAllAsync(urls).toArray(CompletableFuture<?>[]::new);
CompletableFuture.allOf(futures).join();
        Parameters:
        urls - The normal urls to load
        Returns:

        The completable futures of webpages

      • loadAllAsync

         abstract List<CompletableFuture<WebPage>> loadAllAsync(Collection<UrlAware> urls, String args)

        Load all normal urls in java async style.

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        List<UrlAware> urls = Arrays.asList(new Hyperlink("http://example.com"), new Hyperlink("http://example.com/1"));
CompletableFuture<?>[] pages = session.loadAllAsync(urls, "-expire 1d").toArray(CompletableFuture<?>[]::new);
CompletableFuture.allOf(futures).join();
        Parameters:
        urls - The normal urls to load
        Returns:

        The completable futures of webpages

      • loadAllAsync

         abstract List<CompletableFuture<WebPage>> loadAllAsync(Collection<UrlAware> urls, LoadOptions options)

        Load all normal urls in java async style.

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        List<UrlAware> urls = Arrays.asList(new Hyperlink("http://example.com"), new Hyperlink("http://example.com/1"));
CompletableFuture<?>[] pages = session.loadAllAsync(urls, session.options("-expire 1d")).toArray(CompletableFuture<?>[]::new);
CompletableFuture.allOf(futures).join();
        Parameters:
        urls - The normal urls to load
        Returns:

        The completable futures of webpages

      • loadAllAsync

         abstract List<CompletableFuture<WebPage>> loadAllAsync(List<NormURL> urls)

        Load all normal urls in java async style.

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        List<NormURL> urls = Arrays.asList(session.normalize("http://example.com"), session.normalize("http://example.com/1"));
CompletableFuture<?>[] pages = session.loadAllAsync(urls).toArray(CompletableFuture<?>[]::new);
CompletableFuture.allOf(futures).join();
        Parameters:
        urls - The normal urls to load
        Returns:

        The completable futures of webpages

      • submit

         abstract PulsarSession submit(String url)

        Submit a url to the URL pool, and it will be subsequently processed in the crawl loop.

        A submit operation is non-blocking, meaning it returns immediately without blocking the current thread or suspending the current coroutine.

        session.submit("http://example.com")
PulsarContexts.await()
        Parameters:
        url - The url to submit
        Returns:

        The PulsarSession itself to enabled chained operations

      • submit

         abstract PulsarSession submit(String url, String args)

        Submit a url to the URL pool, and it will be subsequently processed in the crawl loop.

        A submit operation is non-blocking, meaning it returns immediately without blocking the current thread or suspending the current coroutine.

        session.submit("http://example.com", "-expire 1d")
PulsarContexts.await()
        Parameters:
        url - The url to submit
        args - The load arguments
        Returns:

        The PulsarSession itself to enabled chained operations

      • submit

         abstract PulsarSession submit(String url, LoadOptions options)

        Submit a url to the URL pool, and it will be subsequently processed in the crawl loop.

        A submit operation is non-blocking, meaning it returns immediately without blocking the current thread or suspending the current coroutine.

        session.submit("http://example.com", session.options("-expire 1d"))
PulsarContexts.await()

        Submit can be used with event listeners to handle the page events.

        The code snippet below shows how to submit a url with an attached load event handler. This handler is invoked once the page is loaded, and it prints the URL of the page.

        val options = session.options("-expire 1d")
options.eventHandlers.loadEvent.onLoaded.addLast { println(it.url) }
session.submit("http://example.com", options)
PulsarContexts.await()

        The code snippet below shows how to submit a hyperlink with every supported event handlers attached. Each event handler will print a message when its associated event is triggered.

        val event = PrintFlowEvent()
val options = session.options("-expire 1d", event)
session.submit("http://example.com", options)
PulsarContexts.await()
        Parameters:
        url - The url to submit
        options - The load options
        Returns:

        The PulsarSession itself to enabled chained operations

      • submit

         abstract PulsarSession submit(UrlAware url)

        Submit a url to the URL pool, and it will be subsequently processed in the crawl loop.

        A submit operation is non-blocking, meaning it returns immediately without blocking the current thread or suspending the current coroutine.

        Basic usage:

        session.submit(Hyperlink("http://example.com"))
PulsarContexts.await()

        Submit methods are commonly be used in conjunction with event listeners to handle page events.

        The code snippet below shows how to submit a hyperlink with an attached load event handler. This handler is invoked once the page is loaded, and it prints the URL of the page.

        val hyperlink = ListenableHyperlink("http://example.com")
hyperlink.eventHandlers.loadEvent.onLoaded.addLast { page -> println(page.url) }
session.submit(hyperlink)
PulsarContexts.await()

        The code snippet below shows how to submit a hyperlink with a event handler attached that processes the parsed document once the relevant event is triggered.

        val hyperlink = ListenableHyperlink("http://example.com", args = "-parse", event = PrintFlowEvent())
hyperlink.eventHandlers.loadEvent.onHTMLDocumentParsed.addLast { page, document ->
     val title = document.selectFirstOrNull(".title")?.text() ?: "Not found"
     println(title)
}
session.submit(hyperlink)
PulsarContexts.await()

        ParsableHyperlink can be used to simplify the load-and-parse tasks.

        val hyperlink = ParsableHyperlink("http://example.com") { page, document ->
val title = document.selectFirstOrNull(".title")?.text() ?: ""
     println(title)
}
session.submit(hyperlink)
PulsarContexts.await()

        The code snippet below shows how to submit a hyperlink with every supported event handlers attached. Each event handler will print a message when its associated event is triggered.

        val event = PrintFlowEvent()
val options = session.options("-expire 1d", event)
session.submit("http://example.com", options)
PulsarContexts.await()
        Parameters:
        url - The url to submit
        Returns:

        The PulsarSession itself to enabled chained operations

      • submit

         abstract PulsarSession submit(UrlAware url, String args)

        Submit a url to the URL pool, and it will be subsequently processed in the crawl loop.

        A submit operation is non-blocking, meaning it returns immediately without blocking the current thread or suspending the current coroutine.

        session.submit(Hyperlink("http://example.com"), "-expire 1d")
PulsarContexts.await()
        Parameters:
        url - The url to submit
        args - The load arguments
        Returns:

        The PulsarSession itself to enabled chained operations

      • submitAll

         abstract PulsarSession submitAll(Iterable<String> urls)

        Submit urls to the URL pool, and they will be subsequently processed in the crawl loop.

        Submit operations are non-blocking, meaning they return immediately without blocking the current thread or suspending the current coroutine.

        session.submitAll(listOf("http://example.com", "http://example.com/1"))
PulsarContexts.await()
        Parameters:
        urls - The urls to submit
        Returns:

        The PulsarSession itself to enabled chained operations

      • submitAll

         abstract PulsarSession submitAll(Iterable<String> urls, String args)

        Submit urls to the URL pool, and they will be subsequently processed in the crawl loop.

        Submit operations are non-blocking, meaning they return immediately without blocking the current thread or suspending the current coroutine.

        session.submitAll(listOf("http://example.com", "http://example.com/1"), "-expire 1d")
PulsarContexts.await()
        Parameters:
        urls - The urls to submit
        args - The load arguments
        Returns:

        The PulsarSession itself to enabled chained operations

      • submitAll

         abstract PulsarSession submitAll(Iterable<String> urls, LoadOptions options)

        Submit urls to the URL pool, and they will be subsequently processed in the crawl loop.

        Submit operations are non-blocking, meaning they return immediately without blocking the current thread or suspending the current coroutine.

        session.submitAll(listOf("http://example.com", "http://example.com/1"), session.options("-expire 1d"))
PulsarContexts.await()
        Parameters:
        urls - The urls to submit
        options - The load options
        Returns:

        The PulsarSession itself to enabled chained operations

      • submitAll

         abstract PulsarSession submitAll(Collection<UrlAware> urls)

        Submit urls to the URL pool, and they will be subsequently processed in the crawl loop.

        A submit operation is non-blocking, meaning it returns immediately without blocking the current thread or suspending the current coroutine.

        val urls = listOf("http://example.com", "http://example.com/1").map { Hyperlink(it) }
session.submitAll(urls)
PulsarContexts.await()
        Parameters:
        urls - The urls to submit
        Returns:

        The PulsarSession itself to enabled chained operations

      • submitAll

         abstract PulsarSession submitAll(Collection<UrlAware> urls, String args)

        Submit urls to the URL pool, and they will be subsequently processed in the crawl loop.

        A submit operation is non-blocking, meaning it returns immediately without blocking the current thread or suspending the current coroutine.

        val urls = listOf("http://example.com", "http://example.com/1").map { Hyperlink(it) }
session.submitAll(urls, "-expire 1d")
PulsarContexts.await()
        Parameters:
        urls - The urls to submit
        Returns:

        The PulsarSession itself to enabled chained operations

      • loadOutPages

         abstract List<WebPage> loadOutPages(String portalUrl, String args)

        Load or fetch the portal page, and then load or fetch the out links selected by -outLink option.

        Option -outLink specifies the cssSelector for links in the portal page to load.

        val pages = session.loadOutPages("http://example.com", "-outLink a")
val pages2 = session.loadOutPages("http://example.com", "-outLink a -expire 1d")
        Parameters:
        portalUrl - The portal url from where to load pages
        args - The load arguments
        Returns:

        The loaded out pages

      • loadOutPages

         abstract List<WebPage> loadOutPages(String portalUrl, LoadOptions options)

        Load or fetch the portal page, and then load or fetch the out links selected by -outLink option.

        Option -outLink specifies the cssSelector for links in the portal page to load.

        val pages = session.loadOutPages("http://example.com", session.options("-outLink a"))
val pages2 = session.loadOutPages("http://example.com", session.options("-outLink a -expire 1d"))
        Parameters:
        portalUrl - The portal url from where to load pages
        options - The load options
        Returns:

        The loaded out pages

      • loadOutPages

         List<WebPage> loadOutPages(UrlAware portalUrl)

        A confusing version, it's too complicated to handle events and should not be implemented.

      • loadOutPages

         abstract List<WebPage> loadOutPages(UrlAware portalUrl, String args)

        Load or fetch the portal page, and then load or fetch the out links selected by -outLink option.

        Option -outLink specifies the cssSelector for links in the portal page to load.

        val pages = session.loadOutPages("http://example.com", "-outLink a")
val pages2 = session.loadOutPages("http://example.com", "-outLink a -expire 1d")
        Parameters:
        portalUrl - The portal url from where to load pages
        args - The load arguments
        Returns:

        The loaded out pages

      • loadOutPages

         abstract List<WebPage> loadOutPages(UrlAware portalUrl, LoadOptions options)

        Load or fetch the portal page, and then load or fetch the out links selected by -outLink option.

        Option -outLink specifies the cssSelector for links in the portal page to load.

        val pages = session.loadOutPages("http://example.com", session.options("-outLink a"))
val pages2 = session.loadOutPages("http://example.com", session.options("-outLink a -expire 1d"))
        Parameters:
        portalUrl - The portal url from where to load pages
        options - The load options
        Returns:

        The loaded out pages

      • loadOutPages

         List<WebPage> loadOutPages(NormURL portalUrl)

        A confusing version, it's too complicated to handle events and should not be implemented.

      • loadOutPagesAsync

         abstract List<CompletableFuture<WebPage>> loadOutPagesAsync(String portalUrl, String args)

        Load or fetch the portal page, and then load or fetch the out links selected by -outLink option asynchronously.

        Option -outLink specifies the cssSelector for links in the portal page to load.

        val pages = session.loadOutPagesAsync("http://example.com", "-outLink a")
val pages2 = session.loadOutPagesAsync("http://example.com", "-outLink a -expire 1d")
        Parameters:
        portalUrl - The portal url from where to load pages
        args - The load arguments
        Returns:

        The loaded out pages

      • loadOutPagesAsync

         abstract List<CompletableFuture<WebPage>> loadOutPagesAsync(String portalUrl, LoadOptions options)

        Load or fetch the portal page, and then load or fetch the out links selected by -outLink option asynchronously.

        Option -outLink specifies the cssSelector for links in the portal page to load.

        val pages = session.loadOutPagesAsync("http://example.com", session.options("-outLink a"))
val pages2 = session.loadOutPagesAsync("http://example.com", session.options("-outLink a -expire 1d"))
        Parameters:
        portalUrl - The portal url from where to load pages
        options - The load options
        Returns:

        The loaded out pages

      • submitForOutPages

         abstract PulsarSession submitForOutPages(String portalUrl, String args)

        Load the portal page and submit the out links specified by the -outLink option to the URL pool.

        Option -outLink specifies the cssSelector for links in the portal page to load.

        The submitted urls will be subsequently processed in the crawl loop.

        Submit operations are non-blocking, meaning they return immediately without blocking the current thread or suspending the current coroutine.

        session
  .submitForOutPages("http://example.com", "-outLink a[href*=review]")
  .submitForOutPages("http://example.com", "-outLink a[href*=item] -expire 1d")
PulsarContexts.await()
        Parameters:
        portalUrl - The portal url from where to load pages
        args - The load arguments
        Returns:

        The PulsarSession itself to enable chained operation

      • submitForOutPages

         abstract PulsarSession submitForOutPages(String portalUrl, LoadOptions options)

        Load the portal page and submit the out links specified by the -outLink option to the URL pool.

        Option -outLink specifies the cssSelector for links in the portal page to load.

        The submitted urls will be subsequently processed in the crawl loop.

        Submit operations are non-blocking, meaning they return immediately without blocking the current thread or suspending the current coroutine.

        session
  .submitForOutPages("http://example.com", session.options("-outLink a[href*=review]"))
  .submitForOutPages("http://example.com", session.options("-outLink a[href*=item] -expire 1d"))
PulsarContexts.await()
        Parameters:
        portalUrl - The portal url from where to load pages
        options - The load options
        Returns:

        The PulsarSession itself to enable chained operation

      • submitForOutPages

         abstract PulsarSession submitForOutPages(UrlAware portalUrl, String args)

        Load the portal page and submit the out links specified by the -outLink option to the URL pool.

        Option -outLink specifies the cssSelector for links in the portal page to load.

        The submitted urls will be subsequently processed in the crawl loop.

        Submit operations are non-blocking, meaning they return immediately without blocking the current thread or suspending the current coroutine.

        session
  .submitForOutPages(Hyperlink("http://example.com"), "-outLink a[href*=review]")
  .submitForOutPages(Hyperlink("http://example.com"), "-outLink a[href*=item] -expire 1d")
PulsarContexts.await()
        Parameters:
        portalUrl - The portal url from where to load pages
        args - The load arguments
        Returns:

        The PulsarSession itself to enable chained operation

      • submitForOutPages

         abstract PulsarSession submitForOutPages(UrlAware portalUrl, LoadOptions options)

        Load the portal page and submit the out links specified by the -outLink option to the URL pool.

        Option -outLink specifies the cssSelector for links in the portal page to load.

        The submitted urls will be subsequently processed in the crawl loop.

        Submit operations are non-blocking, meaning they return immediately without blocking the current thread or suspending the current coroutine.

        session
  .submitForOutPages(Hyperlink("http://example.com"), session.options("-outLink a[href*=review]"))
  .submitForOutPages(Hyperlink("http://example.com"), session.options("-outLink a[href*=item] -expire 1d"))
PulsarContexts.await()
        Parameters:
        portalUrl - The portal url from where to load pages
        options - The load options
        Returns:

        The PulsarSession itself to enable chained operation

      • loadResource

         abstract WebPage loadResource(String url, String referrer)

        Load a url as a resource without browser rendering.

        Referrer will be opened by a browser first to obtain browsing environment, such as headers and cookies, and then the browsing environment will be applied to the later resource fetching.

        val page = session.loadResource("http://example.com/robots.txt", "http://example.com")
        Parameters:
        url - The url to load
        referrer - The referrer URL
        Returns:

        The webpage containing the resource

      • loadResource

         abstract WebPage loadResource(String url, String referrer, String args)

        Load a url as a resource without browser rendering.

        Referrer will be opened by a browser first to obtain browsing environment, such as headers and cookies, and then the browsing environment will be applied to the later resource fetching.

        val page = session.loadResource("http://example.com/robots.txt", "http://example.com", "-expire 1d")
        Parameters:
        url - The url to load
        referrer - The referrer URL
        args - The load arguments
        Returns:

        The webpage containing the resource

      • loadResource

         abstract WebPage loadResource(String url, String referrer, LoadOptions options)

        Load a url as a resource without browser rendering.

        Referrer will be opened by a browser first to obtain browsing environment, such as headers and cookies, and then the browsing environment will be applied to the later resource fetching.

        val page = session.loadResource("http://example.com/robots.txt", "http://example.com", session.options("-expire 1d"))
        Parameters:
        url - The url to load
        referrer - The referrer URL
        options - The load options
        Returns:

        The webpage containing the resource

      • loadResourceDeferred

         abstract WebPage loadResourceDeferred(String url, String referrer)

        Load a url as a resource without browser rendering.

        Referrer will be opened by a browser first to obtain browsing environment, such as headers and cookies, and then the browsing environment will be applied to the later resource fetching.

        This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.

        val page = session.loadResourceDeferred("http://example.com/robots.txt", "http://example.com")
        Parameters:
        url - The url to load
        referrer - The referrer URL
        Returns:

        The webpage containing the resource

      • loadResourceDeferred

         abstract WebPage loadResourceDeferred(String url, String referrer, String args)

        Load a url as a resource without browser rendering.

        Referrer will be opened by a browser first to obtain browsing environment, such as headers and cookies, and then the browsing environment will be applied to the later resource fetching.

        This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.

        val page = session.loadResourceDeferred("http://example.com/robots.txt", "http://example.com", "-expire 1d")
        Parameters:
        url - The url to load
        referrer - The referrer URL
        args - The load arguments
        Returns:

        The webpage containing the resource

      • loadResourceDeferred

         abstract WebPage loadResourceDeferred(String url, String referrer, LoadOptions options)

        Load a url as a resource without browser rendering.

        Referrer will be opened by a browser first to obtain browsing environment, such as headers and cookies, and then the browsing environment will be applied to the later resource fetching.

        This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.

        val page = session.loadResourceDeferred("http://example.com/robots.txt", "http://example.com", session.options("-expire 1d"))
        Parameters:
        url - The url to load
        referrer - The referrer URL
        options - The load options
        Returns:

        The webpage containing the resource

      • parse

         abstract FeaturedDocument parse(WebPage page)

        Parse a webpage into an HTML document.

        val page = session.load("http://example.com")
val document = session.parse(page)
        Parameters:
        page - The webpage to parse
        Returns:

        The parsed HTML document

      • parse

         abstract FeaturedDocument parse(WebPage page, Boolean noCache)

        Parse a webpage into an HTML document.

        val page = session.load("http://example.com")
val document = session.parse(page, true)
        Parameters:
        page - The webpage to parse
        noCache - Whether to skip the cache
        Returns:

        The parsed HTML document

      • loadDocument

         abstract FeaturedDocument loadDocument(String url)

        Load or fetch a webpage and parse it into an HTML document

        val document = session.loadDocument("http://example.com")
        Parameters:
        url - The url to load
        Returns:

        The parsed HTML document

      • loadDocument

         abstract FeaturedDocument loadDocument(String url, String args)

        Load or fetch a webpage and parse it into an HTML document

        val document = session.loadDocument("http://example.com", "-expire 1d")
        Parameters:
        url - The url to load
        args - The load arguments
        Returns:

        The parsed HTML document

      • loadDocument

         abstract FeaturedDocument loadDocument(String url, LoadOptions options)

        Load or fetch a webpage and parse it into an HTML document

        val document = session.loadDocument("http://example.com", session.options("-expire 1d"))
        Parameters:
        url - The url to load
        options - The load options
        Returns:

        The parsed HTML document

      • loadDocument

         abstract FeaturedDocument loadDocument(UrlAware url)

        Load or fetch a webpage and parse it into an HTML document

        val document = session.loadDocument(Hyperlink("http://example.com"))
        Parameters:
        url - The url to load
        Returns:

        The parsed HTML document

      • loadDocument

         abstract FeaturedDocument loadDocument(UrlAware url, String args)

        Load or fetch a webpage and parse it into an HTML document

        val document = session.loadDocument(Hyperlink("http://example.com"), "-expire 1d")
        Parameters:
        url - The url to load
        args - The load arguments
        Returns:

        The parsed HTML document

      • loadDocument

         abstract FeaturedDocument loadDocument(UrlAware url, LoadOptions options)

        Load or fetch a webpage and parse it into an HTML document

        val document = session.loadDocument(Hyperlink("http://example.com"), session.options("-expire 1d"))
        Parameters:
        url - The url to load
        options - The load options
        Returns:

        The parsed HTML document

      • loadDocument

         abstract FeaturedDocument loadDocument(NormURL url)

        Load or fetch a webpage and then parse it into an HTML document.

        val document = session.loadDocument("http://example.com")
        Parameters:
        url - The url to load
        Returns:

        The parsed HTML document

      • scrape

         abstract Map<String, String> scrape(String url, String args, Iterable<String> fieldSelectors)

        Load or fetch a webpage located by the given url, and then extract fields specified by field selectors.

        val fields = session.scrape("http://example.com", "-expire 1d", listOf(".title", ".content"))
        Parameters:
        url - The url to scrape
        args - The load arguments
        fieldSelectors - The selectors to extract fields
        Returns:

        All the extracted fields and their selectors

      • scrape

         abstract Map<String, String> scrape(String url, LoadOptions options, Iterable<String> fieldSelectors)

        Load or fetch a webpage located by the given url, and then extract fields specified by field selectors.

        val fields = session.scrape("http://example.com", session.options("-expire 1d"), listOf(".title", ".content"))
        Parameters:
        url - The url to scrape
        options - The load options
        fieldSelectors - The selectors to extract fields
        Returns:

        All the extracted fields and their selectors

      • scrape

         abstract Map<String, String> scrape(String url, String args, Map<String, String> fieldSelectors)

        Load or fetch a webpage located by the given url, and then extract fields specified by field selectors.

        val fields = session.scrape("http://example.com", "-expire 1d", ".title", ".content")
        Parameters:
        url - The url to scrape
        args - The load arguments
        fieldSelectors - The selectors to extract fields
        Returns:

        All the extracted fields and their names

      • scrape

         abstract Map<String, String> scrape(String url, LoadOptions options, Map<String, String> fieldSelectors)

        Load or fetch a webpage located by the given url, and then extract fields specified by field selectors.

        val fields = session.scrape("http://example.com", session.options("-expire 1d"), ".title", ".content")
        Parameters:
        url - The url to scrape
        options - The load options
        fieldSelectors - The selectors to extract fields
        Returns:

        All the extracted fields and their names

      • scrape

         abstract List<Map<String, String>> scrape(String url, String args, String restrictSelector, Iterable<String> fieldSelectors)

        Load or fetch a webpage located by the given url, and then extract fields specified by field selectors.

        val fields = session.scrape("http://example.com", "-expire 1d", ".container", listOf(".title", ".content"))
        Parameters:
        url - The url to scrape
        args - The load arguments
        restrictSelector - A CSS selector to locate a DOM where all fields are restricted to
        fieldSelectors - The selectors to extract fields
        Returns:

        All the extracted fields and their names

      • scrape

         abstract List<Map<String, String>> scrape(String url, LoadOptions options, String restrictSelector, Iterable<String> fieldSelectors)

        Load or fetch a webpage located by the given url, and then extract fields specified by field selectors.

        val fields = session.scrape("http://example.com", session.options("-expire 1d"), ".container",
     listOf(".title", ".content"))
        Parameters:
        url - The url to scrape
        options - The load options
        restrictSelector - A CSS selector to locate a DOM where all fields are restricted to
        fieldSelectors - The selectors to extract fields
        Returns:

        All the extracted fields and their selectors

      • scrape

         abstract List<Map<String, String>> scrape(String url, String args, String restrictSelector, Map<String, String> fieldSelectors)

        Load or fetch a webpage located by the given url, and then extract fields specified by field selectors.

        val fields = session.scrape("http://example.com", "-expire 1d", ".container",
     mapOf("title" to ".title", "content" to ".content"))
        Parameters:
        url - The url to scrape
        args - The load arguments
        restrictSelector - A CSS selector to locate a DOM where all fields are restricted to
        fieldSelectors - The selectors to extract fields
        Returns:

        All the extracted fields and their names

      • scrape

         abstract List<Map<String, String>> scrape(String url, LoadOptions options, String restrictSelector, Map<String, String> fieldSelectors)

        Load or fetch a webpage located by the given url, and then extract fields specified by field selectors.

        val fields = session.scrape("http://example.com", session.options("-expire 1d"), ".container",
     mapOf("title" to ".title", "content" to ".content"))
        Parameters:
        url - The url to scrape
        options - The load options
        restrictSelector - A CSS selector to locate a DOM where all fields are restricted to
        fieldSelectors - The selectors to extract fields
        Returns:

        All the extracted fields and their names

      • scrapeOutPages

         abstract List<Map<String, String>> scrapeOutPages(String portalUrl, String args, Iterable<String> fieldSelectors)

        Load or fetch out pages specified by out link selector, and then extract fields specified by field selectors from each out page.

        val fields = session.scrapeOutPages("http://example.com", "-outLink a", listOf(".title", ".content"))
        Parameters:
        portalUrl - The portal url to start scraping
        args - Load arguments for both the portal page and out pages
        fieldSelectors - CSS selectors to extract fields from out pages
        Returns:

        All extracted fields. For each out page, fields extracted with their selectors are saved in a map.

      • scrapeOutPages

         abstract List<Map<String, String>> scrapeOutPages(String portalUrl, LoadOptions options, Iterable<String> fieldSelectors)

        Load or fetch out pages specified by out link selector, and then extract fields specified by field selectors from each out page.

        val fields = session.scrapeOutPages("http://example.com", session.options("-outLink a"), listOf(".title", ".content"))
        Parameters:
        portalUrl - The portal url to start scraping
        options - Load options for both the portal page and out pages
        fieldSelectors - CSS selectors to extract fields from out pages
        Returns:

        All extracted fields. For each out page, fields extracted with their selectors are saved in a map.

      • scrapeOutPages

         abstract List<Map<String, String>> scrapeOutPages(String portalUrl, String args, String restrictSelector, Iterable<String> fieldSelectors)

        Load or fetch out pages specified by out link selector, and then extract fields specified by field selectors from each out page.

        val fields = session.scrapeOutPages("http://example.com", "-outLink a", mapOf("title" to ".title", "content" to ".content"))
        Parameters:
        portalUrl - The portal url to start scraping
        args - Load arguments for both the portal page and out pages
        restrictSelector - A CSS selector to locate a DOM where all fields are restricted to
        fieldSelectors - CSS selectors to extract fields from out pages
        Returns:

        All extracted fields. For each out page, fields extracted with their selectors are saved in a map.

      • scrapeOutPages

         abstract List<Map<String, String>> scrapeOutPages(String portalUrl, LoadOptions options, String restrictSelector, Iterable<String> fieldSelectors)

        Load or fetch out pages specified by out link selector, and then extract fields specified by field selectors from each out page.

        val fields = session.scrapeOutPages("http://example.com", session.options("-outLink a"),
     mapOf("title" to ".title", "content" to ".content"))
        Parameters:
        portalUrl - The portal url to start scraping
        options - Load options for both the portal page and out pages
        restrictSelector - A CSS selector to locate a DOM where all fields are restricted to
        fieldSelectors - CSS selectors to extract fields from out pages
        Returns:

        All extracted fields. For each out page, fields extracted with their selectors are saved in a map.

      • scrapeOutPages

         abstract List<Map<String, String>> scrapeOutPages(String portalUrl, String args, Map<String, String> fieldSelectors)

        Load or fetch out pages specified by out link selector, and then extract fields specified by field selectors from each out page.

        val fields = session.scrapeOutPages("http://example.com", "-outLink a", mapOf("title" to ".title", "content" to ".content"))
        Parameters:
        portalUrl - The portal url to start scraping
        args - Load arguments for both the portal page and out pages
        fieldSelectors - CSS selectors to extract fields from out pages
        Returns:

        All extracted fields. For each out page, fields extracted with their names are saved in a map.

      • scrapeOutPages

         abstract List<Map<String, String>> scrapeOutPages(String portalUrl, LoadOptions options, Map<String, String> fieldSelectors)

        Load or fetch out pages specified by out link selector, and then extract fields specified by field selectors from each out page.

        val fields = session.scrapeOutPages("http://example.com", session.options("-outLink a"),
     mapOf("title" to ".title", "content" to ".content"))
        Parameters:
        portalUrl - The portal url to start scraping
        options - Load options for both the portal page and out pages
        fieldSelectors - CSS selectors to extract fields from out pages
        Returns:

        All extracted fields. For each out page, fields extracted with their names are saved in a map.

      • scrapeOutPages

         abstract List<Map<String, String>> scrapeOutPages(String portalUrl, String args, String restrictSelector, Map<String, String> fieldSelectors)

        Load or fetch out pages specified by out link selector, and then extract fields specified by field selectors from each out page.

        val fields = session.scrapeOutPages("http://example.com", "-outLink a", ".container",
     mapOf("title" to ".title", "content" to ".content"))
        Parameters:
        portalUrl - The portal url to start scraping
        args - Load arguments for both the portal page and out pages
        restrictSelector - A CSS selector to locate a DOM where all fields are restricted to
        fieldSelectors - CSS selectors to extract fields from out pages
        Returns:

        All extracted fields. For each out page, fields extracted with their names are saved in a map.

      • scrapeOutPages

         abstract List<Map<String, String>> scrapeOutPages(String portalUrl, LoadOptions options, String restrictSelector, Map<String, String> fieldSelectors)

        Load or fetch out pages specified by out link selector, and then extract fields specified by field selectors from each out page.

        val fields = session.scrapeOutPages("http://example.com", session.options("-outLink a"), ".container",
     mapOf("title" to ".title", "content" to ".content"))
        Parameters:
        portalUrl - The portal url to start scraping
        options - Load options for both the portal page and out pages
        restrictSelector - A CSS selector to locate a DOM where all fields are restricted to
        fieldSelectors - CSS selectors to extract fields from out pages
        Returns:

        All extracted fields. For each out page, fields extracted with their names are saved in a map.

      • harvest

         abstract TextDocument harvest(String url, String args, String engine)

        Harvest the content of a webpage using a web content extractor engine.

        Available engines are "boilerpipe".

        val document = session.harvest("http://example.com", "-expire 1d")
println(document.textContent)
        Parameters:
        url - The url to harvest
        args - The load arguments
        engine - The content extractor engine, default is "boilerpipe".
        Returns:

        The harvested text document

      • harvest

         abstract TextDocument harvest(WebPage page, String engine)

        Harvest the content of a webpage using a web content extractor engine.

        Available engines are "boilerpipe".

        val page = session.load("http://example.com")
val document = session.harvest(page, "boilerpipe")
println(document.textContent)
        Parameters:
        page - The webpage to harvest
        engine - The content extractor engine, default is "boilerpipe".
        Returns:

        The harvested text document

      • chat

         abstract ModelResponse chat(String prompt)

        Chat with the AI model.

        Parameters:
        prompt - The prompt to chat with
        Returns:

        The response from the model

      • chat

         abstract ModelResponse chat(String userMessage, String systemMessage)

        Chat with the AI model.

        Parameters:
        userMessage - Represents a message from a user, typically an end user of the application.
        systemMessage - Represents a system message, typically defined by a developer.
        Returns:

        The response from the model.

      • chat

         abstract ModelResponse chat(WebPage page, String prompt)

        Chat with the AI model about the specified webpage.

        Parameters:
        page - The page to chat with
        prompt - The prompt to chat with
        Returns:

        The response from the model

      • chat

         abstract ModelResponse chat(String prompt, WebPage page)

        Chat with the AI model about the specified webpage.

        Parameters:
        prompt - The prompt to chat with
        page - The page to chat with
        Returns:

        The response from the model

      • chat

         abstract ModelResponse chat(FeaturedDocument document, String prompt)

        Chat with the AI model about the specified document.

        Parameters:
        document - The document to chat with
        prompt - The prompt to chat with
        Returns:

        The response from the model

      • chat

         abstract ModelResponse chat(String prompt, FeaturedDocument document)

        Chat with the AI model about the specified document.

        Parameters:
        prompt - The prompt to chat with
        document - The document to chat with
        Returns:

        The response from the model

      • chat

         abstract ModelResponse chat(Element element, String prompt)

        Chat with the AI model about the specified element.

        Parameters:
        element - The element to chat with
        prompt - The prompt to chat with
        Returns:

        The response from the model

      • chat

         abstract ModelResponse chat(String prompt, Element element)

        Chat with the AI model about the specified element.

        Parameters:
        prompt - The prompt to chat with
        element - The element to chat with
        Returns:

        The response from the model

      • instruct

         abstract InstructionResult instruct(String prompt, WebDriver driver)

        Instructs the webdriver to perform a series of actions based on the given prompt. This function converts the prompt into a sequence of webdriver actions, which are then executed.

        Parameters:
        prompt - The textual prompt that describes the actions to be performed by the webdriver.
        driver - The webdriver instance that will execute the actions.
        Returns:

        The response from the model, though in this implementation, the return value is not explicitly used.

      • export

         abstract Path export(WebPage page)

        Export the content of a webpage.

        val page = session.load("http://example.com")
val path = session.export(page)
        Parameters:
        page - Page to export
        Returns:

        The path of the exported page

      • export

         abstract Path export(WebPage page, String ident)

        Export the content of a webpage.

        val page = session.load("http://example.com")
val path = session.export(page, "example")
        Parameters:
        page - Page to export
        ident - File name identifier used to distinguish from other files
        Returns:

        The path of the exported page

      • export

         abstract Path export(FeaturedDocument doc)

        Export the outer HTML of the document.

        Parameters:
        doc - Document to export
        Returns:

        The path of the exported document

      • export

         abstract Path export(FeaturedDocument doc, String ident)

        Export the outer HTML of the document.

        val document = session.loadDocument("http://example.com")
val path = session.export(document, "example")
        Parameters:
        doc - Document to export
        ident - File name identifier used to distinguish from other files
        Returns:

        The path of the exported document

      • exportTo

         abstract Path exportTo(WebPage page, Path path)

        Export the content of a webpage.

        val page = session.load("http://example.com")
val path = session.exportTo(page, Paths.get("/tmp/example.html"))
        Parameters:
        page - Webpage to export
        path - Path to save the exported content
        Returns:

        The path of the exported document

      • exportTo

         abstract Path exportTo(FeaturedDocument doc, Path path)

        Export the outer HTML of the document.

        val document = session.loadDocument("http://example.com")
val path = session.exportTo(document, Paths.get("/tmp/example.html"))
        Parameters:
        doc - Document to export
        path - Path to save the exported content
        Returns:

        The path of the exported document

      • persist

         abstract Boolean persist(WebPage page)

        Persist the webpage.

        val page = session.load("http://example.com")
session.persist(page)
        Parameters:
        page - Page to persist
        Returns:

        Whether the page is persisted successfully

      • delete

         abstract Unit delete(String url)

        Delete a webpage from the storage

        session.delete("http://example.com")
        Parameters:
        url - The url to delete

      • flush

         abstract Unit flush()

        Flush to the storage

      • getUnmodifiedConfig

         abstract ImmutableConfig getUnmodifiedConfig()

        This is an immutable configuration, loaded from the configuration file during process startup. Once the process has started, this configuration remains unchangeable.

      • getSessionConfig

         abstract VolatileConfig getSessionConfig()

        The session-specific volatile configuration, which allows dynamic adjustments to settings at any point during the session. Unlike the immutable configuration loaded at startup, this configuration is designed to be modified on-the-fly to adapt to runtime requirements.

      • getPageCache

         abstract ConcurrentExpiringLRUCache<String, WebPage> getPageCache()

        The global page cache.

      • getDocumentCache

         abstract ConcurrentExpiringLRUCache<String, FeaturedDocument> getDocumentCache()

        The global document cache.