Package ai.platon.pulsar.skeleton.session

Interface PulsarSession

  • All Implemented Interfaces:
    java.lang.AutoCloseable 
    
public interface PulsarSession
 implements AutoCloseable

    The PulsarSession interface is designed to provide methods for loading pages from either local storage or from the internet. Additionally, it offers functionality for parsing these pages, extracting relevant data, and saving or exporting the web content.

    Key methods:

    • load: load a page from local storage or the Internet.

    • parse: parse a page into a document.

    • submit: submit a url to the URL pool, the url will be processed in the main loop later.

    • scrape: load a page, parse it into a document and then extract fields from the document.

    Basic examples:

    val url = "http://example.com"
val page = session.load(url)
val document = session.parse(page)

val title = document.selectFirstTextOrNull(".title")
val content = document.selectFirstTextOrNull(".content")

val fields = session.scrape("http://example.com", "-expire 1d", listOf(".title", ".content"))

val document2 = session.loadDocument(url)

val url2 = "http://example.com/1"
session.submit(url2)

    And also the batch versions:

    val urls = listOf("http://example.com", "http://example.com/1")
val pages = session.loadAll(urls)

val pages2 = session.loadOutPages("http://example.com", "-outLink a[rel='next']")
val fields = session.scrapeOutPages("http://example.com", "-outLink a", listOf("title", "content"))

session.submitAll(urls)
session.submitForOutPages("http://example.com", "-outLink a[rel='next']")

    The primary consideration is understanding how to load a page efficiently. Load methods, such as load, initially check local storage for the required page. If the page exists locally and meets the necessary criteria, it is retrieved from the local storage; otherwise, the page is fetched from the internet.

    To load a page, Browser4 follows these steps:

    • Check the local storage for the required page.

    • If the page exists in local storage and meets the specified criteria (such as being up-to-date or valid), return the local version.

    • If the page does not exist in local storage or does not meet the criteria, fetch the page from the internet.

    • Handle any errors that occur during the process, such as issues with local storage access or problems with the network fetch.

    To optimize the retrieval of webpages, load parameters or load options are used to establish the conditions under which a page should be fetched from the internet.

    These conditions may encompass:

    • Expiration timestamps to determine if the local page is still valid.

    • A directive for a forced refresh to ensure the latest content is retrieved.

    • Specifications regarding the minimal acceptable page size.

    • Identifiers for essential fields that must be present in the page.

    • Additional custom criteria that align with the application’s needs.

    For example:

    val url = "http://example.com"
val page = session.load(url, "-expire 1d")

val url2 = "http://example.com/1"
val page2 = session.load(url2, "-refresh")

val url3 = "http://example.com/2"
val page3 = session.load(url3, "-requireSize 100000")

    Once a page has been retrieved from either local storage or the internet, the subsequent processing steps include:

    • Parsing the page content to construct an HTML document.

    • Extracting relevant data fields from the parsed HTML document.

    • Recording the extracted data to a designated destination.

    For example:

    val page = session.load("http://example.com")
val document = session.parse(page)

val title = document.selectFirstTextOrNull(".title")
val content = document.selectFirstTextOrNull(".content")

val path = session.exportTo(page, Paths.get("/tmp/example.html"))

    There are many ways to fetch the content of a page from the Internet, but the two primary methods are:

    • through HTTP protocol

    • through a real browser

    Since the webpages are becoming more and more complex, fetching webpages through real browsers is the primary way nowadays.

    When we fetch webpages using a real browser, sometimes we need to interact with pages to ensure the required fields are loaded correctly and completely. Enable PageEventHandlers and use WebDriver to archive such purpose.

    val options = session.options(args)
options.eventHandlers.browseEventHandlers.onDocumentSteady.addLast { page, driver ->
  driver.fill("input[name='search']", "geek")
  driver.click("button[type='submit']")
}
session.load(url, options)

    WebDriver offers a comprehensive method set for browser automation, meticulously designed to replicate real human actions and behaviors with precision.

    • Constructor Detail

    • Method Detail

      • data

         abstract Object data(String name)

        Get a variable which is stored in this session

        Parameters:
        name - The name of the variable
        Returns:

        The value of the variable

      • data

         abstract Unit data(String name, Object value)

        Store a variable in this session

        Parameters:
        name - The name of the variable
        value - The value of the variable

      • normalize

         abstract NormURL normalize(String url)

        Normalize a url.

        Parameters:
        url - The url to normalize
        Returns:

        The normalized url

      • normalize

         abstract NormURL normalize(String url, String args, Boolean toItemOption)

        Normalize a url.

        Parameters:
        url - The url to normalize
        args - The arguments
        toItemOption - If the LoadOptions is converted to item load options
        Returns:

        The normalized url

      • normalize

         abstract NormURL normalize(String url, LoadOptions options, Boolean toItemOption)

        Normalize a url.

        Parameters:
        url - The url to normalize
        options - The LoadOptions applied to the url
        toItemOption - If the LoadOptions is converted to item load options
        Returns:

        The normalized url

      • normalize

         abstract List<NormURL> normalize(Iterable<String> urls)

        Normalize urls, remove invalid ones.

        Parameters:
        urls - The urls to normalize
        Returns:

        All normalized urls

      • normalize

         abstract List<NormURL> normalize(Iterable<String> urls, String args, Boolean toItemOption)

        Normalize urls, remove invalid ones.

        Parameters:
        urls - The urls to normalize
        args - The arguments
        toItemOption - If the LoadOptions is converted to item load options
        Returns:

        All normalized urls

      • normalize

         abstract List<NormURL> normalize(Iterable<String> urls, LoadOptions options, Boolean toItemOption)

        Normalize urls, remove invalid ones.

        Parameters:
        urls - The urls to normalize
        options - The LoadOptions applied to each url
        toItemOption - If the LoadOptions is converted to item load options
        Returns:

        All normalized urls

      • normalize

         abstract NormURL normalize(UrlAware url)

        Normalize a url, return NormURL.NIL if the url is invalid.

        Parameters:
        url - The url to normalize
        Returns:

        The normalized url

      • normalize

         abstract NormURL normalize(UrlAware url, String args, Boolean toItemOption)

        Normalize a url, return NormURL.NIL if the url is invalid.

        Parameters:
        url - The url to normalize
        args - The arguments
        toItemOption - If the LoadOptions is converted to item load options
        Returns:

        The normalized url

      • normalize

         abstract NormURL normalize(UrlAware url, LoadOptions options, Boolean toItemOption)

        Normalize a url, return NormURL.NIL if the url is invalid.

        Parameters:
        url - The url to normalize
        options - The LoadOptions applied to the url
        toItemOption - If the LoadOptions is converted to item load options
        Returns:

        The normalized url

      • normalize

         abstract List<NormURL> normalize(Collection<UrlAware> urls)

        Normalize urls, remove invalid ones.

        Parameters:
        urls - The urls to normalize
        Returns:

        All normalized urls

      • normalize

         abstract List<NormURL> normalize(Collection<UrlAware> urls, String args, Boolean toItemOption)

        Normalize urls, remove invalid ones

        Parameters:
        urls - The urls to normalize
        args - The arguments
        toItemOption - If the LoadOptions is converted to item load options
        Returns:

        All normalized urls, all invalid input urls are removed

      • normalize

         abstract List<NormURL> normalize(Collection<UrlAware> urls, LoadOptions options, Boolean toItemOption)

        Normalize urls, remove invalid ones

        Parameters:
        urls - The urls to normalize
        options - The LoadOptions applied to each url
        toItemOption - If the LoadOptions is converted to item load options
        Returns:

        All normalized urls, all invalid input urls are removed

      • normalizeOrNull

         abstract NormURL normalizeOrNull(String url, LoadOptions options, Boolean toItemOption)

        Normalize a url, return null if the url is invalid.

        Parameters:
        url - The url to normalize
        options - The LoadOptions applied to the url
        toItemOption - If the LoadOptions is converted to item load options
        Returns:

        The normalized url or null

      • normalizeOrNull

         abstract NormURL normalizeOrNull(UrlAware url, LoadOptions options, Boolean toItemOption)

        Normalize a url, return null if the url is invalid.

        Parameters:
        url - The url to normalize
        options - The LoadOptions applied to the url
        toItemOption - If the LoadOptions is converted to item load options
        Returns:

        The normalized url or null

      • get

         abstract WebPage get(String url)

        Get a page from local storage.

        val url = "http://example.com"
val page = session.get(url)
        Parameters:
        url - The url
        Returns:

        The webpage in storage if exists, otherwise returns a NIL page

      • get

         abstract WebPage get(String url, String fields)

        Get a page from local storage.

        val url = "http://example.com"
val page = session.get(url, "content", "metadata")
        Parameters:
        url - The url
        fields - The fields to load from local storage
        Returns:

        The webpage in storage if exists, otherwise returns a NIL page

      • getOrNull

         abstract WebPage getOrNull(String url)

        Get a page from local storage.

        val url = "http://example.com"
val page = session.getOrNull(url)
        Parameters:
        url - The url
        Returns:

        The page in storage if exists, otherwise returns null

      • getOrNull

         abstract WebPage getOrNull(String url, String fields)

        Get a page from local storage.

        val url = "http://example.com"
val page = session.getOrNull(url, "content", "metadata")
        Parameters:
        url - The url
        fields - The fields to load from local storage
        Returns:

        The page in storage if exists, otherwise returns null

      • getContent

         abstract ByteBuffer getContent(String url)

        Get the content of the page from the storage.

        val url = "http://example.com"
val content = session.getContent(url)
        Parameters:
        url - The url of the page to retrieve
        Returns:

        The page content or null

      • getContentAsString

        @Beta() abstract String getContentAsString(String url)

        Get the content of the page from the storage

        val url = "http://example.com"
val content = session.getContentAsString(url)
        Parameters:
        url - The url of the page to retrieve
        Returns:

        The page content in string format or null

      • exists

         abstract Boolean exists(String url)

        Check if the page exists in the storage.

        val url = "http://example.com"
val exists = session.exists(url)
        Parameters:
        url - The url to check
        Returns:

        true if the page exists, false otherwise

      • fetchState

         abstract CheckState fetchState(WebPage page, LoadOptions options)

        Return the fetch state of the page.

        val url = "http://example.com"
val state = session.fetchState(url)
        Parameters:
        page - The webpage
        options - The load options
        Returns:

        The fetch state of the page

      • open

         abstract WebPage open(String url)

        Open a url.

        This method opens the url immediately, regardless of the previous state of the page.

        val url = "http://example.com"
val page = session.open(url)
        Parameters:
        url - The url to open
        Returns:

        The webpage loaded or NIL

      • open

         abstract WebPage open(String url, PageEventHandlers eventHandlers)

        Open a url with page events.

        This method opens the url immediately, regardless of the previous state of the page.

        val url = "http://example.com"
val event = PrintFlowEvent()
val page = session.open(url, event)
        Parameters:
        url - The url to open
        Returns:

        The webpage loaded or NIL

      • open

         abstract WebPage open(String url, WebDriver driver)

        Open a url with webdriver.

        This method opens the url immediately, regardless of the previous state of the page.

        val url = "http://example.com"
val page = session.open(url, driver)
        Parameters:
        url - The url to open
        Returns:

        The webpage loaded or NIL

      • open

         abstract WebPage open(String url, WebDriver driver, PageEventHandlers eventHandlers)

        Open a url with page events and webdriver.

        This method opens the url immediately, regardless of the previous state of the page.

        val url = "http://example.com"
val event = PrintFlowEvent()
val page = session.open(url, event)
        Parameters:
        url - The url to open
        Returns:

        The webpage loaded or NIL

      • capture

         abstract WebPage capture(WebDriver driver, String url, PageEventHandlers eventHandlers)

        Captures the live page currently controlled by the given WebDriver and produces a local static WebPage instance representing its current state.

        val url = driver.currentUrl()
val page = session.capture(url, driver)

        An optional URL can be provided to identify the webpage, If url is not provided, set WebPage's url to driver.currentUrl(). The url will be normalized to identify the webpage and can differ from the active page's URI since the active page may goto or redirect to another page.

        Parameters:
        driver - The WebDriver instance controlling the live browser page.
        url - Optional URL to identify the webpage.
        Returns:

        A WebPage object containing the static representation of the live page, including DOM structure and referenced resources.

      • bindDriver

         abstract Unit bindDriver(WebDriver driver)

        Bind a webdriver to the session.

        session.bindDriver(driver)

      • bindBrowser

         abstract Unit bindBrowser(Browser browser)

        Bind a browser to the session.

        session.bindBrowser(driver)

      • load

         abstract WebPage load(String url)

        Load a url.

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        Other fetch conditions can be specified by load arguments:

        • expiration

        • page size requirement

        • fields requirement

        • other

        val url = "http://example.com"
val page = session.load(url)
        Parameters:
        url - The url to load
        Returns:

        The webpage loaded or NIL

      • load

         abstract WebPage load(String url, String args)

        Load a url with specified arguments.

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        Fetch conditions can be specified by load arguments:

        • expiration

        • page size requirement

        • fields requirement

        • other

        val url = "http://example.com"
val args = "-expire 1d"
val page = session.load(url, args)
        Parameters:
        url - The url to load
        args - The load arguments
        Returns:

        The webpage loaded or NIL

      • load

         abstract WebPage load(String url, LoadOptions options)

        Load a url with specified options.

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        Fetch conditions can be specified by load arguments:

        • expiration

        • page size requirement

        • fields requirement

        • other

        val url = "http://example.com"
val options = session.options("-expire 1d")
val page = session.load(url, options)
        Parameters:
        url - The url to load
        options - The load options
        Returns:

        The webpage loaded or NIL

      • load

         abstract WebPage load(UrlAware url)

        Load a url with the specified arguments.

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        val url = Hyperlink("http://example.com")
val page = session.load(url)
        Parameters:
        url - The url to load
        Returns:

        The webpage loaded or NIL

      • load

         abstract WebPage load(UrlAware url, String args)

        Load a url with the specified arguments.

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        val url = Hyperlink("http://example.com")
val page = session.load(url, "-expire 1d")
        Parameters:
        url - The url to load
        args - The load arguments
        Returns:

        The webpage loaded or NIL

      • load

         abstract WebPage load(UrlAware url, LoadOptions options)

        Load a url with options.

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        val url = Hyperlink("http://example.com")
val options = session.options("-expire 1d")
val page = session.load(url, options)
        Parameters:
        url - The url to load
        options - The load options
        Returns:

        The webpage loaded or NIL

      • load

         abstract WebPage load(NormURL url)

        Load a normal url.

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        val url = session.normalize("http://example.com")
val page = session.load(url)

        • @param url The normal url

        Returns:

        The webpage loaded or NIL

      • loadDeferred

         abstract WebPage loadDeferred(String url, String args)

        Load a url with specified options.

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.

        val url = "http://example.com"
val args = "-expire 1d"
val page = session.loadDeferred(url, args)
        Parameters:
        url - The url to load
        args - The load options
        Returns:

        The webpage loaded or NIL

      • loadDeferred

         abstract WebPage loadDeferred(String url, LoadOptions options)

        Load a url with specified options.

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.

        val url = "http://example.com"
val options = session.options("-expire 1d")
val page = session.loadDeferred(url, options)
        Parameters:
        url - The url to load
        options - The load options
        Returns:

        The webpage loaded or NIL

      • loadDeferred

         abstract WebPage loadDeferred(UrlAware url, String args)

        Load a url with specified arguments.

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.

        val url = Hyperlink("http://example.com")
val page = session.loadDeferred(url, "-expire 1d")
        Parameters:
        url - The url to load
        args - The load args
        Returns:

        The webpage loaded or NIL

      • loadDeferred

         abstract WebPage loadDeferred(UrlAware url, LoadOptions options)

        Load a url with specified options.

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.

        val url = Hyperlink("http://example.com")
val options = session.options("-expire 1d")
val page = session.loadDeferred(url, options)
        Parameters:
        url - The url to load
        options - The load options
        Returns:

        The webpage loaded or NIL

      • loadDeferred

         abstract WebPage loadDeferred(NormURL url)

        Load a url with specified options

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.

        val url = session.normalize("http://example.com")
val page = session.loadDeferred(url)
        Parameters:
        url - The normal url
        Returns:

        The webpage loaded or NIL

      • loadAll

         abstract List<WebPage> loadAll(Iterable<String> urls)

        Load all urls with specified options

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        val urls = listOf("http://example.com", "http://example.com/1")
val pages = session.loadAll(urls)
        Parameters:
        urls - The urls to load
        Returns:

        The successfully loaded webpages, all failed urls are ignored

      • loadAll

         abstract List<WebPage> loadAll(Iterable<String> urls, String args)

        Load all urls with specified options

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        val urls = listOf("http://example.com", "http://example.com/1")
val pages = session.loadAll(urls, "-expire 1d")
        Parameters:
        urls - The urls to load
        args - The load arguments
        Returns:

        The successfully loaded webpages, all failed urls are ignored

      • loadAll

         abstract List<WebPage> loadAll(Iterable<String> urls, LoadOptions options)

        Load all urls with specified options

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        val urls = listOf("http://example.com", "http://example.com/1")
val pages = session.loadAll(urls, session.options("-expire 1d"))
        Parameters:
        urls - The urls to load
        options - The load options
        Returns:

        The successfully loaded webpages, all failed urls are ignored

      • loadAll

         abstract List<WebPage> loadAll(Collection<UrlAware> urls)

        Load all urls with specified options

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        val urls = listOf(Hyperlink("http://example.com"), Hyperlink("http://example.com/1"))
val pages = session.loadAll(urls)
        Parameters:
        urls - The urls to load
        Returns:

        The successfully loaded webpages, all failed urls are ignored

      • loadAll

         abstract List<WebPage> loadAll(Collection<UrlAware> urls, String args)

        Load all urls with specified options

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        val urls = listOf(Hyperlink("http://example.com"), Hyperlink("http://example.com/1"))
val pages = session.loadAll(urls, "-expire 1d")
        Parameters:
        urls - The urls to load
        args - The load arguments
        Returns:

        The successfully loaded webpages, all failed urls are ignored

      • loadAll

         abstract List<WebPage> loadAll(Collection<UrlAware> urls, LoadOptions options)

        Load all urls with specified options

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        val urls = listOf(Hyperlink("http://example.com"), Hyperlink("http://example.com/1"))
val pages = session.loadAll(urls, session.options("-expire 1d"))
        Parameters:
        urls - The urls to load
        options - The load options
        Returns:

        The successfully loaded webpages, all failed urls are ignored

      • loadAll

         abstract List<WebPage> loadAll(List<NormURL> normUrls)

        Load all normal urls with specified options

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        val urls = listOf("http://example.com", "http://example.com/1").map { session.normalize(it) }
val pages = session.loadAll(urls)
        Parameters:
        normUrls - The normal urls to load
        Returns:

        The successfully loaded webpages, all failed urls are ignored

      • loadAsync

         abstract CompletableFuture<WebPage> loadAsync(String url)

        Load a url in java async style

        String url = "http://example.com";
WebPage page = session.loadAsync(url).join();
        Parameters:
        url - The url to load
        Returns:

        A completable future of webpage

      • loadAsync

         abstract CompletableFuture<WebPage> loadAsync(String url, String args)

        Load a url in java async style

        String url = "http://example.com";
WebPage page = session.loadAsync(url, "-expire 1d").join();
        Parameters:
        url - The url to load
        Returns:

        A completable future of webpage

      • loadAsync

         abstract CompletableFuture<WebPage> loadAsync(String url, LoadOptions options)

        Load a url in java async style

        String url = "http://example.com";
WebPage page = session.loadAsync(url, session.options("-expire 1d")).join();
        Parameters:
        url - The url to load
        Returns:

        A completable future of webpage

      • loadAsync

         abstract CompletableFuture<WebPage> loadAsync(UrlAware url)

        Load a url in java async style

        Hyperlink url = new Hyperlink("http://example.com");
WebPage page = session.loadAsync(url).join();
        Parameters:
        url - The url to load
        Returns:

        A completable future of webpage

      • loadAsync

         abstract CompletableFuture<WebPage> loadAsync(UrlAware url, String args)

        Load a url in java async style

        Hyperlink url = new Hyperlink("http://example.com");
WebPage page = session.loadAsync(url, "-expire 1d").join();
        Parameters:
        url - The url to load
        Returns:

        A completable future of webpage

      • loadAsync

         abstract CompletableFuture<WebPage> loadAsync(UrlAware url, LoadOptions options)

        Load a url in java async style

        Hyperlink url = new Hyperlink("http://example.com");
WebPage page = session.loadAsync(url, session.options("-expire 1d")).join();
        Parameters:
        url - The url to load
        Returns:

        A completable future of webpage

      • loadAsync

         abstract CompletableFuture<WebPage> loadAsync(NormURL url)

        Load a normal url in java async style

        NormURL url = session.normalize("http://example.com");
WebPage page = session.loadAsync(url).join();
        Parameters:
        url - The normal url to load
        Returns:

        A completable future of webpage

      • loadAllAsync

         abstract List<CompletableFuture<WebPage>> loadAllAsync(Iterable<String> urls)

        Load all urls in java async style

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        List<String> urls = Arrays.asList("http://example.com", "http://example.com/1");
CompletableFuture<?>[] pages = session.loadAllAsync(urls).toArray(CompletableFuture<?>[]::new);
CompletableFuture.allOf(futures).join();
        Parameters:
        urls - The normal urls to load
        Returns:

        The completable futures of webpages

      • loadAllAsync

         abstract List<CompletableFuture<WebPage>> loadAllAsync(Iterable<String> urls, String args)

        Load all urls in java async style.

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        List<String> urls = Arrays.asList("http://example.com", "http://example.com/1");
CompletableFuture<?>[] pages = session.loadAllAsync(urls, "-expire 1d").toArray(CompletableFuture<?>[]::new);
CompletableFuture.allOf(futures).join();
        Parameters:
        urls - The normal urls to load
        Returns:

        The completable futures of webpages

      • loadAllAsync

         abstract List<CompletableFuture<WebPage>> loadAllAsync(Iterable<String> urls, LoadOptions options)

        Load all normal urls in java async style.

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        List<String> urls = Arrays.asList("http://example.com", "http://example.com/1");
CompletableFuture<?>[] pages = session.loadAllAsync(urls, session.options("-expire 1d")).toArray(CompletableFuture<?>[]::new);
CompletableFuture.allOf(futures).join();
        Parameters:
        urls - The normal urls to load
        Returns:

        The completable futures of webpages

      • loadAllAsync

         abstract List<CompletableFuture<WebPage>> loadAllAsync(Collection<UrlAware> urls)

        Load all urls in java async style.

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        List<UrlAware> urls = Arrays.asList(new Hyperlink("http://example.com"), new Hyperlink("http://example.com/1"));
CompletableFuture<?>[] pages = session.loadAllAsync(urls).toArray(CompletableFuture<?>[]::new);
CompletableFuture.allOf(futures).join();
        Parameters:
        urls - The normal urls to load
        Returns:

        The completable futures of webpages

      • loadAllAsync

         abstract List<CompletableFuture<WebPage>> loadAllAsync(Collection<UrlAware> urls, String args)

        Load all normal urls in java async style.

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        List<UrlAware> urls = Arrays.asList(new Hyperlink("http://example.com"), new Hyperlink("http://example.com/1"));
CompletableFuture<?>[] pages = session.loadAllAsync(urls, "-expire 1d").toArray(CompletableFuture<?>[]::new);
CompletableFuture.allOf(futures).join();
        Parameters:
        urls - The normal urls to load
        args - The load arguments
        Returns:

        The completable futures of webpages

      • loadAllAsync

         abstract List<CompletableFuture<WebPage>> loadAllAsync(Collection<UrlAware> urls, LoadOptions options)

        Load all normal urls in java async style.

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        List<UrlAware> urls = Arrays.asList(new Hyperlink("http://example.com"), new Hyperlink("http://example.com/1"));
CompletableFuture<?>[] pages = session.loadAllAsync(urls, session.options("-expire 1d")).toArray(CompletableFuture<?>[]::new);
CompletableFuture.allOf(futures).join();
        Parameters:
        urls - The normal urls to load
        options - The load options
        Returns:

        The completable futures of webpages

      • loadAllAsync

         abstract List<CompletableFuture<WebPage>> loadAllAsync(List<NormURL> urls)

        Load all normal urls in java async style.

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        List<NormURL> urls = Arrays.asList(session.normalize("http://example.com"), session.normalize("http://example.com/1"));
CompletableFuture<?>[] pages = session.loadAllAsync(urls).toArray(CompletableFuture<?>[]::new);
CompletableFuture.allOf(futures).join();
        Parameters:
        urls - The normal urls to load
        Returns:

        The completable futures of webpages

      • submit

         abstract PulsarSession submit(String url)

        Submit a url to the URL pool, and it will be subsequently processed in the crawl loop.

        A submit operation is non-blocking, meaning it returns immediately without blocking the current thread or suspending the current coroutine.

        session.submit("http://example.com")
PulsarContexts.await()
        Parameters:
        url - The url to submit
        Returns:

        The PulsarSession itself to enabled chained operations

      • submit

         abstract PulsarSession submit(String url, String args)

        Submit a url to the URL pool, and it will be subsequently processed in the crawl loop.

        A submit operation is non-blocking, meaning it returns immediately without blocking the current thread or suspending the current coroutine.

        session.submit("http://example.com", "-expire 1d")
PulsarContexts.await()
        Parameters:
        url - The url to submit
        args - The load arguments
        Returns:

        The PulsarSession itself to enabled chained operations

      • submit

         abstract PulsarSession submit(String url, LoadOptions options)

        Submit a url to the URL pool, and it will be subsequently processed in the crawl loop.

        A submit operation is non-blocking, meaning it returns immediately without blocking the current thread or suspending the current coroutine.

        session.submit("http://example.com", session.options("-expire 1d"))
PulsarContexts.await()

        Submit can be used with event listeners to handle the page events.

        The code snippet below shows how to submit a url with an attached load event handler. This handler is invoked once the page is loaded, and it prints the URL of the page.

        val options = session.options("-expire 1d")
options.eventHandlers.loadEvent.onLoaded.addLast { println(it.url) }
session.submit("http://example.com", options)
PulsarContexts.await()

        The code snippet below shows how to submit a hyperlink with every supported event handlers attached. Each event handler will print a message when its associated event is triggered.

        val event = PrintFlowEvent()
val options = session.options("-expire 1d", event)
session.submit("http://example.com", options)
PulsarContexts.await()
        Parameters:
        url - The url to submit
        options - The load options
        Returns:

        The PulsarSession itself to enabled chained operations

      • submit

         abstract PulsarSession submit(UrlAware url)

        Submit a url to the URL pool, and it will be subsequently processed in the crawl loop.

        A submit operation is non-blocking, meaning it returns immediately without blocking the current thread or suspending the current coroutine.

        Basic usage:

        session.submit(Hyperlink("http://example.com"))
PulsarContexts.await()

        Submit methods are commonly be used in conjunction with event listeners to handle page events.

        The code snippet below shows how to submit a hyperlink with an attached load event handler. This handler is invoked once the page is loaded, and it prints the URL of the page.

        val hyperlink = ListenableHyperlink("http://example.com")
hyperlink.eventHandlers.loadEvent.onLoaded.addLast { page -> println(page.url) }
session.submit(hyperlink)
PulsarContexts.await()

        The code snippet below shows how to submit a hyperlink with a event handler attached that processes the parsed document once the relevant event is triggered.

        val hyperlink = ListenableHyperlink("http://example.com", args = "-parse", event = PrintFlowEvent())
hyperlink.eventHandlers.loadEvent.onHTMLDocumentParsed.addLast { page, document ->
     val title = document.selectFirstOrNull(".title")?.text() ?: "Not found"
     println(title)
}
session.submit(hyperlink)
PulsarContexts.await()

        ParsableHyperlink can be used to simplify the load-and-parse tasks.

        val hyperlink = ParsableHyperlink("http://example.com") { page, document ->
val title = document.selectFirstOrNull(".title")?.text() ?: ""
     println(title)
}
session.submit(hyperlink)
PulsarContexts.await()

        The code snippet below shows how to submit a hyperlink with every supported event handlers attached. Each event handler will print a message when its associated event is triggered.

        val event = PrintFlowEvent()
val options = session.options("-expire 1d", event)
session.submit("http://example.com", options)
PulsarContexts.await()
        Parameters:
        url - The url to submit
        Returns:

        The PulsarSession itself to enabled chained operations

      • submit

         abstract PulsarSession submit(UrlAware url, String args)

        Submit a url to the URL pool, and it will be subsequently processed in the crawl loop.

        A submit operation is non-blocking, meaning it returns immediately without blocking the current thread or suspending the current coroutine.

        session.submit(Hyperlink("http://example.com"), "-expire 1d")
PulsarContexts.await()
        Parameters:
        url - The url to submit
        args - The load arguments
        Returns:

        The PulsarSession itself to enabled chained operations

      • submitAll

         abstract PulsarSession submitAll(Iterable<String> urls)

        Submit urls to the URL pool, and they will be subsequently processed in the crawl loop.

        Submit operations are non-blocking, meaning they return immediately without blocking the current thread or suspending the current coroutine.

        session.submitAll(listOf("http://example.com", "http://example.com/1"))
PulsarContexts.await()
        Parameters:
        urls - The urls to submit
        Returns:

        The PulsarSession itself to enabled chained operations

      • submitAll

         abstract PulsarSession submitAll(Iterable<String> urls, String args)

        Submit urls to the URL pool, and they will be subsequently processed in the crawl loop.

        Submit operations are non-blocking, meaning they return immediately without blocking the current thread or suspending the current coroutine.

        session.submitAll(listOf("http://example.com", "http://example.com/1"), "-expire 1d")
PulsarContexts.await()
        Parameters:
        urls - The urls to submit
        args - The load arguments
        Returns:

        The PulsarSession itself to enabled chained operations

      • submitAll

         abstract PulsarSession submitAll(Iterable<String> urls, LoadOptions options)

        Submit urls to the URL pool, and they will be subsequently processed in the crawl loop.

        Submit operations are non-blocking, meaning they return immediately without blocking the current thread or suspending the current coroutine.

        session.submitAll(listOf("http://example.com", "http://example.com/1"), session.options("-expire 1d"))
PulsarContexts.await()
        Parameters:
        urls - The urls to submit
        options - The load options
        Returns:

        The PulsarSession itself to enabled chained operations

      • submitAll

         abstract PulsarSession submitAll(Collection<UrlAware> urls)

        Submit urls to the URL pool, and they will be subsequently processed in the crawl loop.

        A submit operation is non-blocking, meaning it returns immediately without blocking the current thread or suspending the current coroutine.

        val urls = listOf("http://example.com", "http://example.com/1").map { Hyperlink(it) }
session.submitAll(urls)
PulsarContexts.await()
        Parameters:
        urls - The urls to submit
        Returns:

        The PulsarSession itself to enabled chained operations

      • submitAll

         abstract PulsarSession submitAll(Collection<UrlAware> urls, String args)

        Submit urls to the URL pool, and they will be subsequently processed in the crawl loop.

        A submit operation is non-blocking, meaning it returns immediately without blocking the current thread or suspending the current coroutine.

        val urls = listOf("http://example.com", "http://example.com/1").map { Hyperlink(it) }
session.submitAll(urls, "-expire 1d")
PulsarContexts.await()
        Parameters:
        urls - The urls to submit
        args - The load arguments
        Returns:

        The PulsarSession itself to enabled chained operations

      • loadOutPages

         abstract List<WebPage> loadOutPages(String portalUrl, String args)

        Load or fetch the portal page, and then load or fetch the out links selected by -outLink option.

        Option -outLink specifies the cssSelector for links in the portal page to load.

        val pages = session.loadOutPages("http://example.com", "-outLink a")
val pages2 = session.loadOutPages("http://example.com", "-outLink a -expire 1d")
        Parameters:
        portalUrl - The portal url from where to load pages
        args - The load arguments
        Returns:

        The loaded out pages

      • loadOutPages

         abstract List<WebPage> loadOutPages(String portalUrl, LoadOptions options)

        Load or fetch the portal page, and then load or fetch the out links selected by -outLink option.

        Option -outLink specifies the cssSelector for links in the portal page to load.

        val pages = session.loadOutPages("http://example.com", session.options("-outLink a"))
val pages2 = session.loadOutPages("http://example.com", session.options("-outLink a -expire 1d"))
        Parameters:
        portalUrl - The portal url from where to load pages
        options - The load options
        Returns:

        The loaded out pages

      • loadOutPages

         List<WebPage> loadOutPages(UrlAware portalUrl)

        A confusing version, it's too complicated to handle events and should not be implemented.

      • loadOutPages

         abstract List<WebPage> loadOutPages(UrlAware portalUrl, String args)

        Load or fetch the portal page, and then load or fetch the out links selected by -outLink option.

        Option -outLink specifies the cssSelector for links in the portal page to load.

        val pages = session.loadOutPages("http://example.com", "-outLink a")
val pages2 = session.loadOutPages("http://example.com", "-outLink a -expire 1d")
        Parameters:
        portalUrl - The portal url from where to load pages
        args - The load arguments
        Returns:

        The loaded out pages

      • loadOutPages

         abstract List<WebPage> loadOutPages(UrlAware portalUrl, LoadOptions options)

        Load or fetch the portal page, and then load or fetch the out links selected by -outLink option.

        Option -outLink specifies the cssSelector for links in the portal page to load.

        val pages = session.loadOutPages("http://example.com", session.options("-outLink a"))
val pages2 = session.loadOutPages("http://example.com", session.options("-outLink a -expire 1d"))
        Parameters:
        portalUrl - The portal url from where to load pages
        options - The load options
        Returns:

        The loaded out pages

      • loadOutPages

         List<WebPage> loadOutPages(NormURL portalUrl)

        A confusing version, it's too complicated to handle events and should not be implemented.

      • loadOutPagesAsync

         abstract List<CompletableFuture<WebPage>> loadOutPagesAsync(String portalUrl, String args)

        Load or fetch the portal page, and then load or fetch the out links selected by -outLink option asynchronously.

        Option -outLink specifies the cssSelector for links in the portal page to load.

        val pages = session.loadOutPagesAsync("http://example.com", "-outLink a")
val pages2 = session.loadOutPagesAsync("http://example.com", "-outLink a -expire 1d")
        Parameters:
        portalUrl - The portal url from where to load pages
        args - The load arguments
        Returns:

        The loaded out pages

      • loadOutPagesAsync

         abstract List<CompletableFuture<WebPage>> loadOutPagesAsync(String portalUrl, LoadOptions options)

        Load or fetch the portal page, and then load or fetch the out links selected by -outLink option asynchronously.

        Option -outLink specifies the cssSelector for links in the portal page to load.

        val pages = session.loadOutPagesAsync("http://example.com", session.options("-outLink a"))
val pages2 = session.loadOutPagesAsync("http://example.com", session.options("-outLink a -expire 1d"))
        Parameters:
        portalUrl - The portal url from where to load pages
        options - The load options
        Returns:

        The loaded out pages

      • submitForOutPages

         abstract PulsarSession submitForOutPages(String portalUrl, String args)

        Load the portal page and submit the out links specified by the -outLink option to the URL pool.

        Option -outLink specifies the cssSelector for links in the portal page to load.

        The submitted urls will be subsequently processed in the crawl loop.

        Submit operations are non-blocking, meaning they return immediately without blocking the current thread or suspending the current coroutine.

        session
  .submitForOutPages("http://example.com", "-outLink a[href*=review]")
  .submitForOutPages("http://example.com", "-outLink a[href*=item] -expire 1d")
PulsarContexts.await()
        Parameters:
        portalUrl - The portal url from where to load pages
        args - The load arguments
        Returns:

        The PulsarSession itself to enable chained operation

      • submitForOutPages

         abstract PulsarSession submitForOutPages(String portalUrl, LoadOptions options)

        Load the portal page and submit the out links specified by the -outLink option to the URL pool.

        Option -outLink specifies the cssSelector for links in the portal page to load.

        The submitted urls will be subsequently processed in the crawl loop.

        Submit operations are non-blocking, meaning they return immediately without blocking the current thread or suspending the current coroutine.

        session
  .submitForOutPages("http://example.com", session.options("-outLink a[href*=review]"))
  .submitForOutPages("http://example.com", session.options("-outLink a[href*=item] -expire 1d"))
PulsarContexts.await()
        Parameters:
        portalUrl - The portal url from where to load pages
        options - The load options
        Returns:

        The PulsarSession itself to enable chained operation

      • submitForOutPages

         abstract PulsarSession submitForOutPages(UrlAware portalUrl, String args)

        Load the portal page and submit the out links specified by the -outLink option to the URL pool.

        Option -outLink specifies the cssSelector for links in the portal page to load.

        The submitted urls will be subsequently processed in the crawl loop.

        Submit operations are non-blocking, meaning they return immediately without blocking the current thread or suspending the current coroutine.

        session
  .submitForOutPages(Hyperlink("http://example.com"), "-outLink a[href*=review]")
  .submitForOutPages(Hyperlink("http://example.com"), "-outLink a[href*=item] -expire 1d")
PulsarContexts.await()
        Parameters:
        portalUrl - The portal url from where to load pages
        args - The load arguments
        Returns:

        The PulsarSession itself to enable chained operation

      • submitForOutPages

         abstract PulsarSession submitForOutPages(UrlAware portalUrl, LoadOptions options)

        Load the portal page and submit the out links specified by the -outLink option to the URL pool.

        Option -outLink specifies the cssSelector for links in the portal page to load.

        The submitted urls will be subsequently processed in the crawl loop.

        Submit operations are non-blocking, meaning they return immediately without blocking the current thread or suspending the current coroutine.

        session
  .submitForOutPages(Hyperlink("http://example.com"), session.options("-outLink a[href*=review]"))
  .submitForOutPages(Hyperlink("http://example.com"), session.options("-outLink a[href*=item] -expire 1d"))
PulsarContexts.await()
        Parameters:
        portalUrl - The portal url from where to load pages
        options - The load options
        Returns:

        The PulsarSession itself to enable chained operation

      • loadResource

         abstract WebPage loadResource(String url, String referrer)

        Load a url as a resource without browser rendering.

        Referrer will be opened by a browser first to obtain browsing environment, such as headers and cookies, and then the browsing environment will be applied to the later resource fetching.

        val page = session.loadResource("http://example.com/robots.txt", "http://example.com")
        Parameters:
        url - The url to load
        referrer - The referrer URL
        Returns:

        The webpage containing the resource

      • loadResource

         abstract WebPage loadResource(String url, String referrer, String args)

        Load a url as a resource without browser rendering.

        Referrer will be opened by a browser first to obtain browsing environment, such as headers and cookies, and then the browsing environment will be applied to the later resource fetching.

        val page = session.loadResource("http://example.com/robots.txt", "http://example.com", "-expire 1d")
        Parameters:
        url - The url to load
        referrer - The referrer URL
        args - The load arguments
        Returns:

        The webpage containing the resource

      • loadResource

         abstract WebPage loadResource(String url, String referrer, LoadOptions options)

        Load a url as a resource without browser rendering.

        Referrer will be opened by a browser first to obtain browsing environment, such as headers and cookies, and then the browsing environment will be applied to the later resource fetching.

        val page = session.loadResource("http://example.com/robots.txt", "http://example.com", session.options("-expire 1d"))
        Parameters:
        url - The url to load
        referrer - The referrer URL
        options - The load options
        Returns:

        The webpage containing the resource

      • loadResourceDeferred

         abstract WebPage loadResourceDeferred(String url, String referrer)

        Load a url as a resource without browser rendering.

        Referrer will be opened by a browser first to obtain browsing environment, such as headers and cookies, and then the browsing environment will be applied to the later resource fetching.

        This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.

        val page = session.loadResourceDeferred("http://example.com/robots.txt", "http://example.com")
        Parameters:
        url - The url to load
        referrer - The referrer URL
        Returns:

        The webpage containing the resource

      • loadResourceDeferred

         abstract WebPage loadResourceDeferred(String url, String referrer, String args)

        Load a url as a resource without browser rendering.

        Referrer will be opened by a browser first to obtain browsing environment, such as headers and cookies, and then the browsing environment will be applied to the later resource fetching.

        This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.

        val page = session.loadResourceDeferred("http://example.com/robots.txt", "http://example.com", "-expire 1d")
        Parameters:
        url - The url to load
        referrer - The referrer URL
        args - The load arguments
        Returns:

        The webpage containing the resource

      • loadResourceDeferred

         abstract WebPage loadResourceDeferred(String url, String referrer, LoadOptions options)

        Load a url as a resource without browser rendering.

        Referrer will be opened by a browser first to obtain browsing environment, such as headers and cookies, and then the browsing environment will be applied to the later resource fetching.

        This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.

        val page = session.loadResourceDeferred("http://example.com/robots.txt", "http://example.com", session.options("-expire 1d"))
        Parameters:
        url - The url to load
        referrer - The referrer URL
        options - The load options
        Returns:

        The webpage containing the resource

      • parse

         abstract FeaturedDocument parse(WebPage page)

        Parses the given page into an in-memory HTML document.

        The parse method operates within the Kotlin process, not inside a real browser. It parses the page content into an in-memory lightweight Document Object Model (DOM), making it very fast to analyze within the Kotlin process. To interact with a live DOM in a real browser, use WebDriver.

        val page = session.load("http://example.com")
val document = session.parse(page)
        Parameters:
        page - The webpage to parse
        Returns:

        The parsed HTML document

      • parse

         abstract FeaturedDocument parse(WebPage page, Boolean noCache)

        Parses the given page into an in-memory HTML document.

        The parse method operates within the Kotlin process, not inside a real browser. It parses the page content into an in-memory lightweight Document Object Model (DOM), making it very fast to analyze within the Kotlin process. To interact with a live DOM in a real browser, use WebDriver.

        val page = session.load("http://example.com")
val document = session.parse(page, true)
        Parameters:
        page - The webpage to parse
        noCache - Whether to skip the cache
        Returns:

        The parsed HTML document

      • loadDocument

         abstract FeaturedDocument loadDocument(String url)

        Load or fetch a webpage and parse it into an HTML document

        val document = session.loadDocument("http://example.com")
        Parameters:
        url - The url to load
        Returns:

        The parsed HTML document

      • loadDocument

         abstract FeaturedDocument loadDocument(String url, String args)

        Load or fetch a webpage and parse it into an HTML document

        val document = session.loadDocument("http://example.com", "-expire 1d")
        Parameters:
        url - The url to load
        args - The load arguments
        Returns:

        The parsed HTML document

      • loadDocument

         abstract FeaturedDocument loadDocument(String url, LoadOptions options)

        Load or fetch a webpage and parse it into an HTML document

        val document = session.loadDocument("http://example.com", session.options("-expire 1d"))
        Parameters:
        url - The url to load
        options - The load options
        Returns:

        The parsed HTML document

      • loadDocument

         abstract FeaturedDocument loadDocument(UrlAware url)

        Load or fetch a webpage and parse it into an HTML document

        val document = session.loadDocument(Hyperlink("http://example.com"))
        Parameters:
        url - The url to load
        Returns:

        The parsed HTML document

      • loadDocument

         abstract FeaturedDocument loadDocument(UrlAware url, String args)

        Load or fetch a webpage and parse it into an HTML document

        val document = session.loadDocument(Hyperlink("http://example.com"), "-expire 1d")
        Parameters:
        url - The url to load
        args - The load arguments
        Returns:

        The parsed HTML document

      • loadDocument

         abstract FeaturedDocument loadDocument(UrlAware url, LoadOptions options)

        Load or fetch a webpage and parse it into an HTML document

        val document = session.loadDocument(Hyperlink("http://example.com"), session.options("-expire 1d"))
        Parameters:
        url - The url to load
        options - The load options
        Returns:

        The parsed HTML document

      • loadDocument

         abstract FeaturedDocument loadDocument(NormURL url)

        Load or fetch a webpage and then parse it into an HTML document.

        val document = session.loadDocument("http://example.com")
        Parameters:
        url - The url to load
        Returns:

        The parsed HTML document

      • extract

         abstract Map<String, String> extract(FeaturedDocument document, Iterable<String> fieldSelectors)

        Load or fetch a webpage located by the given url, and then extract fields specified by field selectors.

        val document = session.loadDocument("http://example.com", "-expire 1d")
val fields = session.extract(document, listOf(".title", ".content"))
        Parameters:
        document - The document to extract
        fieldSelectors - The selectors to extract fields
        Returns:

        All the extracted fields and their selectors

      • extract

         abstract List<Map<String, String>> extract(FeaturedDocument document, String restrictSelector, Iterable<String> fieldSelectors)

        Load or fetch a webpage located by the given url, and then extract fields specified by field selectors.

        val document = session.loadDocument("http://example.com", "-expire 1d")
val fields = session.extract(document, "#main-content", listOf(".title", ".content"))
        Parameters:
        document - The document to extract
        restrictSelector - A CSS selector to locate a DOM where all fields are restricted to
        fieldSelectors - The selectors to extract fields
        Returns:

        All the extracted fields and their selectors

      • extract

         abstract Map<String, String> extract(FeaturedDocument document, Map<String, String> fieldSelectors)

        Load or fetch a webpage located by the given url, and then extract fields specified by field selectors.

        val document = session.loadDocument("http://example.com", "-expire 1d")
val fields = session.extract(document, mapOf("title" to ".title", "content" to ".content"))
        Parameters:
        document - The document to extract
        fieldSelectors - The selectors to extract fields
        Returns:

        All the extracted fields and their selectors

      • extract

         abstract List<Map<String, String>> extract(FeaturedDocument document, String restrictSelector, Map<String, String> fieldSelectors)

        Load or fetch a webpage located by the given url, and then extract fields specified by field selectors.

        val document = session.loadDocument("http://example.com", "-expire 1d")
val fields = session.extract(document, "#main-content", mapOf("title" to ".title", "content" to ".content"))
        Parameters:
        document - The document to extract
        restrictSelector - A CSS selector to locate a DOM where all fields are restricted to
        fieldSelectors - The selectors to extract fields
        Returns:

        All the extracted fields and their selectors

      • scrape

         abstract Map<String, String> scrape(String url, String args, Iterable<String> fieldSelectors)

        Load or fetch a webpage located by the given url, and then extract fields specified by field selectors.

        val fields = session.scrape("http://example.com", "-expire 1d", listOf(".title", ".content"))
        Parameters:
        url - The url to scrape
        args - The load arguments
        fieldSelectors - The selectors to extract fields
        Returns:

        All the extracted fields and their selectors

      • scrape

         abstract Map<String, String> scrape(String url, LoadOptions options, Iterable<String> fieldSelectors)

        Load or fetch a webpage located by the given url, and then extract fields specified by field selectors.

        val fields = session.scrape("http://example.com", session.options("-expire 1d"), listOf(".title", ".content"))
        Parameters:
        url - The url to scrape
        options - The load options
        fieldSelectors - The selectors to extract fields
        Returns:

        All the extracted fields and their selectors

      • scrape

         abstract Map<String, String> scrape(String url, String args, Map<String, String> fieldSelectors)

        Load or fetch a webpage located by the given url, and then extract fields specified by field selectors.

        val fields = session.scrape("http://example.com", "-expire 1d", ".title", ".content")
        Parameters:
        url - The url to scrape
        args - The load arguments
        fieldSelectors - The selectors to extract fields
        Returns:

        All the extracted fields and their names

      • scrape

         abstract Map<String, String> scrape(String url, LoadOptions options, Map<String, String> fieldSelectors)

        Load or fetch a webpage located by the given url, and then extract fields specified by field selectors.

        val fields = session.scrape("http://example.com", session.options("-expire 1d"), ".title", ".content")
        Parameters:
        url - The url to scrape
        options - The load options
        fieldSelectors - The selectors to extract fields
        Returns:

        All the extracted fields and their names

      • scrape

         abstract List<Map<String, String>> scrape(String url, String args, String restrictSelector, Iterable<String> fieldSelectors)

        Load or fetch a webpage located by the given url, and then extract fields specified by field selectors.

        val fields = session.scrape("http://example.com", "-expire 1d", ".container", listOf(".title", ".content"))
        Parameters:
        url - The url to scrape
        args - The load arguments
        restrictSelector - A CSS selector to locate a DOM where all fields are restricted to
        fieldSelectors - The selectors to extract fields
        Returns:

        All the extracted fields and their names

      • scrape

         abstract List<Map<String, String>> scrape(String url, LoadOptions options, String restrictSelector, Iterable<String> fieldSelectors)

        Load or fetch a webpage located by the given url, and then extract fields specified by field selectors.

        val fields = session.scrape("http://example.com", session.options("-expire 1d"), ".container",
     listOf(".title", ".content"))
        Parameters:
        url - The url to scrape
        options - The load options
        restrictSelector - A CSS selector to locate a DOM where all fields are restricted to
        fieldSelectors - The selectors to extract fields
        Returns:

        All the extracted fields and their selectors

      • scrape

         abstract List<Map<String, String>> scrape(String url, String args, String restrictSelector, Map<String, String> fieldSelectors)

        Load or fetch a webpage located by the given url, and then extract fields specified by field selectors.

        val fields = session.scrape("http://example.com", "-expire 1d", ".container",
     mapOf("title" to ".title", "content" to ".content"))
        Parameters:
        url - The url to scrape
        args - The load arguments
        restrictSelector - A CSS selector to locate a DOM where all fields are restricted to
        fieldSelectors - The selectors to extract fields
        Returns:

        All the extracted fields and their names

      • scrape

         abstract List<Map<String, String>> scrape(String url, LoadOptions options, String restrictSelector, Map<String, String> fieldSelectors)

        Load or fetch a webpage located by the given url, and then extract fields specified by field selectors.

        val fields = session.scrape("http://example.com", session.options("-expire 1d"), ".container",
     mapOf("title" to ".title", "content" to ".content"))
        Parameters:
        url - The url to scrape
        options - The load options
        restrictSelector - A CSS selector to locate a DOM where all fields are restricted to
        fieldSelectors - The selectors to extract fields
        Returns:

        All the extracted fields and their names

      • scrapeOutPages

         abstract List<Map<String, String>> scrapeOutPages(String portalUrl, String args, Iterable<String> fieldSelectors)

        Load or fetch out pages specified by out link selector, and then extract fields specified by field selectors from each out page.

        val fields = session.scrapeOutPages("http://example.com", "-outLink a", listOf(".title", ".content"))
        Parameters:
        portalUrl - The portal url to start scraping
        args - Load arguments for both the portal page and out pages
        fieldSelectors - CSS selectors to extract fields from out pages
        Returns:

        All extracted fields. For each out page, fields extracted with their selectors are saved in a map.

      • scrapeOutPages

         abstract List<Map<String, String>> scrapeOutPages(String portalUrl, LoadOptions options, Iterable<String> fieldSelectors)

        Load or fetch out pages specified by out link selector, and then extract fields specified by field selectors from each out page.

        val fields = session.scrapeOutPages("http://example.com", session.options("-outLink a"), listOf(".title", ".content"))
        Parameters:
        portalUrl - The portal url to start scraping
        options - Load options for both the portal page and out pages
        fieldSelectors - CSS selectors to extract fields from out pages
        Returns:

        All extracted fields. For each out page, fields extracted with their selectors are saved in a map.

      • scrapeOutPages

         abstract List<Map<String, String>> scrapeOutPages(String portalUrl, String args, String restrictSelector, Iterable<String> fieldSelectors)

        Load or fetch out pages specified by out link selector, and then extract fields specified by field selectors from each out page.

        val fields = session.scrapeOutPages("http://example.com", "-outLink a", mapOf("title" to ".title", "content" to ".content"))
        Parameters:
        portalUrl - The portal url to start scraping
        args - Load arguments for both the portal page and out pages
        restrictSelector - A CSS selector to locate a DOM where all fields are restricted to
        fieldSelectors - CSS selectors to extract fields from out pages
        Returns:

        All extracted fields. For each out page, fields extracted with their selectors are saved in a map.

      • scrapeOutPages

         abstract List<Map<String, String>> scrapeOutPages(String portalUrl, LoadOptions options, String restrictSelector, Iterable<String> fieldSelectors)

        Load or fetch out pages specified by out link selector, and then extract fields specified by field selectors from each out page.

        val fields = session.scrapeOutPages("http://example.com", session.options("-outLink a"),
     mapOf("title" to ".title", "content" to ".content"))
        Parameters:
        portalUrl - The portal url to start scraping
        options - Load options for both the portal page and out pages
        restrictSelector - A CSS selector to locate a DOM where all fields are restricted to
        fieldSelectors - CSS selectors to extract fields from out pages
        Returns:

        All extracted fields. For each out page, fields extracted with their selectors are saved in a map.

      • scrapeOutPages

         abstract List<Map<String, String>> scrapeOutPages(String portalUrl, String args, Map<String, String> fieldSelectors)

        Load or fetch out pages specified by out link selector, and then extract fields specified by field selectors from each out page.

        val fields = session.scrapeOutPages("http://example.com", "-outLink a", mapOf("title" to ".title", "content" to ".content"))
        Parameters:
        portalUrl - The portal url to start scraping
        args - Load arguments for both the portal page and out pages
        fieldSelectors - CSS selectors to extract fields from out pages
        Returns:

        All extracted fields. For each out page, fields extracted with their names are saved in a map.

      • scrapeOutPages

         abstract List<Map<String, String>> scrapeOutPages(String portalUrl, LoadOptions options, Map<String, String> fieldSelectors)

        Load or fetch out pages specified by out link selector, and then extract fields specified by field selectors from each out page.

        val fields = session.scrapeOutPages("http://example.com", session.options("-outLink a"),
     mapOf("title" to ".title", "content" to ".content"))
        Parameters:
        portalUrl - The portal url to start scraping
        options - Load options for both the portal page and out pages
        fieldSelectors - CSS selectors to extract fields from out pages
        Returns:

        All extracted fields. For each out page, fields extracted with their names are saved in a map.

      • scrapeOutPages

         abstract List<Map<String, String>> scrapeOutPages(String portalUrl, String args, String restrictSelector, Map<String, String> fieldSelectors)

        Load or fetch out pages specified by out link selector, and then extract fields specified by field selectors from each out page.

        val fields = session.scrapeOutPages("http://example.com", "-outLink a", ".container",
     mapOf("title" to ".title", "content" to ".content"))
        Parameters:
        portalUrl - The portal url to start scraping
        args - Load arguments for both the portal page and out pages
        restrictSelector - A CSS selector to locate a DOM where all fields are restricted to
        fieldSelectors - CSS selectors to extract fields from out pages
        Returns:

        All extracted fields. For each out page, fields extracted with their names are saved in a map.

      • scrapeOutPages

         abstract List<Map<String, String>> scrapeOutPages(String portalUrl, LoadOptions options, String restrictSelector, Map<String, String> fieldSelectors)

        Load or fetch out pages specified by out link selector, and then extract fields specified by field selectors from each out page.

        val fields = session.scrapeOutPages("http://example.com", session.options("-outLink a"), ".container",
     mapOf("title" to ".title", "content" to ".content"))
        Parameters:
        portalUrl - The portal url to start scraping
        options - Load options for both the portal page and out pages
        restrictSelector - A CSS selector to locate a DOM where all fields are restricted to
        fieldSelectors - CSS selectors to extract fields from out pages
        Returns:

        All extracted fields. For each out page, fields extracted with their names are saved in a map.

      • chat

         abstract ModelResponse chat(String prompt)

        Initiates a chat with the AI model using a general prompt. This method sends the provided prompt to the AI model without any additional context.

        Parameters:
        prompt - The prompt or query to be sent to the AI model.
        Returns:

        The response from the AI model encapsulated in a ModelResponse object.

      • chat

         abstract ModelResponse chat(String prompt, WebPage page)

        Initiates a chat with the AI model about a specific webpage. The method sends the provided prompt along with the HTML source code of the specified webpage to the AI model.

        Parameters:
        prompt - The prompt or query to be sent to the AI model.
        page - The WebPage object containing the webpage's content to provide context for the chat.
        Returns:

        The response from the AI model encapsulated in a ModelResponse object.

      • chat

         abstract ModelResponse chat(String prompt, FeaturedDocument document)

        Initiates a chat with the AI model about a specific document. The method sends the provided prompt along with the text content of the specified document to the AI model.

        Parameters:
        prompt - The prompt or query to be sent to the AI model.
        document - The FeaturedDocument object containing the document's text content to provide context for the chat.
        Returns:

        The response from the AI model encapsulated in a ModelResponse object.

      • chat

         abstract ModelResponse chat(String prompt, Element element)

        Initiates a chat with the AI model about a specific HTML element. The method sends the provided prompt along with the text content of the specified HTML element to the AI model.

        Parameters:
        prompt - The prompt or query to be sent to the AI model.
        element - The Element object containing the HTML element's text content to provide context for the chat.
        Returns:

        The response from the AI model encapsulated in a ModelResponse object.

      • export

         abstract Path export(WebPage page)

        Exports the content of a webpage to a file.

        This method saves the content of the given webpage to a file and returns the path where the file is stored.

        Example usage:

        val page = session.load("http://example.com")
val path = session.export(page)
        Parameters:
        page - The webpage to export.
        Returns:

        The path of the exported file.

      • export

         abstract Path export(WebPage page, String ident)

        Exports the content of a webpage to a file with a specified identifier.

        This method saves the content of the given webpage to a file, using the provided identifier to distinguish the file from others, and returns the path where the file is stored.

        Example usage:

        val page = session.load("http://example.com")
val path = session.export(page, "example")
        Parameters:
        page - The webpage to export.
        ident - A file name identifier used to distinguish the exported file.
        Returns:

        The path of the exported file.

      • export

         abstract Path export(FeaturedDocument doc)

        Exports the outer HTML of a document to a file.

        This method saves the outer HTML of the given document to a file and returns the path where the file is stored.

        Parameters:
        doc - The document to export.
        Returns:

        The path of the exported file.

      • export

         abstract Path export(FeaturedDocument doc, String ident)

        Exports the outer HTML of a document to a file with a specified identifier.

        This method saves the outer HTML of the given document to a file, using the provided identifier to distinguish the file from others, and returns the path where the file is stored.

        Example usage:

        val document = session.loadDocument("http://example.com")
val path = session.export(document, "example")
        Parameters:
        doc - The document to export.
        ident - A file name identifier used to distinguish the exported file.
        Returns:

        The path of the exported file.

      • exportTo

         abstract Path exportTo(WebPage page, Path path)

        Exports the content of a webpage to a specified file path.

        This method saves the content of the given webpage to the specified file path and returns the path where the file is stored.

        Example usage:

        val page = session.load("http://example.com")
val path = session.exportTo(page, Paths.get("/tmp/example.html"))
        Parameters:
        page - The webpage to export.
        path - The file path where the content will be saved.
        Returns:

        The path of the exported file.

      • exportTo

         abstract Path exportTo(FeaturedDocument doc, Path path)

        Exports the outer HTML of a document to a specified file path.

        This method saves the outer HTML of the given document to the specified file path and returns the path where the file is stored.

        Example usage:

        val document = session.loadDocument("http://example.com")
val path = session.exportTo(document, Paths.get("/tmp/example.html"))
        Parameters:
        doc - The document to export.
        path - The file path where the content will be saved.
        Returns:

        The path of the exported file.

      • persist

         abstract Boolean persist(WebPage page)

        Persist the webpage.

        val page = session.load("http://example.com")
session.persist(page)
        Parameters:
        page - Page to persist
        Returns:

        Whether the page is persisted successfully

      • delete

         abstract Unit delete(String url)

        Delete a webpage from the storage

        session.delete("http://example.com")
        Parameters:
        url - The url to delete

      • flush

         abstract Unit flush()

        Flush to the storage

      • getId

         abstract Long getId()

        The in-process unique id.

      • getUuid

         abstract String getUuid()

        The universally unique identifier (UUID). A UUID represents a 128-bit value.

      • getConfiguration

         abstract ImmutableConfig getConfiguration()

        The main configuration.

        Browser4 supports multiple configuration sources in order of precedence:

        • 🔧 Environment Variables

        • ⚙️ JVM System Properties

        • 📝 Spring Boot application.properties or application.yml

      • getSessionConfig

         abstract VolatileConfig getSessionConfig()

        The session-specific volatile configuration, which allows dynamic adjustments to settings at any point during the session. Unlike the main configuration, this configuration is designed to be modified on-the-fly to adapt to runtime requirements.

      • getPageCache

         abstract ConcurrentExpiringLRUCache<String, WebPage> getPageCache()

        The global page cache.

      • getDocumentCache

         abstract ConcurrentExpiringLRUCache<String, FeaturedDocument> getDocumentCache()

        The global document cache.

      • getBoundDriver

         abstract WebDriver getBoundDriver()

        The bound driver. If there is a bound driver, all subsequential actions that needed a driver use the bound one.

      • getBoundBrowser

         abstract Browser getBoundBrowser()

        The bound browser. If there is a bound browser, all subsequential actions that needed a browser use the bound one.