Package ai.platon.pulsar.skeleton.session

Class AbstractPulsarSession

  • All Implemented Interfaces:
    ai.platon.pulsar.skeleton.session.PulsarSession , java.lang.AutoCloseable 
    
public abstract class AbstractPulsarSession
 implements PulsarSession

    Created by Vincent on 18-1-17. Copyright @ 2013-2023 Platon AI. All rights reserved

    • Constructor Detail

    • Method Detail

      • getUuid

         String getUuid()

        The universally unique identifier (UUID). A UUID represents a 128-bit value.

      • getConfiguration

         ImmutableConfig getConfiguration()

        The main configuration.

        Browser4 supports multiple configuration sources in order of precedence:

        • 🔧 Environment Variables

        • ⚙️ JVM System Properties

        • 📝 Spring Boot application.properties or application.yml

      • getBoundDriver

         WebDriver getBoundDriver()

        The bound driver. If there is a bound driver, all subsequential actions that needed a driver use the bound one.

      • getBoundBrowser

         Browser getBoundBrowser()

        The bound browser. If there is a bound browser, all subsequential actions that needed a browser use the bound one.

      • getSessionConfig

         VolatileConfig getSessionConfig()

        The session scope volatile config, every setting is supposed to be changed at any time and any place

      • getId

         Long getId()

        The session id. Session id is expected to be set by the container, e.g. the h2 database runtime

      • normalize

         NormURL normalize(String url)

        Normalize a url.

        Parameters:
        url - The url to normalize
        Returns:

        The normalized url

      • normalize

         NormURL normalize(String url, String args, Boolean toItemOption)

        Normalize a url.

        Parameters:
        url - The url to normalize
        args - The arguments
        toItemOption - If the LoadOptions is converted to item load options
        Returns:

        The normalized url

      • normalize

         NormURL normalize(String url, LoadOptions options, Boolean toItemOption)

        Normalize a url.

        Parameters:
        url - The url to normalize
        options - The LoadOptions applied to the url
        toItemOption - If the LoadOptions is converted to item load options
        Returns:

        The normalized url

      • normalize

         List<NormURL> normalize(Iterable<String> urls, String args, Boolean toItemOption)

        Normalize urls, remove invalid ones.

        Parameters:
        urls - The urls to normalize
        args - The arguments
        toItemOption - If the LoadOptions is converted to item load options
        Returns:

        All normalized urls

      • normalize

         List<NormURL> normalize(Iterable<String> urls, LoadOptions options, Boolean toItemOption)

        Normalize urls, remove invalid ones.

        Parameters:
        urls - The urls to normalize
        options - The LoadOptions applied to each url
        toItemOption - If the LoadOptions is converted to item load options
        Returns:

        All normalized urls

      • normalize

         NormURL normalize(UrlAware url)

        Normalize a url, return NormURL.NIL if the url is invalid.

        Parameters:
        url - The url to normalize
        Returns:

        The normalized url

      • normalize

         NormURL normalize(UrlAware url, String args, Boolean toItemOption)

        Normalize a url, return NormURL.NIL if the url is invalid.

        Parameters:
        url - The url to normalize
        args - The arguments
        toItemOption - If the LoadOptions is converted to item load options
        Returns:

        The normalized url

      • normalize

         NormURL normalize(UrlAware url, LoadOptions options, Boolean toItemOption)

        Normalize a url, return NormURL.NIL if the url is invalid.

        Parameters:
        url - The url to normalize
        options - The LoadOptions applied to the url
        toItemOption - If the LoadOptions is converted to item load options
        Returns:

        The normalized url

      • normalize

         List<NormURL> normalize(Collection<UrlAware> urls)

        Normalize urls, remove invalid ones.

        Parameters:
        urls - The urls to normalize
        Returns:

        All normalized urls

      • normalize

         List<NormURL> normalize(Collection<UrlAware> urls, String args, Boolean toItemOption)

        Normalize urls, remove invalid ones

        Parameters:
        urls - The urls to normalize
        args - The arguments
        toItemOption - If the LoadOptions is converted to item load options
        Returns:

        All normalized urls, all invalid input urls are removed

      • normalize

         List<NormURL> normalize(Collection<UrlAware> urls, LoadOptions options, Boolean toItemOption)

        Normalize urls, remove invalid ones

        Parameters:
        urls - The urls to normalize
        options - The LoadOptions applied to each url
        toItemOption - If the LoadOptions is converted to item load options
        Returns:

        All normalized urls, all invalid input urls are removed

      • normalizeOrNull

         NormURL normalizeOrNull(String url, LoadOptions options, Boolean toItemOption)

        Normalize a url, return null if the url is invalid.

        Parameters:
        url - The url to normalize
        options - The LoadOptions applied to the url
        toItemOption - If the LoadOptions is converted to item load options
        Returns:

        The normalized url or null

      • normalizeOrNull

         NormURL normalizeOrNull(UrlAware url, LoadOptions options, Boolean toItemOption)

        Normalize a url, return null if the url is invalid.

        Parameters:
        url - The url to normalize
        options - The LoadOptions applied to the url
        toItemOption - If the LoadOptions is converted to item load options
        Returns:

        The normalized url or null

      • get

         WebPage get(String url)

        Get a page from local storage.

        val url = "http://example.com"
val page = session.get(url)
        Parameters:
        url - The url
        Returns:

        The webpage in storage if exists, otherwise returns a NIL page

      • get

         WebPage get(String url, String fields)

        Get a page from local storage.

        val url = "http://example.com"
val page = session.get(url, "content", "metadata")
        Parameters:
        url - The url
        fields - The fields to load from local storage
        Returns:

        The webpage in storage if exists, otherwise returns a NIL page

      • getOrNull

         WebPage getOrNull(String url)

        Get a page from local storage.

        val url = "http://example.com"
val page = session.getOrNull(url)
        Parameters:
        url - The url
        Returns:

        The page in storage if exists, otherwise returns null

      • getOrNull

         WebPage getOrNull(String url, String fields)

        Get a page from local storage.

        val url = "http://example.com"
val page = session.getOrNull(url, "content", "metadata")
        Parameters:
        url - The url
        fields - The fields to load from local storage
        Returns:

        The page in storage if exists, otherwise returns null

      • getContent

         ByteBuffer getContent(String url)

        Get the content of the page from the storage.

        val url = "http://example.com"
val content = session.getContent(url)
        Parameters:
        url - The url of the page to retrieve
        Returns:

        The page content or null

      • getContentAsString

         String getContentAsString(String url)

        Get the content of the page from the storage

        val url = "http://example.com"
val content = session.getContentAsString(url)
        Parameters:
        url - The url of the page to retrieve
        Returns:

        The page content in string format or null

      • exists

         Boolean exists(String url)

        Check if the page exists in the storage.

        val url = "http://example.com"
val exists = session.exists(url)
        Parameters:
        url - The url to check
        Returns:

        true if the page exists, false otherwise

      • fetchState

         CheckState fetchState(WebPage page, LoadOptions options)

        Return the fetch state of the page.

        val url = "http://example.com"
val state = session.fetchState(url)
        Parameters:
        page - The webpage
        options - The load options
        Returns:

        The fetch state of the page

      • open

         WebPage open(String url)

        Open a url.

        This method opens the url immediately, regardless of the previous state of the page.

        val url = "http://example.com"
val page = session.open(url)
        Parameters:
        url - The url to open
        Returns:

        The webpage loaded or NIL

      • open

         WebPage open(String url, PageEventHandlers eventHandlers)

        Open a url with page events.

        This method opens the url immediately, regardless of the previous state of the page.

        val url = "http://example.com"
val event = PrintFlowEvent()
val page = session.open(url, event)
        Parameters:
        url - The url to open
        Returns:

        The webpage loaded or NIL

      • open

         WebPage open(String url, WebDriver driver)

        Open a url with webdriver.

        This method opens the url immediately, regardless of the previous state of the page.

        val url = "http://example.com"
val page = session.open(url, driver)
        Parameters:
        url - The url to open
        Returns:

        The webpage loaded or NIL

      • open

         WebPage open(String url, WebDriver driver, PageEventHandlers eventHandlers)

        Open a url with page events and webdriver.

        This method opens the url immediately, regardless of the previous state of the page.

        val url = "http://example.com"
val event = PrintFlowEvent()
val page = session.open(url, event)
        Parameters:
        url - The url to open
        Returns:

        The webpage loaded or NIL

      • capture

         WebPage capture(WebDriver driver, String url, PageEventHandlers eventHandlers)

        Captures the live page currently controlled by the given WebDriver and produces a local static WebPage instance representing its current state.

        val url = driver.currentUrl()
val page = session.capture(url, driver)

        An optional URL can be provided to identify the webpage, If url is not provided, set WebPage's url to driver.currentUrl(). The url will be normalized to identify the webpage and can differ from the active page's URI since the active page may goto or redirect to another page.

        Parameters:
        driver - The WebDriver instance controlling the live browser page.
        url - Optional URL to identify the webpage.
        Returns:

        A WebPage object containing the static representation of the live page, including DOM structure and referenced resources.

      • createBoundDriver

         WebDriver createBoundDriver()

        Create the default driver and bind it to the session.

      • getOrCreateBoundDriver

         WebDriver getOrCreateBoundDriver()

        Create the default driver and bind it to the session.

      • bindDriver

         Unit bindDriver(WebDriver driver)

        Bind a webdriver to the session.

        session.bindDriver(driver)

      • bindBrowser

         Unit bindBrowser(Browser browser)

        Bind a browser to the session.

        session.bindBrowser(driver)

      • load

         WebPage load(String url)

        Load a url.

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        Other fetch conditions can be specified by load arguments:

        • expiration

        • page size requirement

        • fields requirement

        • other

        val url = "http://example.com"
val page = session.load(url)
        Parameters:
        url - The url to load
        Returns:

        The webpage loaded or NIL

      • load

         WebPage load(String url, String args)

        Load a url with specified arguments.

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        Fetch conditions can be specified by load arguments:

        • expiration

        • page size requirement

        • fields requirement

        • other

        val url = "http://example.com"
val args = "-expire 1d"
val page = session.load(url, args)
        Parameters:
        url - The url to load
        args - The load arguments
        Returns:

        The webpage loaded or NIL

      • load

         WebPage load(String url, LoadOptions options)

        Load a url with specified options.

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        Fetch conditions can be specified by load arguments:

        • expiration

        • page size requirement

        • fields requirement

        • other

        val url = "http://example.com"
val options = session.options("-expire 1d")
val page = session.load(url, options)
        Parameters:
        url - The url to load
        options - The load options
        Returns:

        The webpage loaded or NIL

      • load

         WebPage load(UrlAware url)

        Load a url with the specified arguments.

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        val url = Hyperlink("http://example.com")
val page = session.load(url)
        Parameters:
        url - The url to load
        Returns:

        The webpage loaded or NIL

      • load

         WebPage load(UrlAware url, String args)

        Load a url with the specified arguments.

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        val url = Hyperlink("http://example.com")
val page = session.load(url, "-expire 1d")
        Parameters:
        url - The url to load
        args - The load arguments
        Returns:

        The webpage loaded or NIL

      • load

         WebPage load(UrlAware url, LoadOptions options)

        Load a url with options.

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        val url = Hyperlink("http://example.com")
val options = session.options("-expire 1d")
val page = session.load(url, options)
        Parameters:
        url - The url to load
        options - The load options
        Returns:

        The webpage loaded or NIL

      • load

         WebPage load(NormURL url)

        Load a normal url.

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        val url = session.normalize("http://example.com")
val page = session.load(url)

        • @param url The normal url

        Returns:

        The webpage loaded or NIL

      • loadDeferred

         WebPage loadDeferred(String url, String args)

        Load a url with specified options.

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.

        val url = "http://example.com"
val args = "-expire 1d"
val page = session.loadDeferred(url, args)
        Parameters:
        url - The url to load
        args - The load options
        Returns:

        The webpage loaded or NIL

      • loadDeferred

         WebPage loadDeferred(String url, LoadOptions options)

        Load a url with specified options.

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.

        val url = "http://example.com"
val options = session.options("-expire 1d")
val page = session.loadDeferred(url, options)
        Parameters:
        url - The url to load
        options - The load options
        Returns:

        The webpage loaded or NIL

      • loadDeferred

         WebPage loadDeferred(UrlAware url, String args)

        Load a url with specified arguments.

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.

        val url = Hyperlink("http://example.com")
val page = session.loadDeferred(url, "-expire 1d")
        Parameters:
        url - The url to load
        args - The load args
        Returns:

        The webpage loaded or NIL

      • loadDeferred

         WebPage loadDeferred(UrlAware url, LoadOptions options)

        Load a url with specified options.

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.

        val url = Hyperlink("http://example.com")
val options = session.options("-expire 1d")
val page = session.loadDeferred(url, options)
        Parameters:
        url - The url to load
        options - The load options
        Returns:

        The webpage loaded or NIL

      • loadDeferred

         WebPage loadDeferred(NormURL url)

        Load a url with specified options

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.

        val url = session.normalize("http://example.com")
val page = session.loadDeferred(url)
        Parameters:
        url - The normal url
        Returns:

        The webpage loaded or NIL

      • loadAll

         List<WebPage> loadAll(Iterable<String> urls)

        Load all urls with specified options

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        val urls = listOf("http://example.com", "http://example.com/1")
val pages = session.loadAll(urls)
        Parameters:
        urls - The urls to load
        Returns:

        The successfully loaded webpages, all failed urls are ignored

      • loadAll

         List<WebPage> loadAll(Iterable<String> urls, String args)

        Load all urls with specified options

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        val urls = listOf("http://example.com", "http://example.com/1")
val pages = session.loadAll(urls, "-expire 1d")
        Parameters:
        urls - The urls to load
        args - The load arguments
        Returns:

        The successfully loaded webpages, all failed urls are ignored

      • loadAll

         List<WebPage> loadAll(Iterable<String> urls, LoadOptions options)

        Load all urls with specified options

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        val urls = listOf("http://example.com", "http://example.com/1")
val pages = session.loadAll(urls, session.options("-expire 1d"))
        Parameters:
        urls - The urls to load
        options - The load options
        Returns:

        The successfully loaded webpages, all failed urls are ignored

      • loadAll

         List<WebPage> loadAll(Collection<UrlAware> urls)

        Load all urls with specified options

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        val urls = listOf(Hyperlink("http://example.com"), Hyperlink("http://example.com/1"))
val pages = session.loadAll(urls)
        Parameters:
        urls - The urls to load
        Returns:

        The successfully loaded webpages, all failed urls are ignored

      • loadAll

         List<WebPage> loadAll(Collection<UrlAware> urls, String args)

        Load all urls with specified options

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        val urls = listOf(Hyperlink("http://example.com"), Hyperlink("http://example.com/1"))
val pages = session.loadAll(urls, "-expire 1d")
        Parameters:
        urls - The urls to load
        args - The load arguments
        Returns:

        The successfully loaded webpages, all failed urls are ignored

      • loadAll

         List<WebPage> loadAll(Collection<UrlAware> urls, LoadOptions options)

        Load all urls with specified options

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        val urls = listOf(Hyperlink("http://example.com"), Hyperlink("http://example.com/1"))
val pages = session.loadAll(urls, session.options("-expire 1d"))
        Parameters:
        urls - The urls to load
        options - The load options
        Returns:

        The successfully loaded webpages, all failed urls are ignored

      • loadAll

         List<WebPage> loadAll(List<NormURL> normUrls)

        Load all normal urls with specified options

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        val urls = listOf("http://example.com", "http://example.com/1").map { session.normalize(it) }
val pages = session.loadAll(urls)
        Parameters:
        normUrls - The normal urls to load
        Returns:

        The successfully loaded webpages, all failed urls are ignored

      • loadAsync

         CompletableFuture<WebPage> loadAsync(String url)

        Load a url in java async style

        String url = "http://example.com";
WebPage page = session.loadAsync(url).join();
        Parameters:
        url - The url to load
        Returns:

        A completable future of webpage

      • loadAsync

         CompletableFuture<WebPage> loadAsync(String url, String args)

        Load a url in java async style

        String url = "http://example.com";
WebPage page = session.loadAsync(url, "-expire 1d").join();
        Parameters:
        url - The url to load
        Returns:

        A completable future of webpage

      • loadAsync

         CompletableFuture<WebPage> loadAsync(String url, LoadOptions options)

        Load a url in java async style

        String url = "http://example.com";
WebPage page = session.loadAsync(url, session.options("-expire 1d")).join();
        Parameters:
        url - The url to load
        Returns:

        A completable future of webpage

      • loadAsync

         CompletableFuture<WebPage> loadAsync(UrlAware url)

        Load a url in java async style

        Hyperlink url = new Hyperlink("http://example.com");
WebPage page = session.loadAsync(url).join();
        Parameters:
        url - The url to load
        Returns:

        A completable future of webpage

      • loadAsync

         CompletableFuture<WebPage> loadAsync(UrlAware url, String args)

        Load a url in java async style

        Hyperlink url = new Hyperlink("http://example.com");
WebPage page = session.loadAsync(url, "-expire 1d").join();
        Parameters:
        url - The url to load
        Returns:

        A completable future of webpage

      • loadAsync

         CompletableFuture<WebPage> loadAsync(UrlAware url, LoadOptions options)

        Load a url in java async style

        Hyperlink url = new Hyperlink("http://example.com");
WebPage page = session.loadAsync(url, session.options("-expire 1d")).join();
        Parameters:
        url - The url to load
        Returns:

        A completable future of webpage

      • loadAsync

         CompletableFuture<WebPage> loadAsync(NormURL url)

        Load a normal url in java async style

        NormURL url = session.normalize("http://example.com");
WebPage page = session.loadAsync(url).join();
        Parameters:
        url - The normal url to load
        Returns:

        A completable future of webpage

      • loadAllAsync

         List<CompletableFuture<WebPage>> loadAllAsync(Iterable<String> urls)

        Load all urls in java async style

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        List<String> urls = Arrays.asList("http://example.com", "http://example.com/1");
CompletableFuture<?>[] pages = session.loadAllAsync(urls).toArray(CompletableFuture<?>[]::new);
CompletableFuture.allOf(futures).join();
        Parameters:
        urls - The normal urls to load
        Returns:

        The completable futures of webpages

      • loadAllAsync

         List<CompletableFuture<WebPage>> loadAllAsync(Iterable<String> urls, String args)

        Load all urls in java async style.

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        List<String> urls = Arrays.asList("http://example.com", "http://example.com/1");
CompletableFuture<?>[] pages = session.loadAllAsync(urls, "-expire 1d").toArray(CompletableFuture<?>[]::new);
CompletableFuture.allOf(futures).join();
        Parameters:
        urls - The normal urls to load
        Returns:

        The completable futures of webpages

      • loadAllAsync

         List<CompletableFuture<WebPage>> loadAllAsync(Iterable<String> urls, LoadOptions options)

        Load all normal urls in java async style.

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        List<String> urls = Arrays.asList("http://example.com", "http://example.com/1");
CompletableFuture<?>[] pages = session.loadAllAsync(urls, session.options("-expire 1d")).toArray(CompletableFuture<?>[]::new);
CompletableFuture.allOf(futures).join();
        Parameters:
        urls - The normal urls to load
        Returns:

        The completable futures of webpages

      • loadAllAsync

         List<CompletableFuture<WebPage>> loadAllAsync(Collection<UrlAware> urls)

        Load all urls in java async style.

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        List<UrlAware> urls = Arrays.asList(new Hyperlink("http://example.com"), new Hyperlink("http://example.com/1"));
CompletableFuture<?>[] pages = session.loadAllAsync(urls).toArray(CompletableFuture<?>[]::new);
CompletableFuture.allOf(futures).join();
        Parameters:
        urls - The normal urls to load
        Returns:

        The completable futures of webpages

      • loadAllAsync

         List<CompletableFuture<WebPage>> loadAllAsync(Collection<UrlAware> urls, String args)

        Load all normal urls in java async style.

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        List<UrlAware> urls = Arrays.asList(new Hyperlink("http://example.com"), new Hyperlink("http://example.com/1"));
CompletableFuture<?>[] pages = session.loadAllAsync(urls, "-expire 1d").toArray(CompletableFuture<?>[]::new);
CompletableFuture.allOf(futures).join();
        Parameters:
        urls - The normal urls to load
        args - The load arguments
        Returns:

        The completable futures of webpages

      • loadAllAsync

         List<CompletableFuture<WebPage>> loadAllAsync(Collection<UrlAware> urls, LoadOptions options)

        Load all normal urls in java async style.

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        List<UrlAware> urls = Arrays.asList(new Hyperlink("http://example.com"), new Hyperlink("http://example.com/1"));
CompletableFuture<?>[] pages = session.loadAllAsync(urls, session.options("-expire 1d")).toArray(CompletableFuture<?>[]::new);
CompletableFuture.allOf(futures).join();
        Parameters:
        urls - The normal urls to load
        options - The load options
        Returns:

        The completable futures of webpages

      • loadAllAsync

         List<CompletableFuture<WebPage>> loadAllAsync(List<NormURL> urls)

        Load all normal urls in java async style.

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        List<NormURL> urls = Arrays.asList(session.normalize("http://example.com"), session.normalize("http://example.com/1"));
CompletableFuture<?>[] pages = session.loadAllAsync(urls).toArray(CompletableFuture<?>[]::new);
CompletableFuture.allOf(futures).join();
        Parameters:
        urls - The normal urls to load
        Returns:

        The completable futures of webpages

      • submit

         <Error class: unknown class> submit(String url)

        Submit a url to the URL pool, and it will be subsequently processed in the Main loop.

        A submit operation is non-blocking, meaning it returns immediately without blocking the current thread or suspending the current coroutine.

        session.submit("http://example.com")
PulsarContexts.await()
        Parameters:
        url - The url to submit
        Returns:

        The PulsarSession itself to enabled chained operations

      • submit

         <Error class: unknown class> submit(String url, String args)

        Submit a url to the URL pool, and it will be subsequently processed in the Main loop.

        A submit operation is non-blocking, meaning it returns immediately without blocking the current thread or suspending the current coroutine.

        session.submit("http://example.com", "-expire 1d")
PulsarContexts.await()
        Parameters:
        url - The url to submit
        args - The load arguments
        Returns:

        The PulsarSession itself to enabled chained operations

      • submit

         <Error class: unknown class> submit(String url, LoadOptions options)

        Submit a url to the URL pool, and it will be subsequently processed in the Main loop.

        A submit operation is non-blocking, meaning it returns immediately without blocking the current thread or suspending the current coroutine.

        session.submit("http://example.com", session.options("-expire 1d"))
PulsarContexts.await()

        Submit can be used with event listeners to handle the page events.

        The code snippet below shows how to submit a url with an attached load event handler. This handler is invoked once the page is loaded, and it prints the URL of the page.

        val options = session.options("-expire 1d")
options.eventHandlers.loadEvent.onLoaded.addLast { println(it.url) }
session.submit("http://example.com", options)
PulsarContexts.await()

        The code snippet below shows how to submit a hyperlink with every supported event handlers attached. Each event handler will print a message when its associated event is triggered.

        val event = PrintFlowEvent()
val options = session.options("-expire 1d", event)
session.submit("http://example.com", options)
PulsarContexts.await()
        Parameters:
        url - The url to submit
        options - The load options
        Returns:

        The PulsarSession itself to enabled chained operations

      • submit

         <Error class: unknown class> submit(UrlAware url)

        Submit a url to the URL pool, and it will be subsequently processed in the Main loop.

        A submit operation is non-blocking, meaning it returns immediately without blocking the current thread or suspending the current coroutine.

        Basic usage:

        session.submit(Hyperlink("http://example.com"))
PulsarContexts.await()

        Submit methods are commonly be used in conjunction with event listeners to handle page events.

        The code snippet below shows how to submit a hyperlink with an attached load event handler. This handler is invoked once the page is loaded, and it prints the URL of the page.

        val hyperlink = ListenableHyperlink("http://example.com")
hyperlink.eventHandlers.loadEvent.onLoaded.addLast { page -> println(page.url) }
session.submit(hyperlink)
PulsarContexts.await()

        The code snippet below shows how to submit a hyperlink with a event handler attached that processes the parsed document once the relevant event is triggered.

        val hyperlink = ListenableHyperlink("http://example.com", args = "-parse", event = PrintFlowEvent())
hyperlink.eventHandlers.loadEvent.onHTMLDocumentParsed.addLast { page, document ->
     val title = document.selectFirstOrNull(".title")?.text() ?: "Not found"
     println(title)
}
session.submit(hyperlink)
PulsarContexts.await()

        ParsableHyperlink can be used to simplify the load-and-parse tasks.

        val hyperlink = ParsableHyperlink("http://example.com") { page, document ->
val title = document.selectFirstOrNull(".title")?.text() ?: ""
     println(title)
}
session.submit(hyperlink)
PulsarContexts.await()

        The code snippet below shows how to submit a hyperlink with every supported event handlers attached. Each event handler will print a message when its associated event is triggered.

        val event = PrintFlowEvent()
val options = session.options("-expire 1d", event)
session.submit("http://example.com", options)
PulsarContexts.await()
        Parameters:
        url - The url to submit
        Returns:

        The PulsarSession itself to enabled chained operations

      • submit

         <Error class: unknown class> submit(UrlAware url, String args)

        Submit a url to the URL pool, and it will be subsequently processed in the Main loop.

        A submit operation is non-blocking, meaning it returns immediately without blocking the current thread or suspending the current coroutine.

        session.submit(Hyperlink("http://example.com"), "-expire 1d")
PulsarContexts.await()
        Parameters:
        url - The url to submit
        args - The load arguments
        Returns:

        The PulsarSession itself to enabled chained operations

      • submitAll

         <Error class: unknown class> submitAll(Iterable<String> urls)

        Submit urls to the URL pool, and they will be subsequently processed in the Main loop.

        Submit operations are non-blocking, meaning they return immediately without blocking the current thread or suspending the current coroutine.

        session.submitAll(listOf("http://example.com", "http://example.com/1"))
PulsarContexts.await()
        Parameters:
        urls - The urls to submit
        Returns:

        The PulsarSession itself to enabled chained operations

      • submitAll

         <Error class: unknown class> submitAll(Iterable<String> urls, String args)

        Submit urls to the URL pool, and they will be subsequently processed in the Main loop.

        Submit operations are non-blocking, meaning they return immediately without blocking the current thread or suspending the current coroutine.

        session.submitAll(listOf("http://example.com", "http://example.com/1"), "-expire 1d")
PulsarContexts.await()
        Parameters:
        urls - The urls to submit
        args - The load arguments
        Returns:

        The PulsarSession itself to enabled chained operations

      • submitAll

         <Error class: unknown class> submitAll(Iterable<String> urls, LoadOptions options)

        Submit urls to the URL pool, and they will be subsequently processed in the Main loop.

        Submit operations are non-blocking, meaning they return immediately without blocking the current thread or suspending the current coroutine.

        session.submitAll(listOf("http://example.com", "http://example.com/1"), session.options("-expire 1d"))
PulsarContexts.await()
        Parameters:
        urls - The urls to submit
        options - The load options
        Returns:

        The PulsarSession itself to enabled chained operations

      • submitAll

         <Error class: unknown class> submitAll(Collection<UrlAware> urls)

        Submit urls to the URL pool, and they will be subsequently processed in the Main loop.

        A submit operation is non-blocking, meaning it returns immediately without blocking the current thread or suspending the current coroutine.

        val urls = listOf("http://example.com", "http://example.com/1").map { Hyperlink(it) }
session.submitAll(urls)
PulsarContexts.await()
        Parameters:
        urls - The urls to submit
        Returns:

        The PulsarSession itself to enabled chained operations

      • submitAll

         <Error class: unknown class> submitAll(Collection<UrlAware> urls, String args)

        Submit urls to the URL pool, and they will be subsequently processed in the Main loop.

        A submit operation is non-blocking, meaning it returns immediately without blocking the current thread or suspending the current coroutine.

        val urls = listOf("http://example.com", "http://example.com/1").map { Hyperlink(it) }
session.submitAll(urls, "-expire 1d")
PulsarContexts.await()
        Parameters:
        urls - The urls to submit
        args - The load arguments
        Returns:

        The PulsarSession itself to enabled chained operations

      • loadOutPages

         List<WebPage> loadOutPages(String portalUrl, String args)

        Load or fetch the portal page, and then load or fetch the out links selected by -outLink option.

        Option -outLink specifies the cssSelector for links in the portal page to load.

        val pages = session.loadOutPages("http://example.com", "-outLink a")
val pages2 = session.loadOutPages("http://example.com", "-outLink a -expire 1d")
        Parameters:
        portalUrl - The portal url from where to load pages
        args - The load arguments
        Returns:

        The loaded out pages

      • loadOutPages

         List<WebPage> loadOutPages(String portalUrl, LoadOptions options)

        Load or fetch the portal page, and then load or fetch the out links selected by -outLink option.

        Option -outLink specifies the cssSelector for links in the portal page to load.

        val pages = session.loadOutPages("http://example.com", session.options("-outLink a"))
val pages2 = session.loadOutPages("http://example.com", session.options("-outLink a -expire 1d"))
        Parameters:
        portalUrl - The portal url from where to load pages
        options - The load options
        Returns:

        The loaded out pages

      • loadOutPages

         List<WebPage> loadOutPages(UrlAware portalUrl, String args)

        Load or fetch the portal page, and then load or fetch the out links selected by -outLink option.

        Option -outLink specifies the cssSelector for links in the portal page to load.

        val pages = session.loadOutPages("http://example.com", "-outLink a")
val pages2 = session.loadOutPages("http://example.com", "-outLink a -expire 1d")
        Parameters:
        portalUrl - The portal url from where to load pages
        args - The load arguments
        Returns:

        The loaded out pages

      • loadOutPages

         List<WebPage> loadOutPages(UrlAware portalUrl, LoadOptions options)

        Load or fetch the portal page, and then load or fetch the out links selected by -outLink option.

        Option -outLink specifies the cssSelector for links in the portal page to load.

        val pages = session.loadOutPages("http://example.com", session.options("-outLink a"))
val pages2 = session.loadOutPages("http://example.com", session.options("-outLink a -expire 1d"))
        Parameters:
        portalUrl - The portal url from where to load pages
        options - The load options
        Returns:

        The loaded out pages

      • submitForOutPages

         AbstractPulsarSession submitForOutPages(String portalUrl, String args)

        Load the portal page and submit the out links specified by the -outLink option to the URL pool.

        Option -outLink specifies the cssSelector for links in the portal page to load.

        The submitted urls will be subsequently processed in the Main loop.

        Submit operations are non-blocking, meaning they return immediately without blocking the current thread or suspending the current coroutine.

        session
  .submitForOutPages("http://example.com", "-outLink a[href*=review]")
  .submitForOutPages("http://example.com", "-outLink a[href*=item] -expire 1d")
PulsarContexts.await()
        Parameters:
        portalUrl - The portal url from where to load pages
        args - The load arguments
        Returns:

        The PulsarSession itself to enable chained operation

      • submitForOutPages

         AbstractPulsarSession submitForOutPages(String portalUrl, LoadOptions options)

        Load the portal page and submit the out links specified by the -outLink option to the URL pool.

        Option -outLink specifies the cssSelector for links in the portal page to load.

        The submitted urls will be subsequently processed in the Main loop.

        Submit operations are non-blocking, meaning they return immediately without blocking the current thread or suspending the current coroutine.

        session
  .submitForOutPages("http://example.com", session.options("-outLink a[href*=review]"))
  .submitForOutPages("http://example.com", session.options("-outLink a[href*=item] -expire 1d"))
PulsarContexts.await()
        Parameters:
        portalUrl - The portal url from where to load pages
        options - The load options
        Returns:

        The PulsarSession itself to enable chained operation

      • submitForOutPages

         AbstractPulsarSession submitForOutPages(UrlAware portalUrl, String args)

        Load the portal page and submit the out links specified by the -outLink option to the URL pool.

        Option -outLink specifies the cssSelector for links in the portal page to load.

        The submitted urls will be subsequently processed in the Main loop.

        Submit operations are non-blocking, meaning they return immediately without blocking the current thread or suspending the current coroutine.

        session
  .submitForOutPages(Hyperlink("http://example.com"), "-outLink a[href*=review]")
  .submitForOutPages(Hyperlink("http://example.com"), "-outLink a[href*=item] -expire 1d")
PulsarContexts.await()
        Parameters:
        portalUrl - The portal url from where to load pages
        args - The load arguments
        Returns:

        The PulsarSession itself to enable chained operation

      • submitForOutPages

         AbstractPulsarSession submitForOutPages(UrlAware portalUrl, LoadOptions options)

        Load the portal page and submit the out links specified by the -outLink option to the URL pool.

        Option -outLink specifies the cssSelector for links in the portal page to load.

        The submitted urls will be subsequently processed in the Main loop.

        Submit operations are non-blocking, meaning they return immediately without blocking the current thread or suspending the current coroutine.

        session
  .submitForOutPages(Hyperlink("http://example.com"), session.options("-outLink a[href*=review]"))
  .submitForOutPages(Hyperlink("http://example.com"), session.options("-outLink a[href*=item] -expire 1d"))
PulsarContexts.await()
        Parameters:
        portalUrl - The portal url from where to load pages
        options - The load options
        Returns:

        The PulsarSession itself to enable chained operation

      • loadOutPagesAsync

         List<CompletableFuture<WebPage>> loadOutPagesAsync(String portalUrl, String args)

        Load or fetch the portal page, and then load or fetch the out links selected by -outLink option asynchronously.

        Option -outLink specifies the cssSelector for links in the portal page to load.

        val pages = session.loadOutPagesAsync("http://example.com", "-outLink a")
val pages2 = session.loadOutPagesAsync("http://example.com", "-outLink a -expire 1d")
        Parameters:
        portalUrl - The portal url from where to load pages
        args - The load arguments
        Returns:

        The loaded out pages

      • loadOutPagesAsync

         List<CompletableFuture<WebPage>> loadOutPagesAsync(String portalUrl, LoadOptions options)

        Load or fetch the portal page, and then load or fetch the out links selected by -outLink option asynchronously.

        Option -outLink specifies the cssSelector for links in the portal page to load.

        val pages = session.loadOutPagesAsync("http://example.com", session.options("-outLink a"))
val pages2 = session.loadOutPagesAsync("http://example.com", session.options("-outLink a -expire 1d"))
        Parameters:
        portalUrl - The portal url from where to load pages
        options - The load options
        Returns:

        The loaded out pages

      • loadResource

         <Error class: unknown class> loadResource(String url, String referrer)

        Load a url as a resource without browser rendering.

        Referrer will be opened by a browser first to obtain browsing environment, such as headers and cookies, and then the browsing environment will be applied to the later resource fetching.

        val page = session.loadResource("http://example.com/robots.txt", "http://example.com")
        Parameters:
        url - The url to load
        referrer - The referrer URL
        Returns:

        The webpage containing the resource

      • loadResource

         <Error class: unknown class> loadResource(String url, String referrer, String args)

        Load a url as a resource without browser rendering.

        Referrer will be opened by a browser first to obtain browsing environment, such as headers and cookies, and then the browsing environment will be applied to the later resource fetching.

        val page = session.loadResource("http://example.com/robots.txt", "http://example.com", "-expire 1d")
        Parameters:
        url - The url to load
        referrer - The referrer URL
        args - The load arguments
        Returns:

        The webpage containing the resource

      • loadResource

         <Error class: unknown class> loadResource(String url, String referrer, LoadOptions options)

        Load a url as a resource without browser rendering.

        Referrer will be opened by a browser first to obtain browsing environment, such as headers and cookies, and then the browsing environment will be applied to the later resource fetching.

        val page = session.loadResource("http://example.com/robots.txt", "http://example.com", session.options("-expire 1d"))
        Parameters:
        url - The url to load
        referrer - The referrer URL
        options - The load options
        Returns:

        The webpage containing the resource

      • loadResourceDeferred

         <Error class: unknown class> loadResourceDeferred(String url, String referrer)

        Load a url as a resource without browser rendering.

        Referrer will be opened by a browser first to obtain browsing environment, such as headers and cookies, and then the browsing environment will be applied to the later resource fetching.

        This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.

        val page = session.loadResourceDeferred("http://example.com/robots.txt", "http://example.com")
        Parameters:
        url - The url to load
        referrer - The referrer URL
        Returns:

        The webpage containing the resource

      • loadResourceDeferred

         <Error class: unknown class> loadResourceDeferred(String url, String referrer, String args)

        Load a url as a resource without browser rendering.

        Referrer will be opened by a browser first to obtain browsing environment, such as headers and cookies, and then the browsing environment will be applied to the later resource fetching.

        This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.

        val page = session.loadResourceDeferred("http://example.com/robots.txt", "http://example.com", "-expire 1d")
        Parameters:
        url - The url to load
        referrer - The referrer URL
        args - The load arguments
        Returns:

        The webpage containing the resource

      • loadResourceDeferred

         <Error class: unknown class> loadResourceDeferred(String url, String referrer, LoadOptions options)

        Load a url as a resource without browser rendering.

        Referrer will be opened by a browser first to obtain browsing environment, such as headers and cookies, and then the browsing environment will be applied to the later resource fetching.

        This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.

        val page = session.loadResourceDeferred("http://example.com/robots.txt", "http://example.com", session.options("-expire 1d"))
        Parameters:
        url - The url to load
        referrer - The referrer URL
        options - The load options
        Returns:

        The webpage containing the resource

      • parse

         FeaturedDocument parse(WebPage page)

        Parses the given page into an in-memory HTML document.

        The parse method operates within the Kotlin process, not inside a real browser. It parses the page content into an in-memory lightweight Document Object Model (DOM), making it very fast to analyze within the Kotlin process. To interact with a live DOM in a real browser, use WebDriver.

        val page = session.load("http://example.com")
val document = session.parse(page)
        Parameters:
        page - The webpage to parse
        Returns:

        The parsed HTML document

      • parse

         FeaturedDocument parse(WebPage page, Boolean noCache)

        Parses the given page into an in-memory HTML document.

        The parse method operates within the Kotlin process, not inside a real browser. It parses the page content into an in-memory lightweight Document Object Model (DOM), making it very fast to analyze within the Kotlin process. To interact with a live DOM in a real browser, use WebDriver.

        val page = session.load("http://example.com")
val document = session.parse(page, true)
        Parameters:
        page - The webpage to parse
        noCache - Whether to skip the cache
        Returns:

        The parsed HTML document

      • loadDocument

         FeaturedDocument loadDocument(String url)

        Load or fetch a webpage and parse it into an HTML document

        val document = session.loadDocument("http://example.com")
        Parameters:
        url - The url to load
        Returns:

        The parsed HTML document

      • loadDocument

         FeaturedDocument loadDocument(String url, String args)

        Load or fetch a webpage and parse it into an HTML document

        val document = session.loadDocument("http://example.com", "-expire 1d")
        Parameters:
        url - The url to load
        args - The load arguments
        Returns:

        The parsed HTML document

      • loadDocument

         FeaturedDocument loadDocument(String url, LoadOptions options)

        Load or fetch a webpage and parse it into an HTML document

        val document = session.loadDocument("http://example.com", session.options("-expire 1d"))
        Parameters:
        url - The url to load
        options - The load options
        Returns:

        The parsed HTML document

      • loadDocument

         FeaturedDocument loadDocument(UrlAware url)

        Load or fetch a webpage and parse it into an HTML document

        val document = session.loadDocument(Hyperlink("http://example.com"))
        Parameters:
        url - The url to load
        Returns:

        The parsed HTML document

      • loadDocument

         FeaturedDocument loadDocument(UrlAware url, String args)

        Load or fetch a webpage and parse it into an HTML document

        val document = session.loadDocument(Hyperlink("http://example.com"), "-expire 1d")
        Parameters:
        url - The url to load
        args - The load arguments
        Returns:

        The parsed HTML document

      • loadDocument

         FeaturedDocument loadDocument(UrlAware url, LoadOptions options)

        Load or fetch a webpage and parse it into an HTML document

        val document = session.loadDocument(Hyperlink("http://example.com"), session.options("-expire 1d"))
        Parameters:
        url - The url to load
        options - The load options
        Returns:

        The parsed HTML document

      • loadDocument

         FeaturedDocument loadDocument(NormURL url)

        Load or fetch a webpage and then parse it into an HTML document.

        val document = session.loadDocument("http://example.com")
        Parameters:
        url - The url to load
        Returns:

        The parsed HTML document

      • extract

         Map<String, String> extract(FeaturedDocument document, Iterable<String> fieldSelectors)

        Load or fetch a webpage located by the given url, and then extract fields specified by field selectors.

        val document = session.loadDocument("http://example.com", "-expire 1d")
val fields = session.extract(document, listOf(".title", ".content"))
        Parameters:
        document - The document to extract
        fieldSelectors - The selectors to extract fields
        Returns:

        All the extracted fields and their selectors

      • extract

         List<Map<String, String>> extract(FeaturedDocument document, String restrictSelector, Iterable<String> fieldSelectors)

        Load or fetch a webpage located by the given url, and then extract fields specified by field selectors.

        val document = session.loadDocument("http://example.com", "-expire 1d")
val fields = session.extract(document, "#main-content", listOf(".title", ".content"))
        Parameters:
        document - The document to extract
        restrictSelector - A CSS selector to locate a DOM where all fields are restricted to
        fieldSelectors - The selectors to extract fields
        Returns:

        All the extracted fields and their selectors

      • extract

         Map<String, String> extract(FeaturedDocument document, Map<String, String> fieldSelectors)

        Load or fetch a webpage located by the given url, and then extract fields specified by field selectors.

        val document = session.loadDocument("http://example.com", "-expire 1d")
val fields = session.extract(document, mapOf("title" to ".title", "content" to ".content"))
        Parameters:
        document - The document to extract
        fieldSelectors - The selectors to extract fields
        Returns:

        All the extracted fields and their selectors

      • extract

         List<Map<String, String>> extract(FeaturedDocument document, String restrictSelector, Map<String, String> fieldSelectors)

        Load or fetch a webpage located by the given url, and then extract fields specified by field selectors.

        val document = session.loadDocument("http://example.com", "-expire 1d")
val fields = session.extract(document, "#main-content", mapOf("title" to ".title", "content" to ".content"))
        Parameters:
        document - The document to extract
        restrictSelector - A CSS selector to locate a DOM where all fields are restricted to
        fieldSelectors - The selectors to extract fields
        Returns:

        All the extracted fields and their selectors

      • scrape

         Map<String, String> scrape(String url, String args, Iterable<String> fieldSelectors)

        Load or fetch a webpage located by the given url, and then extract fields specified by field selectors.

        val fields = session.scrape("http://example.com", "-expire 1d", listOf(".title", ".content"))
        Parameters:
        url - The url to scrape
        args - The load arguments
        fieldSelectors - The selectors to extract fields
        Returns:

        All the extracted fields and their selectors

      • scrape

         Map<String, String> scrape(String url, LoadOptions options, Iterable<String> fieldSelectors)

        Load or fetch a webpage located by the given url, and then extract fields specified by field selectors.

        val fields = session.scrape("http://example.com", session.options("-expire 1d"), listOf(".title", ".content"))
        Parameters:
        url - The url to scrape
        options - The load options
        fieldSelectors - The selectors to extract fields
        Returns:

        All the extracted fields and their selectors

      • scrape

         Map<String, String> scrape(String url, String args, Map<String, String> fieldSelectors)

        Load or fetch a webpage located by the given url, and then extract fields specified by field selectors.

        val fields = session.scrape("http://example.com", "-expire 1d", ".title", ".content")
        Parameters:
        url - The url to scrape
        args - The load arguments
        fieldSelectors - The selectors to extract fields
        Returns:

        All the extracted fields and their names

      • scrape

         Map<String, String> scrape(String url, LoadOptions options, Map<String, String> fieldSelectors)

        Load or fetch a webpage located by the given url, and then extract fields specified by field selectors.

        val fields = session.scrape("http://example.com", session.options("-expire 1d"), ".title", ".content")
        Parameters:
        url - The url to scrape
        options - The load options
        fieldSelectors - The selectors to extract fields
        Returns:

        All the extracted fields and their names

      • scrape

         List<Map<String, String>> scrape(String url, String args, String restrictSelector, Iterable<String> fieldSelectors)

        Load or fetch a webpage located by the given url, and then extract fields specified by field selectors.

        val fields = session.scrape("http://example.com", "-expire 1d", ".container", listOf(".title", ".content"))
        Parameters:
        url - The url to scrape
        args - The load arguments
        restrictSelector - A CSS selector to locate a DOM where all fields are restricted to
        fieldSelectors - The selectors to extract fields
        Returns:

        All the extracted fields and their names

      • scrape

         List<Map<String, String>> scrape(String url, LoadOptions options, String restrictSelector, Iterable<String> fieldSelectors)

        Load or fetch a webpage located by the given url, and then extract fields specified by field selectors.

        val fields = session.scrape("http://example.com", session.options("-expire 1d"), ".container",
     listOf(".title", ".content"))
        Parameters:
        url - The url to scrape
        options - The load options
        restrictSelector - A CSS selector to locate a DOM where all fields are restricted to
        fieldSelectors - The selectors to extract fields
        Returns:

        All the extracted fields and their selectors

      • scrape

         List<Map<String, String>> scrape(String url, String args, String restrictSelector, Map<String, String> fieldSelectors)

        Load or fetch a webpage located by the given url, and then extract fields specified by field selectors.

        val fields = session.scrape("http://example.com", "-expire 1d", ".container",
     mapOf("title" to ".title", "content" to ".content"))
        Parameters:
        url - The url to scrape
        args - The load arguments
        restrictSelector - A CSS selector to locate a DOM where all fields are restricted to
        fieldSelectors - The selectors to extract fields
        Returns:

        All the extracted fields and their names

      • scrape

         List<Map<String, String>> scrape(String url, LoadOptions options, String restrictSelector, Map<String, String> fieldSelectors)

        Load or fetch a webpage located by the given url, and then extract fields specified by field selectors.

        val fields = session.scrape("http://example.com", session.options("-expire 1d"), ".container",
     mapOf("title" to ".title", "content" to ".content"))
        Parameters:
        url - The url to scrape
        options - The load options
        restrictSelector - A CSS selector to locate a DOM where all fields are restricted to
        fieldSelectors - The selectors to extract fields
        Returns:

        All the extracted fields and their names

      • scrapeOutPages

         List<Map<String, String>> scrapeOutPages(String portalUrl, String args, Iterable<String> fieldSelectors)

        Load or fetch out pages specified by out link selector, and then extract fields specified by field selectors from each out page.

        val fields = session.scrapeOutPages("http://example.com", "-outLink a", listOf(".title", ".content"))
        Parameters:
        portalUrl - The portal url to start scraping
        args - Load arguments for both the portal page and out pages
        fieldSelectors - CSS selectors to extract fields from out pages
        Returns:

        All extracted fields. For each out page, fields extracted with their selectors are saved in a map.

      • scrapeOutPages

         List<Map<String, String>> scrapeOutPages(String portalUrl, LoadOptions options, Iterable<String> fieldSelectors)

        Load or fetch out pages specified by out link selector, and then extract fields specified by field selectors from each out page.

        val fields = session.scrapeOutPages("http://example.com", session.options("-outLink a"), listOf(".title", ".content"))
        Parameters:
        portalUrl - The portal url to start scraping
        options - Load options for both the portal page and out pages
        fieldSelectors - CSS selectors to extract fields from out pages
        Returns:

        All extracted fields. For each out page, fields extracted with their selectors are saved in a map.

      • scrapeOutPages

         List<Map<String, String>> scrapeOutPages(String portalUrl, String args, String restrictSelector, Iterable<String> fieldSelectors)

        Load or fetch out pages specified by out link selector, and then extract fields specified by field selectors from each out page.

        val fields = session.scrapeOutPages("http://example.com", "-outLink a", mapOf("title" to ".title", "content" to ".content"))
        Parameters:
        portalUrl - The portal url to start scraping
        args - Load arguments for both the portal page and out pages
        restrictSelector - A CSS selector to locate a DOM where all fields are restricted to
        fieldSelectors - CSS selectors to extract fields from out pages
        Returns:

        All extracted fields. For each out page, fields extracted with their selectors are saved in a map.

      • scrapeOutPages

         List<Map<String, String>> scrapeOutPages(String portalUrl, LoadOptions options, String restrictSelector, Iterable<String> fieldSelectors)

        Load or fetch out pages specified by out link selector, and then extract fields specified by field selectors from each out page.

        val fields = session.scrapeOutPages("http://example.com", session.options("-outLink a"),
     mapOf("title" to ".title", "content" to ".content"))
        Parameters:
        portalUrl - The portal url to start scraping
        options - Load options for both the portal page and out pages
        restrictSelector - A CSS selector to locate a DOM where all fields are restricted to
        fieldSelectors - CSS selectors to extract fields from out pages
        Returns:

        All extracted fields. For each out page, fields extracted with their selectors are saved in a map.

      • scrapeOutPages

         List<Map<String, String>> scrapeOutPages(String portalUrl, String args, Map<String, String> fieldSelectors)

        Load or fetch out pages specified by out link selector, and then extract fields specified by field selectors from each out page.

        val fields = session.scrapeOutPages("http://example.com", "-outLink a", mapOf("title" to ".title", "content" to ".content"))
        Parameters:
        portalUrl - The portal url to start scraping
        args - Load arguments for both the portal page and out pages
        fieldSelectors - CSS selectors to extract fields from out pages
        Returns:

        All extracted fields. For each out page, fields extracted with their names are saved in a map.

      • scrapeOutPages

         List<Map<String, String>> scrapeOutPages(String portalUrl, LoadOptions options, Map<String, String> fieldSelectors)

        Load or fetch out pages specified by out link selector, and then extract fields specified by field selectors from each out page.

        val fields = session.scrapeOutPages("http://example.com", session.options("-outLink a"),
     mapOf("title" to ".title", "content" to ".content"))
        Parameters:
        portalUrl - The portal url to start scraping
        options - Load options for both the portal page and out pages
        fieldSelectors - CSS selectors to extract fields from out pages
        Returns:

        All extracted fields. For each out page, fields extracted with their names are saved in a map.

      • scrapeOutPages

         List<Map<String, String>> scrapeOutPages(String portalUrl, String args, String restrictSelector, Map<String, String> fieldSelectors)

        Load or fetch out pages specified by out link selector, and then extract fields specified by field selectors from each out page.

        val fields = session.scrapeOutPages("http://example.com", "-outLink a", ".container",
     mapOf("title" to ".title", "content" to ".content"))
        Parameters:
        portalUrl - The portal url to start scraping
        args - Load arguments for both the portal page and out pages
        restrictSelector - A CSS selector to locate a DOM where all fields are restricted to
        fieldSelectors - CSS selectors to extract fields from out pages
        Returns:

        All extracted fields. For each out page, fields extracted with their names are saved in a map.

      • scrapeOutPages

         List<Map<String, String>> scrapeOutPages(String portalUrl, LoadOptions options, String restrictSelector, Map<String, String> fieldSelectors)

        Load or fetch out pages specified by out link selector, and then extract fields specified by field selectors from each out page.

        val fields = session.scrapeOutPages("http://example.com", session.options("-outLink a"), ".container",
     mapOf("title" to ".title", "content" to ".content"))
        Parameters:
        portalUrl - The portal url to start scraping
        options - Load options for both the portal page and out pages
        restrictSelector - A CSS selector to locate a DOM where all fields are restricted to
        fieldSelectors - CSS selectors to extract fields from out pages
        Returns:

        All extracted fields. For each out page, fields extracted with their names are saved in a map.

      • chat

         ModelResponse chat(String prompt)

        Initiates a chat with the AI model using a general prompt. This method sends the provided prompt to the AI model without any additional context.

        Parameters:
        prompt - The prompt or query to be sent to the AI model.
        Returns:

        The response from the AI model encapsulated in a ModelResponse object.

      • chat

         ModelResponse chat(String prompt, WebPage page)

        Initiates a chat with the AI model about a specific webpage. The method sends the provided prompt along with the HTML source code of the specified webpage to the AI model.

        Parameters:
        prompt - The prompt or query to be sent to the AI model.
        page - The WebPage object containing the webpage's content to provide context for the chat.
        Returns:

        The response from the AI model encapsulated in a ModelResponse object.

      • chat

         ModelResponse chat(String prompt, FeaturedDocument document)

        Initiates a chat with the AI model about a specific document. The method sends the provided prompt along with the text content of the specified document to the AI model.

        Parameters:
        prompt - The prompt or query to be sent to the AI model.
        document - The FeaturedDocument object containing the document's text content to provide context for the chat.
        Returns:

        The response from the AI model encapsulated in a ModelResponse object.

      • chat

         ModelResponse chat(String prompt, Element element)

        Initiates a chat with the AI model about a specific HTML element. The method sends the provided prompt along with the text content of the specified HTML element to the AI model.

        Parameters:
        prompt - The prompt or query to be sent to the AI model.
        element - The Element object containing the HTML element's text content to provide context for the chat.
        Returns:

        The response from the AI model encapsulated in a ModelResponse object.

      • data

         Object data(String name)

        Get a variable which is stored in this session

        Parameters:
        name - The name of the variable
        Returns:

        The value of the variable

      • data

         <Error class: unknown class> data(String name, Object value)

        Store a variable in this session

        Parameters:
        name - The name of the variable
        value - The value of the variable

      • delete

         Unit delete(String url)

        Delete a webpage from the storage

        session.delete("http://example.com")
        Parameters:
        url - The url to delete

      • persist

         Boolean persist(WebPage page)

        Persist the webpage.

        val page = session.load("http://example.com")
session.persist(page)
        Parameters:
        page - Page to persist
        Returns:

        Whether the page is persisted successfully

      • export

         Path export(WebPage page)

        Exports the content of a webpage to a file.

        This method saves the content of the given webpage to a file and returns the path where the file is stored.

        Example usage:

        val page = session.load("http://example.com")
val path = session.export(page)
        Parameters:
        page - The webpage to export.
        Returns:

        The path of the exported file.

      • export

         Path export(WebPage page, String ident)

        Exports the content of a webpage to a file with a specified identifier.

        This method saves the content of the given webpage to a file, using the provided identifier to distinguish the file from others, and returns the path where the file is stored.

        Example usage:

        val page = session.load("http://example.com")
val path = session.export(page, "example")
        Parameters:
        page - The webpage to export.
        ident - A file name identifier used to distinguish the exported file.
        Returns:

        The path of the exported file.

      • export

         Path export(FeaturedDocument doc)

        Exports the outer HTML of a document to a file.

        This method saves the outer HTML of the given document to a file and returns the path where the file is stored.

        Parameters:
        doc - The document to export.
        Returns:

        The path of the exported file.

      • export

         Path export(FeaturedDocument doc, String ident)

        Exports the outer HTML of a document to a file with a specified identifier.

        This method saves the outer HTML of the given document to a file, using the provided identifier to distinguish the file from others, and returns the path where the file is stored.

        Example usage:

        val document = session.loadDocument("http://example.com")
val path = session.export(document, "example")
        Parameters:
        doc - The document to export.
        ident - A file name identifier used to distinguish the exported file.
        Returns:

        The path of the exported file.

      • exportTo

         Path exportTo(WebPage page, Path path)

        Exports the content of a webpage to a specified file path.

        This method saves the content of the given webpage to the specified file path and returns the path where the file is stored.

        Example usage:

        val page = session.load("http://example.com")
val path = session.exportTo(page, Paths.get("/tmp/example.html"))
        Parameters:
        page - The webpage to export.
        path - The file path where the content will be saved.
        Returns:

        The path of the exported file.

      • exportTo

         Path exportTo(FeaturedDocument doc, Path path)

        Exports the outer HTML of a document to a specified file path.

        This method saves the outer HTML of the given document to the specified file path and returns the path where the file is stored.

        Example usage:

        val document = session.loadDocument("http://example.com")
val path = session.exportTo(document, Paths.get("/tmp/example.html"))
        Parameters:
        doc - The document to export.
        path - The file path where the content will be saved.
        Returns:

        The path of the exported file.