Package ai.platon.pulsar.skeleton.session

Class AbstractPulsarSession

  • All Implemented Interfaces:
    ai.platon.pulsar.skeleton.session.PulsarSession , java.lang.AutoCloseable 
    
public abstract class AbstractPulsarSession
 implements PulsarSession

    Created by vincent on 18-1-17. Copyright @ 2013-2023 Platon AI. All rights reserved

    • Method Detail

      • getUnmodifiedConfig

         ImmutableConfig getUnmodifiedConfig()

        This is an immutable configuration, loaded from the configuration file during process startup. Once the process has started, this configuration remains unchangeable.

      • getPageCache

         ConcurrentExpiringLRUCache<String, WebPage> getPageCache()

        The global page cache.

      • getDocumentCache

         ConcurrentExpiringLRUCache<String, FeaturedDocument> getDocumentCache()

        The global document cache.

      • getSessionConfig

         VolatileConfig getSessionConfig()

        The session scope volatile config, every setting is supposed to be changed at any time and any place

      • getId

         Integer getId()

        The session id. Session id is expected to be set by the container, e.g. the h2 database runtime

      • normalize

         LoadOptions normalize(LoadOptions options)

        Parse the args and create a LoadOptions instance.

        Note: if the url is submitted, the session is the one from a crawler loop such as StreamingCrawlerLoop.

        Parameters:
        options - The options to normalize.

      • normalize

         NormURL normalize(String url)

        Normalize a url.

        Parameters:
        url - The url to normalize
        Returns:

        The normalized url

      • normalize

         NormURL normalize(String url, String args, Boolean toItemOption)

        Normalize a url.

        Parameters:
        url - The url to normalize
        args - The arguments
        toItemOption - If the LoadOptions is converted to item load options
        Returns:

        The normalized url

      • normalize

         NormURL normalize(String url, LoadOptions options, Boolean toItemOption)

        Normalize a url.

        Parameters:
        url - The url to normalize
        options - The LoadOptions applied to the url
        toItemOption - If the LoadOptions is converted to item load options
        Returns:

        The normalized url

      • normalize

         List<NormURL> normalize(Iterable<String> urls, String args, Boolean toItemOption)

        Normalize urls, remove invalid ones.

        Parameters:
        urls - The urls to normalize
        args - The arguments
        toItemOption - If the LoadOptions is converted to item load options
        Returns:

        All normalized urls

      • normalize

         List<NormURL> normalize(Iterable<String> urls, LoadOptions options, Boolean toItemOption)

        Normalize urls, remove invalid ones.

        Parameters:
        urls - The urls to normalize
        options - The LoadOptions applied to each url
        toItemOption - If the LoadOptions is converted to item load options
        Returns:

        All normalized urls

      • normalize

         NormURL normalize(UrlAware url)

        Normalize a url, return NormURL.NIL if the url is invalid.

        Parameters:
        url - The url to normalize
        Returns:

        The normalized url

      • normalize

         NormURL normalize(UrlAware url, String args, Boolean toItemOption)

        Normalize a url, return NormURL.NIL if the url is invalid.

        Parameters:
        url - The url to normalize
        args - The arguments
        toItemOption - If the LoadOptions is converted to item load options
        Returns:

        The normalized url

      • normalize

         NormURL normalize(UrlAware url, LoadOptions options, Boolean toItemOption)

        Normalize a url, return NormURL.NIL if the url is invalid.

        Parameters:
        url - The url to normalize
        options - The LoadOptions applied to the url
        toItemOption - If the LoadOptions is converted to item load options
        Returns:

        The normalized url

      • normalize

         List<NormURL> normalize(Collection<UrlAware> urls)

        Normalize urls, remove invalid ones.

        Parameters:
        urls - The urls to normalize
        Returns:

        All normalized urls

      • normalize

         List<NormURL> normalize(Collection<UrlAware> urls, String args, Boolean toItemOption)

        Normalize urls, remove invalid ones

        Parameters:
        urls - The urls to normalize
        args - The arguments
        toItemOption - If the LoadOptions is converted to item load options
        Returns:

        All normalized urls, all invalid input urls are removed

      • normalize

         List<NormURL> normalize(Collection<UrlAware> urls, LoadOptions options, Boolean toItemOption)

        Normalize urls, remove invalid ones

        Parameters:
        urls - The urls to normalize
        options - The LoadOptions applied to each url
        toItemOption - If the LoadOptions is converted to item load options
        Returns:

        All normalized urls, all invalid input urls are removed

      • normalizeOrNull

         NormURL normalizeOrNull(String url, LoadOptions options, Boolean toItemOption)

        Normalize a url, return null if the url is invalid.

        Parameters:
        url - The url to normalize
        options - The LoadOptions applied to the url
        toItemOption - If the LoadOptions is converted to item load options
        Returns:

        The normalized url or null

      • normalizeOrNull

         NormURL normalizeOrNull(UrlAware url, LoadOptions options, Boolean toItemOption)

        Normalize a url, return null if the url is invalid.

        Parameters:
        url - The url to normalize
        options - The LoadOptions applied to the url
        toItemOption - If the LoadOptions is converted to item load options
        Returns:

        The normalized url or null

      • get

         WebPage get(String url)

        Get a page from local storage.

        val url = "http://example.com"
val page = session.get(url)
        Parameters:
        url - The url
        Returns:

        The webpage in storage if exists, otherwise returns a NIL page

      • get

         WebPage get(String url, String fields)

        Get a page from local storage.

        val url = "http://example.com"
val page = session.get(url, "content", "metadata")
        Parameters:
        url - The url
        fields - The fields to load from local storage
        Returns:

        The webpage in storage if exists, otherwise returns a NIL page

      • getOrNull

         WebPage getOrNull(String url)

        Get a page from local storage.

        val url = "http://example.com"
val page = session.getOrNull(url)
        Parameters:
        url - The url
        Returns:

        The page in storage if exists, otherwise returns null

      • getOrNull

         WebPage getOrNull(String url, String fields)

        Get a page from local storage.

        val url = "http://example.com"
val page = session.getOrNull(url, "content", "metadata")
        Parameters:
        url - The url
        fields - The fields to load from local storage
        Returns:

        The page in storage if exists, otherwise returns null

      • getContent

         ByteBuffer getContent(String url)

        Get the content of the page from the storage.

        val url = "http://example.com"
val content = session.getContent(url)
        Parameters:
        url - The url of the page to retrieve
        Returns:

        The page content or null

      • getContentAsString

         String getContentAsString(String url)

        Get the content of the page from the storage

        val url = "http://example.com"
val content = session.getContentAsString(url)
        Parameters:
        url - The url of the page to retrieve
        Returns:

        The page content in string format or null

      • exists

         Boolean exists(String url)

        Check if the page exists in the storage.

        val url = "http://example.com"
val exists = session.exists(url)
        Parameters:
        url - The url to check
        Returns:

        true if the page exists, false otherwise

      • fetchState

         CheckState fetchState(WebPage page, LoadOptions options)

        Return the fetch state of the page.

        val url = "http://example.com"
val state = session.fetchState(url)
        Parameters:
        page - The webpage
        options - The load options
        Returns:

        The fetch state of the page

      • open

         WebPage open(String url)

        Open a url.

        This method opens the url immediately, regardless of the previous state of the page.

        val url = "http://example.com"
val page = session.open(url)
        Parameters:
        url - The url to open
        Returns:

        The webpage loaded or NIL

      • open

         WebPage open(String url, PageEventHandlers eventHandlers)

        Open a url with page events.

        This method opens the url immediately, regardless of the previous state of the page.

        val url = "http://example.com"
val event = PrintFlowEvent()
val page = session.open(url, event)
        Parameters:
        url - The url to open
        Returns:

        The webpage loaded or NIL

      • open

         WebPage open(String url, WebDriver driver)

        Open a url with webdriver.

        This method opens the url immediately, regardless of the previous state of the page.

        val url = "http://example.com"
val page = session.open(url, driver)
        Parameters:
        url - The url to open
        Returns:

        The webpage loaded or NIL

      • open

         WebPage open(String url, WebDriver driver, PageEventHandlers eventHandlers)

        Open a url with page events and webdriver.

        This method opens the url immediately, regardless of the previous state of the page.

        val url = "http://example.com"
val event = PrintFlowEvent()
val page = session.open(url, event)
        Parameters:
        url - The url to open
        Returns:

        The webpage loaded or NIL

      • connect

         Unit connect(WebDriver driver)

        Connect a webpage to a webdriver.

        val page = session.connect(driver)
        Returns:

        The webpage connected to the webdriver or NIL

      • load

         WebPage load(String url)

        Load an url.

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        Other fetch conditions can be specified by load arguments:

        • expiration

        • page size requirement

        • fields requirement

        • other

        val url = "http://example.com"
val page = session.load(url)
        Parameters:
        url - The url to load
        Returns:

        The webpage loaded or NIL

      • load

         WebPage load(String url, String args)

        Load a url with specified arguments.

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        Fetch conditions can be specified by load arguments:

        • expiration

        • page size requirement

        • fields requirement

        • other

        val url = "http://example.com"
val args = "-expire 1d"
val page = session.load(url, args)
        Parameters:
        url - The url to load
        args - The load arguments
        Returns:

        The webpage loaded or NIL

      • load

         WebPage load(String url, LoadOptions options)

        Load a url with specified options.

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        Fetch conditions can be specified by load arguments:

        • expiration

        • page size requirement

        • fields requirement

        • other

        val url = "http://example.com"
val options = session.options("-expire 1d")
val page = session.load(url, options)
        Parameters:
        url - The url to load
        options - The load options
        Returns:

        The webpage loaded or NIL

      • load

         WebPage load(UrlAware url)

        Load a url with the specified arguments.

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        val url = Hyperlink("http://example.com")
val page = session.load(url)
        Parameters:
        url - The url to load
        Returns:

        The webpage loaded or NIL

      • load

         WebPage load(UrlAware url, String args)

        Load a url with the specified arguments.

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        val url = Hyperlink("http://example.com")
val page = session.load(url, "-expire 1d")
        Parameters:
        url - The url to load
        args - The load arguments
        Returns:

        The webpage loaded or NIL

      • load

         WebPage load(UrlAware url, LoadOptions options)

        Load a url with options.

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        val url = Hyperlink("http://example.com")
val options = session.options("-expire 1d")
val page = session.load(url, options)
        Parameters:
        url - The url to load
        options - The load options
        Returns:

        The webpage loaded or NIL

      • load

         WebPage load(NormURL url)

        Load a normal url.

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        val url = session.normalize("http://example.com")
val page = session.load(url)

        • @param url The normal url

        Returns:

        The webpage loaded or NIL

      • loadDeferred

         WebPage loadDeferred(String url, String args)

        Load a url with specified options.

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.

        val url = "http://example.com"
val args = "-expire 1d"
val page = session.loadDeferred(url, args)
        Parameters:
        url - The url to load
        args - The load options
        Returns:

        The webpage loaded or NIL

      • loadDeferred

         WebPage loadDeferred(String url, LoadOptions options)

        Load a url with specified options.

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.

        val url = "http://example.com"
val options = session.options("-expire 1d")
val page = session.loadDeferred(url, options)
        Parameters:
        url - The url to load
        options - The load options
        Returns:

        The webpage loaded or NIL

      • loadDeferred

         WebPage loadDeferred(UrlAware url, String args)

        Load a url with specified arguments.

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.

        val url = Hyperlink("http://example.com")
val page = session.loadDeferred(url, "-expire 1d")
        Parameters:
        url - The url to load
        args - The load args
        Returns:

        The webpage loaded or NIL

      • loadDeferred

         WebPage loadDeferred(UrlAware url, LoadOptions options)

        Load a url with specified options.

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.

        val url = Hyperlink("http://example.com")
val options = session.options("-expire 1d")
val page = session.loadDeferred(url, options)
        Parameters:
        url - The url to load
        options - The load options
        Returns:

        The webpage loaded or NIL

      • loadDeferred

         WebPage loadDeferred(NormURL url)

        Load a url with specified options

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.

        val url = session.normalize("http://example.com")
val page = session.loadDeferred(url)
        Parameters:
        url - The normal url
        Returns:

        The webpage loaded or NIL

      • loadAll

         List<WebPage> loadAll(Iterable<String> urls)

        Load all urls with specified options

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        val urls = listOf("http://example.com", "http://example.com/1")
val pages = session.loadAll(urls)
        Parameters:
        urls - The urls to load
        Returns:

        The successfully loaded webpages, all failed urls are ignored

      • loadAll

         List<WebPage> loadAll(Iterable<String> urls, String args)

        Load all urls with specified options

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        val urls = listOf("http://example.com", "http://example.com/1")
val pages = session.loadAll(urls, "-expire 1d")
        Parameters:
        urls - The urls to load
        args - The load arguments
        Returns:

        The successfully loaded webpages, all failed urls are ignored

      • loadAll

         List<WebPage> loadAll(Iterable<String> urls, LoadOptions options)

        Load all urls with specified options

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        val urls = listOf("http://example.com", "http://example.com/1")
val pages = session.loadAll(urls, session.options("-expire 1d"))
        Parameters:
        urls - The urls to load
        options - The load options
        Returns:

        The successfully loaded webpages, all failed urls are ignored

      • loadAll

         List<WebPage> loadAll(Collection<UrlAware> urls)

        Load all urls with specified options

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        val urls = listOf(Hyperlink("http://example.com"), Hyperlink("http://example.com/1"))
val pages = session.loadAll(urls)
        Parameters:
        urls - The urls to load
        Returns:

        The successfully loaded webpages, all failed urls are ignored

      • loadAll

         List<WebPage> loadAll(Collection<UrlAware> urls, String args)

        Load all urls with specified options

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        val urls = listOf(Hyperlink("http://example.com"), Hyperlink("http://example.com/1"))
val pages = session.loadAll(urls, "-expire 1d")
        Parameters:
        urls - The urls to load
        args - The load arguments
        Returns:

        The successfully loaded webpages, all failed urls are ignored

      • loadAll

         List<WebPage> loadAll(Collection<UrlAware> urls, LoadOptions options)

        Load all urls with specified options

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        val urls = listOf(Hyperlink("http://example.com"), Hyperlink("http://example.com/1"))
val pages = session.loadAll(urls, session.options("-expire 1d"))
        Parameters:
        urls - The urls to load
        options - The load options
        Returns:

        The successfully loaded webpages, all failed urls are ignored

      • loadAll

         List<WebPage> loadAll(List<NormURL> normUrls)

        Load all normal urls with specified options

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        val urls = listOf("http://example.com", "http://example.com/1").map { session.normalize(it) }
val pages = session.loadAll(urls)
        Parameters:
        normUrls - The normal urls to load
        Returns:

        The successfully loaded webpages, all failed urls are ignored

      • loadAsync

         CompletableFuture<WebPage> loadAsync(String url)

        Load a url in java async style

        String url = "http://example.com";
WebPage page = session.loadAsync(url).join();
        Parameters:
        url - The url to load
        Returns:

        A completable future of webpage

      • loadAsync

         CompletableFuture<WebPage> loadAsync(String url, String args)

        Load a url in java async style

        String url = "http://example.com";
WebPage page = session.loadAsync(url, "-expire 1d").join();
        Parameters:
        url - The url to load
        Returns:

        A completable future of webpage

      • loadAsync

         CompletableFuture<WebPage> loadAsync(String url, LoadOptions options)

        Load a url in java async style

        String url = "http://example.com";
WebPage page = session.loadAsync(url, session.options("-expire 1d")).join();
        Parameters:
        url - The url to load
        Returns:

        A completable future of webpage

      • loadAsync

         CompletableFuture<WebPage> loadAsync(UrlAware url)

        Load a url in java async style

        Hyperlink url = new Hyperlink("http://example.com");
WebPage page = session.loadAsync(url).join();
        Parameters:
        url - The url to load
        Returns:

        A completable future of webpage

      • loadAsync

         CompletableFuture<WebPage> loadAsync(UrlAware url, String args)

        Load a url in java async style

        Hyperlink url = new Hyperlink("http://example.com");
WebPage page = session.loadAsync(url, "-expire 1d").join();
        Parameters:
        url - The url to load
        Returns:

        A completable future of webpage

      • loadAsync

         CompletableFuture<WebPage> loadAsync(UrlAware url, LoadOptions options)

        Load a url in java async style

        Hyperlink url = new Hyperlink("http://example.com");
WebPage page = session.loadAsync(url, session.options("-expire 1d")).join();
        Parameters:
        url - The url to load
        Returns:

        A completable future of webpage

      • loadAsync

         CompletableFuture<WebPage> loadAsync(NormURL url)

        Load a normal url in java async style

        NormURL url = session.normalize("http://example.com");
WebPage page = session.loadAsync(url).join();
        Parameters:
        url - The normal url to load
        Returns:

        A completable future of webpage

      • loadAllAsync

         List<CompletableFuture<WebPage>> loadAllAsync(Iterable<String> urls)

        Load all urls in java async style

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        List<String> urls = Arrays.asList("http://example.com", "http://example.com/1");
CompletableFuture<?>[] pages = session.loadAllAsync(urls).toArray(CompletableFuture<?>[]::new);
CompletableFuture.allOf(futures).join();
        Parameters:
        urls - The normal urls to load
        Returns:

        The completable futures of webpages

      • loadAllAsync

         List<CompletableFuture<WebPage>> loadAllAsync(Iterable<String> urls, String args)

        Load all urls in java async style.

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        List<String> urls = Arrays.asList("http://example.com", "http://example.com/1");
CompletableFuture<?>[] pages = session.loadAllAsync(urls, "-expire 1d").toArray(CompletableFuture<?>[]::new);
CompletableFuture.allOf(futures).join();
        Parameters:
        urls - The normal urls to load
        Returns:

        The completable futures of webpages

      • loadAllAsync

         List<CompletableFuture<WebPage>> loadAllAsync(Iterable<String> urls, LoadOptions options)

        Load all normal urls in java async style.

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        List<String> urls = Arrays.asList("http://example.com", "http://example.com/1");
CompletableFuture<?>[] pages = session.loadAllAsync(urls, session.options("-expire 1d")).toArray(CompletableFuture<?>[]::new);
CompletableFuture.allOf(futures).join();
        Parameters:
        urls - The normal urls to load
        Returns:

        The completable futures of webpages

      • loadAllAsync

         List<CompletableFuture<WebPage>> loadAllAsync(Collection<UrlAware> urls)

        Load all urls in java async style.

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        List<UrlAware> urls = Arrays.asList(new Hyperlink("http://example.com"), new Hyperlink("http://example.com/1"));
CompletableFuture<?>[] pages = session.loadAllAsync(urls).toArray(CompletableFuture<?>[]::new);
CompletableFuture.allOf(futures).join();
        Parameters:
        urls - The normal urls to load
        Returns:

        The completable futures of webpages

      • loadAllAsync

         List<CompletableFuture<WebPage>> loadAllAsync(Collection<UrlAware> urls, String args)

        Load all normal urls in java async style.

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        List<UrlAware> urls = Arrays.asList(new Hyperlink("http://example.com"), new Hyperlink("http://example.com/1"));
CompletableFuture<?>[] pages = session.loadAllAsync(urls, "-expire 1d").toArray(CompletableFuture<?>[]::new);
CompletableFuture.allOf(futures).join();
        Parameters:
        urls - The normal urls to load
        Returns:

        The completable futures of webpages

      • loadAllAsync

         List<CompletableFuture<WebPage>> loadAllAsync(Collection<UrlAware> urls, LoadOptions options)

        Load all normal urls in java async style.

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        List<UrlAware> urls = Arrays.asList(new Hyperlink("http://example.com"), new Hyperlink("http://example.com/1"));
CompletableFuture<?>[] pages = session.loadAllAsync(urls, session.options("-expire 1d")).toArray(CompletableFuture<?>[]::new);
CompletableFuture.allOf(futures).join();
        Parameters:
        urls - The normal urls to load
        Returns:

        The completable futures of webpages

      • loadAllAsync

         List<CompletableFuture<WebPage>> loadAllAsync(List<NormURL> urls)

        Load all normal urls in java async style.

        This method initially verifies the presence of the page in the local store. If the page exists and meets the specified requirements, it returns the local version. Otherwise, it fetches the page from the Internet.

        List<NormURL> urls = Arrays.asList(session.normalize("http://example.com"), session.normalize("http://example.com/1"));
CompletableFuture<?>[] pages = session.loadAllAsync(urls).toArray(CompletableFuture<?>[]::new);
CompletableFuture.allOf(futures).join();
        Parameters:
        urls - The normal urls to load
        Returns:

        The completable futures of webpages

      • submit

         AbstractPulsarSession submit(String url)

        Submit a url to the URL pool, and it will be subsequently processed in the crawl loop.

        A submit operation is non-blocking, meaning it returns immediately without blocking the current thread or suspending the current coroutine.

        session.submit("http://example.com")
PulsarContexts.await()
        Parameters:
        url - The url to submit
        Returns:

        The PulsarSession itself to enabled chained operations

      • submit

         AbstractPulsarSession submit(String url, String args)

        Submit a url to the URL pool, and it will be subsequently processed in the crawl loop.

        A submit operation is non-blocking, meaning it returns immediately without blocking the current thread or suspending the current coroutine.

        session.submit("http://example.com", "-expire 1d")
PulsarContexts.await()
        Parameters:
        url - The url to submit
        args - The load arguments
        Returns:

        The PulsarSession itself to enabled chained operations

      • submit

         AbstractPulsarSession submit(String url, LoadOptions options)

        Submit a url to the URL pool, and it will be subsequently processed in the crawl loop.

        A submit operation is non-blocking, meaning it returns immediately without blocking the current thread or suspending the current coroutine.

        session.submit("http://example.com", session.options("-expire 1d"))
PulsarContexts.await()

        Submit can be used with event listeners to handle the page events.

        The code snippet below shows how to submit a url with an attached load event handler. This handler is invoked once the page is loaded, and it prints the URL of the page.

        val options = session.options("-expire 1d")
options.eventHandlers.loadEvent.onLoaded.addLast { println(it.url) }
session.submit("http://example.com", options)
PulsarContexts.await()

        The code snippet below shows how to submit a hyperlink with every supported event handlers attached. Each event handler will print a message when its associated event is triggered.

        val event = PrintFlowEvent()
val options = session.options("-expire 1d", event)
session.submit("http://example.com", options)
PulsarContexts.await()
        Parameters:
        url - The url to submit
        options - The load options
        Returns:

        The PulsarSession itself to enabled chained operations

      • submit

         AbstractPulsarSession submit(UrlAware url)

        Submit a url to the URL pool, and it will be subsequently processed in the crawl loop.

        A submit operation is non-blocking, meaning it returns immediately without blocking the current thread or suspending the current coroutine.

        Basic usage:

        session.submit(Hyperlink("http://example.com"))
PulsarContexts.await()

        Submit methods are commonly be used in conjunction with event listeners to handle page events.

        The code snippet below shows how to submit a hyperlink with an attached load event handler. This handler is invoked once the page is loaded, and it prints the URL of the page.

        val hyperlink = ListenableHyperlink("http://example.com")
hyperlink.eventHandlers.loadEvent.onLoaded.addLast { page -> println(page.url) }
session.submit(hyperlink)
PulsarContexts.await()

        The code snippet below shows how to submit a hyperlink with a event handler attached that processes the parsed document once the relevant event is triggered.

        val hyperlink = ListenableHyperlink("http://example.com", args = "-parse", event = PrintFlowEvent())
hyperlink.eventHandlers.loadEvent.onHTMLDocumentParsed.addLast { page, document ->
     val title = document.selectFirstOrNull(".title")?.text() ?: "Not found"
     println(title)
}
session.submit(hyperlink)
PulsarContexts.await()

        ParsableHyperlink can be used to simplify the load-and-parse tasks.

        val hyperlink = ParsableHyperlink("http://example.com") { page, document ->
val title = document.selectFirstOrNull(".title")?.text() ?: ""
     println(title)
}
session.submit(hyperlink)
PulsarContexts.await()

        The code snippet below shows how to submit a hyperlink with every supported event handlers attached. Each event handler will print a message when its associated event is triggered.

        val event = PrintFlowEvent()
val options = session.options("-expire 1d", event)
session.submit("http://example.com", options)
PulsarContexts.await()
        Parameters:
        url - The url to submit
        Returns:

        The PulsarSession itself to enabled chained operations

      • submit

         AbstractPulsarSession submit(UrlAware url, String args)

        Submit a url to the URL pool, and it will be subsequently processed in the crawl loop.

        A submit operation is non-blocking, meaning it returns immediately without blocking the current thread or suspending the current coroutine.

        session.submit(Hyperlink("http://example.com"), "-expire 1d")
PulsarContexts.await()
        Parameters:
        url - The url to submit
        args - The load arguments
        Returns:

        The PulsarSession itself to enabled chained operations

      • submitAll

         AbstractPulsarSession submitAll(Iterable<String> urls)

        Submit urls to the URL pool, and they will be subsequently processed in the crawl loop.

        Submit operations are non-blocking, meaning they return immediately without blocking the current thread or suspending the current coroutine.

        session.submitAll(listOf("http://example.com", "http://example.com/1"))
PulsarContexts.await()
        Parameters:
        urls - The urls to submit
        Returns:

        The PulsarSession itself to enabled chained operations

      • submitAll

         AbstractPulsarSession submitAll(Iterable<String> urls, String args)

        Submit urls to the URL pool, and they will be subsequently processed in the crawl loop.

        Submit operations are non-blocking, meaning they return immediately without blocking the current thread or suspending the current coroutine.

        session.submitAll(listOf("http://example.com", "http://example.com/1"), "-expire 1d")
PulsarContexts.await()
        Parameters:
        urls - The urls to submit
        args - The load arguments
        Returns:

        The PulsarSession itself to enabled chained operations

      • submitAll

         AbstractPulsarSession submitAll(Iterable<String> urls, LoadOptions options)

        Submit urls to the URL pool, and they will be subsequently processed in the crawl loop.

        Submit operations are non-blocking, meaning they return immediately without blocking the current thread or suspending the current coroutine.

        session.submitAll(listOf("http://example.com", "http://example.com/1"), session.options("-expire 1d"))
PulsarContexts.await()
        Parameters:
        urls - The urls to submit
        options - The load options
        Returns:

        The PulsarSession itself to enabled chained operations

      • submitAll

         AbstractPulsarSession submitAll(Collection<UrlAware> urls)

        Submit urls to the URL pool, and they will be subsequently processed in the crawl loop.

        A submit operation is non-blocking, meaning it returns immediately without blocking the current thread or suspending the current coroutine.

        val urls = listOf("http://example.com", "http://example.com/1").map { Hyperlink(it) }
session.submitAll(urls)
PulsarContexts.await()
        Parameters:
        urls - The urls to submit
        Returns:

        The PulsarSession itself to enabled chained operations

      • submitAll

         AbstractPulsarSession submitAll(Collection<UrlAware> urls, String args)

        Submit urls to the URL pool, and they will be subsequently processed in the crawl loop.

        A submit operation is non-blocking, meaning it returns immediately without blocking the current thread or suspending the current coroutine.

        val urls = listOf("http://example.com", "http://example.com/1").map { Hyperlink(it) }
session.submitAll(urls, "-expire 1d")
PulsarContexts.await()
        Parameters:
        urls - The urls to submit
        Returns:

        The PulsarSession itself to enabled chained operations

      • loadOutPages

         List<WebPage> loadOutPages(String portalUrl, String args)

        Load or fetch the portal page, and then load or fetch the out links selected by -outLink option.

        Option -outLink specifies the cssSelector for links in the portal page to load.

        val pages = session.loadOutPages("http://example.com", "-outLink a")
val pages2 = session.loadOutPages("http://example.com", "-outLink a -expire 1d")
        Parameters:
        portalUrl - The portal url from where to load pages
        args - The load arguments
        Returns:

        The loaded out pages

      • loadOutPages

         List<WebPage> loadOutPages(String portalUrl, LoadOptions options)

        Load or fetch the portal page, and then load or fetch the out links selected by -outLink option.

        Option -outLink specifies the cssSelector for links in the portal page to load.

        val pages = session.loadOutPages("http://example.com", session.options("-outLink a"))
val pages2 = session.loadOutPages("http://example.com", session.options("-outLink a -expire 1d"))
        Parameters:
        portalUrl - The portal url from where to load pages
        options - The load options
        Returns:

        The loaded out pages

      • loadOutPages

         List<WebPage> loadOutPages(UrlAware portalUrl, String args)

        Load or fetch the portal page, and then load or fetch the out links selected by -outLink option.

        Option -outLink specifies the cssSelector for links in the portal page to load.

        val pages = session.loadOutPages("http://example.com", "-outLink a")
val pages2 = session.loadOutPages("http://example.com", "-outLink a -expire 1d")
        Parameters:
        portalUrl - The portal url from where to load pages
        args - The load arguments
        Returns:

        The loaded out pages

      • loadOutPages

         List<WebPage> loadOutPages(UrlAware portalUrl, LoadOptions options)

        Load or fetch the portal page, and then load or fetch the out links selected by -outLink option.

        Option -outLink specifies the cssSelector for links in the portal page to load.

        val pages = session.loadOutPages("http://example.com", session.options("-outLink a"))
val pages2 = session.loadOutPages("http://example.com", session.options("-outLink a -expire 1d"))
        Parameters:
        portalUrl - The portal url from where to load pages
        options - The load options
        Returns:

        The loaded out pages

      • submitForOutPages

         AbstractPulsarSession submitForOutPages(String portalUrl, String args)

        Load the portal page and submit the out links specified by the -outLink option to the URL pool.

        Option -outLink specifies the cssSelector for links in the portal page to load.

        The submitted urls will be subsequently processed in the crawl loop.

        Submit operations are non-blocking, meaning they return immediately without blocking the current thread or suspending the current coroutine.

        session
  .submitForOutPages("http://example.com", "-outLink a[href*=review]")
  .submitForOutPages("http://example.com", "-outLink a[href*=item] -expire 1d")
PulsarContexts.await()
        Parameters:
        portalUrl - The portal url from where to load pages
        args - The load arguments
        Returns:

        The PulsarSession itself to enable chained operation

      • submitForOutPages

         AbstractPulsarSession submitForOutPages(String portalUrl, LoadOptions options)

        Load the portal page and submit the out links specified by the -outLink option to the URL pool.

        Option -outLink specifies the cssSelector for links in the portal page to load.

        The submitted urls will be subsequently processed in the crawl loop.

        Submit operations are non-blocking, meaning they return immediately without blocking the current thread or suspending the current coroutine.

        session
  .submitForOutPages("http://example.com", session.options("-outLink a[href*=review]"))
  .submitForOutPages("http://example.com", session.options("-outLink a[href*=item] -expire 1d"))
PulsarContexts.await()
        Parameters:
        portalUrl - The portal url from where to load pages
        options - The load options
        Returns:

        The PulsarSession itself to enable chained operation

      • submitForOutPages

         AbstractPulsarSession submitForOutPages(UrlAware portalUrl, String args)

        Load the portal page and submit the out links specified by the -outLink option to the URL pool.

        Option -outLink specifies the cssSelector for links in the portal page to load.

        The submitted urls will be subsequently processed in the crawl loop.

        Submit operations are non-blocking, meaning they return immediately without blocking the current thread or suspending the current coroutine.

        session
  .submitForOutPages(Hyperlink("http://example.com"), "-outLink a[href*=review]")
  .submitForOutPages(Hyperlink("http://example.com"), "-outLink a[href*=item] -expire 1d")
PulsarContexts.await()
        Parameters:
        portalUrl - The portal url from where to load pages
        args - The load arguments
        Returns:

        The PulsarSession itself to enable chained operation

      • submitForOutPages

         AbstractPulsarSession submitForOutPages(UrlAware portalUrl, LoadOptions options)

        Load the portal page and submit the out links specified by the -outLink option to the URL pool.

        Option -outLink specifies the cssSelector for links in the portal page to load.

        The submitted urls will be subsequently processed in the crawl loop.

        Submit operations are non-blocking, meaning they return immediately without blocking the current thread or suspending the current coroutine.

        session
  .submitForOutPages(Hyperlink("http://example.com"), session.options("-outLink a[href*=review]"))
  .submitForOutPages(Hyperlink("http://example.com"), session.options("-outLink a[href*=item] -expire 1d"))
PulsarContexts.await()
        Parameters:
        portalUrl - The portal url from where to load pages
        options - The load options
        Returns:

        The PulsarSession itself to enable chained operation

      • loadOutPagesAsync

         List<CompletableFuture<WebPage>> loadOutPagesAsync(String portalUrl, String args)

        Load or fetch the portal page, and then load or fetch the out links selected by -outLink option asynchronously.

        Option -outLink specifies the cssSelector for links in the portal page to load.

        val pages = session.loadOutPagesAsync("http://example.com", "-outLink a")
val pages2 = session.loadOutPagesAsync("http://example.com", "-outLink a -expire 1d")
        Parameters:
        portalUrl - The portal url from where to load pages
        args - The load arguments
        Returns:

        The loaded out pages

      • loadOutPagesAsync

         List<CompletableFuture<WebPage>> loadOutPagesAsync(String portalUrl, LoadOptions options)

        Load or fetch the portal page, and then load or fetch the out links selected by -outLink option asynchronously.

        Option -outLink specifies the cssSelector for links in the portal page to load.

        val pages = session.loadOutPagesAsync("http://example.com", session.options("-outLink a"))
val pages2 = session.loadOutPagesAsync("http://example.com", session.options("-outLink a -expire 1d"))
        Parameters:
        portalUrl - The portal url from where to load pages
        options - The load options
        Returns:

        The loaded out pages

      • loadResource

         WebPage loadResource(String url, String referrer)

        Load a url as a resource without browser rendering.

        Referrer will be opened by a browser first to obtain browsing environment, such as headers and cookies, and then the browsing environment will be applied to the later resource fetching.

        val page = session.loadResource("http://example.com/robots.txt", "http://example.com")
        Parameters:
        url - The url to load
        referrer - The referrer URL
        Returns:

        The webpage containing the resource

      • loadResource

         WebPage loadResource(String url, String referrer, String args)

        Load a url as a resource without browser rendering.

        Referrer will be opened by a browser first to obtain browsing environment, such as headers and cookies, and then the browsing environment will be applied to the later resource fetching.

        val page = session.loadResource("http://example.com/robots.txt", "http://example.com", "-expire 1d")
        Parameters:
        url - The url to load
        referrer - The referrer URL
        args - The load arguments
        Returns:

        The webpage containing the resource

      • loadResource

         WebPage loadResource(String url, String referrer, LoadOptions options)

        Load a url as a resource without browser rendering.

        Referrer will be opened by a browser first to obtain browsing environment, such as headers and cookies, and then the browsing environment will be applied to the later resource fetching.

        val page = session.loadResource("http://example.com/robots.txt", "http://example.com", session.options("-expire 1d"))
        Parameters:
        url - The url to load
        referrer - The referrer URL
        options - The load options
        Returns:

        The webpage containing the resource

      • loadResourceDeferred

         WebPage loadResourceDeferred(String url, String referrer)

        Load a url as a resource without browser rendering.

        Referrer will be opened by a browser first to obtain browsing environment, such as headers and cookies, and then the browsing environment will be applied to the later resource fetching.

        This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.

        val page = session.loadResourceDeferred("http://example.com/robots.txt", "http://example.com")
        Parameters:
        url - The url to load
        referrer - The referrer URL
        Returns:

        The webpage containing the resource

      • loadResourceDeferred

         WebPage loadResourceDeferred(String url, String referrer, String args)

        Load a url as a resource without browser rendering.

        Referrer will be opened by a browser first to obtain browsing environment, such as headers and cookies, and then the browsing environment will be applied to the later resource fetching.

        This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.

        val page = session.loadResourceDeferred("http://example.com/robots.txt", "http://example.com", "-expire 1d")
        Parameters:
        url - The url to load
        referrer - The referrer URL
        args - The load arguments
        Returns:

        The webpage containing the resource

      • loadResourceDeferred

         WebPage loadResourceDeferred(String url, String referrer, LoadOptions options)

        Load a url as a resource without browser rendering.

        Referrer will be opened by a browser first to obtain browsing environment, such as headers and cookies, and then the browsing environment will be applied to the later resource fetching.

        This function is a kotlin suspend function, which could be started, paused, and resume. Suspend functions are only allowed to be called from a coroutine or another suspend function.

        val page = session.loadResourceDeferred("http://example.com/robots.txt", "http://example.com", session.options("-expire 1d"))
        Parameters:
        url - The url to load
        referrer - The referrer URL
        options - The load options
        Returns:

        The webpage containing the resource

      • parse

         FeaturedDocument parse(WebPage page)

        Parse a webpage into an HTML document.

        val page = session.load("http://example.com")
val document = session.parse(page)
        Parameters:
        page - The webpage to parse
        Returns:

        The parsed HTML document

      • parse

         FeaturedDocument parse(WebPage page, Boolean noCache)

        Parse a webpage into an HTML document.

        val page = session.load("http://example.com")
val document = session.parse(page, true)
        Parameters:
        page - The webpage to parse
        noCache - Whether to skip the cache
        Returns:

        The parsed HTML document

      • loadDocument

         FeaturedDocument loadDocument(String url)

        Load or fetch a webpage and parse it into an HTML document

        val document = session.loadDocument("http://example.com")
        Parameters:
        url - The url to load
        Returns:

        The parsed HTML document

      • loadDocument

         FeaturedDocument loadDocument(String url, String args)

        Load or fetch a webpage and parse it into an HTML document

        val document = session.loadDocument("http://example.com", "-expire 1d")
        Parameters:
        url - The url to load
        args - The load arguments
        Returns:

        The parsed HTML document

      • loadDocument

         FeaturedDocument loadDocument(String url, LoadOptions options)

        Load or fetch a webpage and parse it into an HTML document

        val document = session.loadDocument("http://example.com", session.options("-expire 1d"))
        Parameters:
        url - The url to load
        options - The load options
        Returns:

        The parsed HTML document

      • loadDocument

         FeaturedDocument loadDocument(UrlAware url)

        Load or fetch a webpage and parse it into an HTML document

        val document = session.loadDocument(Hyperlink("http://example.com"))
        Parameters:
        url - The url to load
        Returns:

        The parsed HTML document

      • loadDocument

         FeaturedDocument loadDocument(UrlAware url, String args)

        Load or fetch a webpage and parse it into an HTML document

        val document = session.loadDocument(Hyperlink("http://example.com"), "-expire 1d")
        Parameters:
        url - The url to load
        args - The load arguments
        Returns:

        The parsed HTML document

      • loadDocument

         FeaturedDocument loadDocument(UrlAware url, LoadOptions options)

        Load or fetch a webpage and parse it into an HTML document

        val document = session.loadDocument(Hyperlink("http://example.com"), session.options("-expire 1d"))
        Parameters:
        url - The url to load
        options - The load options
        Returns:

        The parsed HTML document

      • loadDocument

         FeaturedDocument loadDocument(NormURL url)

        Load or fetch a webpage and then parse it into an HTML document.

        val document = session.loadDocument("http://example.com")
        Parameters:
        url - The url to load
        Returns:

        The parsed HTML document

      • scrape

         Map<String, String> scrape(String url, String args, Iterable<String> fieldSelectors)

        Load or fetch a webpage located by the given url, and then extract fields specified by field selectors.

        val fields = session.scrape("http://example.com", "-expire 1d", listOf(".title", ".content"))
        Parameters:
        url - The url to scrape
        args - The load arguments
        fieldSelectors - The selectors to extract fields
        Returns:

        All the extracted fields and their selectors

      • scrape

         Map<String, String> scrape(String url, LoadOptions options, Iterable<String> fieldSelectors)

        Load or fetch a webpage located by the given url, and then extract fields specified by field selectors.

        val fields = session.scrape("http://example.com", session.options("-expire 1d"), listOf(".title", ".content"))
        Parameters:
        url - The url to scrape
        options - The load options
        fieldSelectors - The selectors to extract fields
        Returns:

        All the extracted fields and their selectors

      • scrape

         Map<String, String> scrape(String url, String args, Map<String, String> fieldSelectors)

        Load or fetch a webpage located by the given url, and then extract fields specified by field selectors.

        val fields = session.scrape("http://example.com", "-expire 1d", ".title", ".content")
        Parameters:
        url - The url to scrape
        args - The load arguments
        fieldSelectors - The selectors to extract fields
        Returns:

        All the extracted fields and their names

      • scrape

         Map<String, String> scrape(String url, LoadOptions options, Map<String, String> fieldSelectors)

        Load or fetch a webpage located by the given url, and then extract fields specified by field selectors.

        val fields = session.scrape("http://example.com", session.options("-expire 1d"), ".title", ".content")
        Parameters:
        url - The url to scrape
        options - The load options
        fieldSelectors - The selectors to extract fields
        Returns:

        All the extracted fields and their names

      • scrape

         List<Map<String, String>> scrape(String url, String args, String restrictSelector, Iterable<String> fieldSelectors)

        Load or fetch a webpage located by the given url, and then extract fields specified by field selectors.

        val fields = session.scrape("http://example.com", "-expire 1d", ".container", listOf(".title", ".content"))
        Parameters:
        url - The url to scrape
        args - The load arguments
        restrictSelector - A CSS selector to locate a DOM where all fields are restricted to
        fieldSelectors - The selectors to extract fields
        Returns:

        All the extracted fields and their names

      • scrape

         List<Map<String, String>> scrape(String url, LoadOptions options, String restrictSelector, Iterable<String> fieldSelectors)

        Load or fetch a webpage located by the given url, and then extract fields specified by field selectors.

        val fields = session.scrape("http://example.com", session.options("-expire 1d"), ".container",
     listOf(".title", ".content"))
        Parameters:
        url - The url to scrape
        options - The load options
        restrictSelector - A CSS selector to locate a DOM where all fields are restricted to
        fieldSelectors - The selectors to extract fields
        Returns:

        All the extracted fields and their selectors

      • scrape

         List<Map<String, String>> scrape(String url, String args, String restrictSelector, Map<String, String> fieldSelectors)

        Load or fetch a webpage located by the given url, and then extract fields specified by field selectors.

        val fields = session.scrape("http://example.com", "-expire 1d", ".container",
     mapOf("title" to ".title", "content" to ".content"))
        Parameters:
        url - The url to scrape
        args - The load arguments
        restrictSelector - A CSS selector to locate a DOM where all fields are restricted to
        fieldSelectors - The selectors to extract fields
        Returns:

        All the extracted fields and their names

      • scrape

         List<Map<String, String>> scrape(String url, LoadOptions options, String restrictSelector, Map<String, String> fieldSelectors)

        Load or fetch a webpage located by the given url, and then extract fields specified by field selectors.

        val fields = session.scrape("http://example.com", session.options("-expire 1d"), ".container",
     mapOf("title" to ".title", "content" to ".content"))
        Parameters:
        url - The url to scrape
        options - The load options
        restrictSelector - A CSS selector to locate a DOM where all fields are restricted to
        fieldSelectors - The selectors to extract fields
        Returns:

        All the extracted fields and their names

      • scrapeOutPages

         List<Map<String, String>> scrapeOutPages(String portalUrl, String args, Iterable<String> fieldSelectors)

        Load or fetch out pages specified by out link selector, and then extract fields specified by field selectors from each out page.

        val fields = session.scrapeOutPages("http://example.com", "-outLink a", listOf(".title", ".content"))
        Parameters:
        portalUrl - The portal url to start scraping
        args - Load arguments for both the portal page and out pages
        fieldSelectors - CSS selectors to extract fields from out pages
        Returns:

        All extracted fields. For each out page, fields extracted with their selectors are saved in a map.

      • scrapeOutPages

         List<Map<String, String>> scrapeOutPages(String portalUrl, LoadOptions options, Iterable<String> fieldSelectors)

        Load or fetch out pages specified by out link selector, and then extract fields specified by field selectors from each out page.

        val fields = session.scrapeOutPages("http://example.com", session.options("-outLink a"), listOf(".title", ".content"))
        Parameters:
        portalUrl - The portal url to start scraping
        options - Load options for both the portal page and out pages
        fieldSelectors - CSS selectors to extract fields from out pages
        Returns:

        All extracted fields. For each out page, fields extracted with their selectors are saved in a map.

      • scrapeOutPages

         List<Map<String, String>> scrapeOutPages(String portalUrl, String args, String restrictSelector, Iterable<String> fieldSelectors)

        Load or fetch out pages specified by out link selector, and then extract fields specified by field selectors from each out page.

        val fields = session.scrapeOutPages("http://example.com", "-outLink a", mapOf("title" to ".title", "content" to ".content"))
        Parameters:
        portalUrl - The portal url to start scraping
        args - Load arguments for both the portal page and out pages
        restrictSelector - A CSS selector to locate a DOM where all fields are restricted to
        fieldSelectors - CSS selectors to extract fields from out pages
        Returns:

        All extracted fields. For each out page, fields extracted with their selectors are saved in a map.

      • scrapeOutPages

         List<Map<String, String>> scrapeOutPages(String portalUrl, LoadOptions options, String restrictSelector, Iterable<String> fieldSelectors)

        Load or fetch out pages specified by out link selector, and then extract fields specified by field selectors from each out page.

        val fields = session.scrapeOutPages("http://example.com", session.options("-outLink a"),
     mapOf("title" to ".title", "content" to ".content"))
        Parameters:
        portalUrl - The portal url to start scraping
        options - Load options for both the portal page and out pages
        restrictSelector - A CSS selector to locate a DOM where all fields are restricted to
        fieldSelectors - CSS selectors to extract fields from out pages
        Returns:

        All extracted fields. For each out page, fields extracted with their selectors are saved in a map.

      • scrapeOutPages

         List<Map<String, String>> scrapeOutPages(String portalUrl, String args, Map<String, String> fieldSelectors)

        Load or fetch out pages specified by out link selector, and then extract fields specified by field selectors from each out page.

        val fields = session.scrapeOutPages("http://example.com", "-outLink a", mapOf("title" to ".title", "content" to ".content"))
        Parameters:
        portalUrl - The portal url to start scraping
        args - Load arguments for both the portal page and out pages
        fieldSelectors - CSS selectors to extract fields from out pages
        Returns:

        All extracted fields. For each out page, fields extracted with their names are saved in a map.

      • scrapeOutPages

         List<Map<String, String>> scrapeOutPages(String portalUrl, LoadOptions options, Map<String, String> fieldSelectors)

        Load or fetch out pages specified by out link selector, and then extract fields specified by field selectors from each out page.

        val fields = session.scrapeOutPages("http://example.com", session.options("-outLink a"),
     mapOf("title" to ".title", "content" to ".content"))
        Parameters:
        portalUrl - The portal url to start scraping
        options - Load options for both the portal page and out pages
        fieldSelectors - CSS selectors to extract fields from out pages
        Returns:

        All extracted fields. For each out page, fields extracted with their names are saved in a map.

      • scrapeOutPages

         List<Map<String, String>> scrapeOutPages(String portalUrl, String args, String restrictSelector, Map<String, String> fieldSelectors)

        Load or fetch out pages specified by out link selector, and then extract fields specified by field selectors from each out page.

        val fields = session.scrapeOutPages("http://example.com", "-outLink a", ".container",
     mapOf("title" to ".title", "content" to ".content"))
        Parameters:
        portalUrl - The portal url to start scraping
        args - Load arguments for both the portal page and out pages
        restrictSelector - A CSS selector to locate a DOM where all fields are restricted to
        fieldSelectors - CSS selectors to extract fields from out pages
        Returns:

        All extracted fields. For each out page, fields extracted with their names are saved in a map.

      • scrapeOutPages

         List<Map<String, String>> scrapeOutPages(String portalUrl, LoadOptions options, String restrictSelector, Map<String, String> fieldSelectors)

        Load or fetch out pages specified by out link selector, and then extract fields specified by field selectors from each out page.

        val fields = session.scrapeOutPages("http://example.com", session.options("-outLink a"), ".container",
     mapOf("title" to ".title", "content" to ".content"))
        Parameters:
        portalUrl - The portal url to start scraping
        options - Load options for both the portal page and out pages
        restrictSelector - A CSS selector to locate a DOM where all fields are restricted to
        fieldSelectors - CSS selectors to extract fields from out pages
        Returns:

        All extracted fields. For each out page, fields extracted with their names are saved in a map.

      • harvest

         TextDocument harvest(String url, String args, String engine)

        Harvest the content of a webpage using a web content extractor engine.

        Available engines are "boilerpipe".

        val document = session.harvest("http://example.com", "-expire 1d")
println(document.textContent)
        Parameters:
        url - The url to harvest
        args - The load arguments
        engine - The content extractor engine, default is "boilerpipe".
        Returns:

        The harvested text document

      • harvest

         TextDocument harvest(WebPage page, String engine)

        Harvest the content of a webpage using a web content extractor engine.

        Available engines are "boilerpipe".

        val page = session.load("http://example.com")
val document = session.harvest(page, "boilerpipe")
println(document.textContent)
        Parameters:
        page - The webpage to harvest
        engine - The content extractor engine, default is "boilerpipe".
        Returns:

        The harvested text document

      • chat

         ModelResponse chat(String prompt)

        Chat with the AI model.

        Parameters:
        prompt - The prompt to chat with
        Returns:

        The response from the model

      • chat

         ModelResponse chat(String userMessage, String systemMessage)

        Chat with the AI model.

        Parameters:
        userMessage - Represents a message from a user, typically an end user of the application.
        systemMessage - Represents a system message, typically defined by a developer.
        Returns:

        The response from the model.

      • chat

         ModelResponse chat(WebPage page, String prompt)

        Chat with the AI model about the specified webpage.

        Parameters:
        page - The page to chat with
        prompt - The prompt to chat with
        Returns:

        The response from the model

      • chat

         ModelResponse chat(String prompt, WebPage page)

        Chat with the AI model about the specified webpage.

        Parameters:
        prompt - The prompt to chat with
        page - The page to chat with
        Returns:

        The response from the model

      • chat

         ModelResponse chat(FeaturedDocument document, String prompt)

        Chat with the AI model about the specified document.

        Parameters:
        document - The document to chat with
        prompt - The prompt to chat with
        Returns:

        The response from the model

      • chat

         ModelResponse chat(String prompt, FeaturedDocument document)

        Chat with the AI model about the specified document.

        Parameters:
        prompt - The prompt to chat with
        document - The document to chat with
        Returns:

        The response from the model

      • chat

         ModelResponse chat(Element element, String prompt)

        Chat with the AI model about the specified element.

        Parameters:
        element - The element to chat with
        prompt - The prompt to chat with
        Returns:

        The response from the model

      • chat

         ModelResponse chat(String prompt, Element element)

        Chat with the AI model about the specified element.

        Parameters:
        prompt - The prompt to chat with
        element - The element to chat with
        Returns:

        The response from the model

      • instruct

         InstructionResult instruct(String prompt, WebDriver driver)

        Instructs the webdriver to perform a series of actions based on the given prompt. This function converts the prompt into a sequence of webdriver actions, which are then executed.

        Parameters:
        prompt - The textual prompt that describes the actions to be performed by the webdriver.
        driver - The webdriver instance that will execute the actions.
        Returns:

        The response from the model, though in this implementation, the return value is not explicitly used.

      • data

         Object data(String name)

        Get a variable which is stored in this session

        Parameters:
        name - The name of the variable
        Returns:

        The value of the variable

      • data

         Unit data(String name, Object value)

        Store a variable in this session

        Parameters:
        name - The name of the variable
        value - The value of the variable

      • delete

         Unit delete(String url)

        Delete a webpage from the storage

        session.delete("http://example.com")
        Parameters:
        url - The url to delete

      • persist

         Boolean persist(WebPage page)

        Persist the webpage.

        val page = session.load("http://example.com")
session.persist(page)
        Parameters:
        page - Page to persist
        Returns:

        Whether the page is persisted successfully

      • export

         Path export(WebPage page)

        Export the content of a webpage.

        val page = session.load("http://example.com")
val path = session.export(page)
        Parameters:
        page - Page to export
        Returns:

        The path of the exported page

      • export

         Path export(WebPage page, String ident)

        Export the content of a webpage.

        val page = session.load("http://example.com")
val path = session.export(page, "example")
        Parameters:
        page - Page to export
        ident - File name identifier used to distinguish from other files
        Returns:

        The path of the exported page

      • export

         Path export(FeaturedDocument doc)

        Export the outer HTML of the document.

        Parameters:
        doc - Document to export
        Returns:

        The path of the exported document

      • export

         Path export(FeaturedDocument doc, String ident)

        Export the outer HTML of the document.

        val document = session.loadDocument("http://example.com")
val path = session.export(document, "example")
        Parameters:
        doc - Document to export
        ident - File name identifier used to distinguish from other files
        Returns:

        The path of the exported document

      • exportTo

         Path exportTo(WebPage page, Path path)

        Export the content of a webpage.

        val page = session.load("http://example.com")
val path = session.exportTo(page, Paths.get("/tmp/example.html"))
        Parameters:
        page - Webpage to export
        path - Path to save the exported content
        Returns:

        The path of the exported document

      • exportTo

         Path exportTo(FeaturedDocument doc, Path path)

        Export the outer HTML of the document.

        val document = session.loadDocument("http://example.com")
val path = session.exportTo(document, Paths.get("/tmp/example.html"))
        Parameters:
        doc - Document to export
        path - Path to save the exported content
        Returns:

        The path of the exported document