Package ai.platon.pulsar.skeleton

Class PulsarSettings

  • All Implemented Interfaces:
    
public class PulsarSettings

    The PulsarSettings class defines a convenient interface to control the behavior of PulsarRPA.

    For example, to run multiple temporary browsers in headless mode, which is usually used in the spider scenario, you can use the following code:

    PulsarSettings()
   .headless()
   .maxBrowsers(4)
   .maxOpenTabs(12)
   .enableUrlBlocking()

    The above code will:

    • Run the browser in headless mode

    • Set the number of privacy contexts to 4

    • Set the max number of open tabs in each browser context to 12

    • Enable url blocking

    If you want to run your system's default browser in GUI mode, and interact with the webpage, you can use the following code:

    PulsarSettings().withSystemDefaultBrowser().withGUI().withSPA()

    The above code will:

    • Use the system's default browser

    • Run the browser in GUI mode

    • Set the system to work with single page application

    • Constructor Detail

      • PulsarSettings

        PulsarSettings()

    • Method Detail

      • withSystemDefaultBrowser

         final PulsarSettings withSystemDefaultBrowser()

        Use the system's default Chrome browser, so PulsarRPA visits websites just like you do. Any change to the browser will be kept.

      • withSystemDefaultBrowser

         final PulsarSettings withSystemDefaultBrowser(BrowserType browserType)

        Use the system's default browser with the given type, so PulsarRPA visits websites just like you do. Any change to the browser will be kept.

      • withDefaultBrowser

         final PulsarSettings withDefaultBrowser()

        Use the default browser which has an isolated profile and user data directory. Any modifications made to the browser will be preserved, including the cookies, history, etc.

      • withDefaultBrowser

         final PulsarSettings withDefaultBrowser(BrowserType browserType)

        Use the default browser which has an isolated profile and user data directory. Any modifications made to the browser will be preserved, including the cookies, history, etc.

      • withPrototypeBrowser

         final PulsarSettings withPrototypeBrowser()

        Use Google Chrome with the prototype environment. Any modifications made to the browser will be preserved. Sequential and temporary browsers will inherit the environment from the prototype browser.

      • withPrototypeBrowser

         final PulsarSettings withPrototypeBrowser(BrowserType browserType)

        Use the specified browser with the prototype environment. Any modifications made to the browser will be preserved. Sequential and temporary browsers will inherit the environment from the prototype browser.

      • withSequentialBrowsers

         final PulsarSettings withSequentialBrowsers()

        Use sequential browsers that inherits the prototype browser’s environment. The sequential browsers are permanent unless the context directories are deleted manually.

        Returns:

        the PulsarSettings itself

      • withSequentialBrowsers

         final PulsarSettings withSequentialBrowsers(BrowserType browserType)

        Use sequential browsers that inherits the prototype browser’s environment. The sequential browsers are permanent unless the context directories are deleted manually.

        Returns:

        the PulsarSettings itself

      • withSequentialBrowsers

         final PulsarSettings withSequentialBrowsers(BrowserType browserType, Integer maxAgents)

        Use sequential browsers that inherits from the prototype browser’s environment. The sequential browsers are permanent unless the context directories are deleted manually.

        Parameters:
        maxAgents - The maximum number of sequential privacy agents, the active privacy contexts is chosen from them.
        Returns:

        the PulsarSettings itself

      • withTemporaryBrowser

         final PulsarSettings withTemporaryBrowser()

        Use a temporary browser that inherits from the prototype browser’s environment. The temporary browser will not be used again after it is shut down.

      • withTemporaryBrowser

         final PulsarSettings withTemporaryBrowser(BrowserType browserType)

        Use a temporary browser that inherits from the prototype browser’s environment. The temporary browser will not be used again after it is shut down.

      • withSPA

         final PulsarSettings withSPA()

        Tell the system to work with single page application. To collect SPA data, the execution needs to have no timeout limit.

      • withInteractLevel

         final PulsarSettings withInteractLevel(InteractLevel level)

        Defines the level of interaction with a webpage during crawling.

        Higher levels involve more interaction (e.g., scrolling, clicking), which may improve content extraction quality at the cost of speed.

      • enableUrlBlocking

         final PulsarSettings enableUrlBlocking()

        Enable url blocking. If url blocking is enabled and the blocking rules are set, resources matching the rules will be blocked by the browser.

      • enableOriginalPageContentAutoExporting

         final PulsarSettings enableOriginalPageContentAutoExporting()

        Export all pages automatically once they are fetched.

        The export directory is under AppPaths.WEB_CACHE_DIR. A typical export path is:

        • AppPaths.WEB_CACHE_DIR/default/pulsar_chrome/OK/amazon-com

        • C:\Users\pereg\AppData\Local\Temp\pulsar-pereg\cache\web\default\pulsar_chrome\OK\amazon-com

      • enableOriginalPageContentAutoExporting

         final PulsarSettings enableOriginalPageContentAutoExporting(Integer limit)

        Export at most limit pages once they are fetched.

        The export directory is under AppPaths.WEB_CACHE_DIR. A typical export path is:

        • AppPaths.WEB_CACHE_DIR/default/pulsar_chrome/OK/amazon-com

        • C:\Users\pereg\AppData\Local\Temp\pulsar-pereg\cache\web\default\pulsar_chrome\OK\amazon-com

      • withLLM

         final PulsarSettings withLLM(String provider, String name, String apiKey)

        Sets the Large Language Model (LLM) provider for the PulsarRPA settings.

        This function allows specifying the LLM provider to be used. The provider must be a non-null string.

        Supported LLM providers include:

        • <a href='https://www.volcengine.com/docs/82379/1399008'>Volcengine API</a>

        • <a href='https://api-docs.deepseek.com/'>DeepSeek API</a>

        For example, you can use the following code to set the LLM provider:

        // use volcengine as the LLM provider
PulsarSettings().withLLM("volcengine", "ep-20250218132011-2scs8", apiKey)
        Parameters:
        provider - The name of the LLM provider to be used.
        Returns:

        The current instance of PulsarSettings to allow method chaining.

      • withLLMProvider

         final PulsarSettings withLLMProvider(String provider)

        Sets the Large Language Model (LLM) provider for the PulsarRPA settings.

        This function allows specifying the LLM provider to be used. The provider must be a non-null string.

        Supported LLM providers include:

        • <a href='https://www.volcengine.com/docs/82379/1399008'>Volcengine API</a>

        • <a href='https://api-docs.deepseek.com/'>DeepSeek API</a>

        For example, you can use the following code to set the LLM provider:

        PulsarSettings()
    .withLLMProvider("volcengine") // use volcengine as the LLM provider
    .withLLMName("ep-20250218132011-2scs8") // the LLM name, you should change it to your own
    .withLLMAPIKey(apiKey) // the LLM api key, you should change it to your own
        Parameters:
        provider - The name of the LLM provider to be used.
        Returns:

        The current instance of PulsarSettings to allow method chaining.

      • withLLMName

         final PulsarSettings withLLMName(String name)

        Sets the Large Language Model (LLM) name for the PulsarRPA settings.

        Supported LLM providers include:

        • <a href='https://www.volcengine.com/docs/82379/1399008'>Volcengine API</a>

        • <a href='https://api-docs.deepseek.com/'>DeepSeek API</a>

        For example, you can use the following code to set the LLM provider:

        PulsarSettings()
    .withLLMProvider("volcengine") // use volcengine as the LLM provider
    .withLLMName("ep-20250218132011-2scs8") // the LLM name, you should change it to your own
    .withLLMAPIKey(apiKey) // the LLM api key, you should change it to your own
        Parameters:
        name - The name of the Large Language Model (LLM) to be set.
        Returns:

        The current instance of PulsarSettings to allow for method chaining.

      • withLLMAPIKey

         final PulsarSettings withLLMAPIKey(String key)

        Sets the Large Language Model (LLM) API key for the PulsarRPA settings.

        Supported LLM providers include:

        • <a href='https://www.volcengine.com/docs/82379/1399008'>Volcengine API</a>

        • <a href='https://api-docs.deepseek.com/'>DeepSeek API</a>

        For example, you can use the following code to set the LLM provider:

        PulsarSettings()
    .withLLMProvider("volcengine") // use volcengine as the LLM provider
    .withLLMName("ep-20250218132011-2scs8") // the LLM name, you should change it to your own
    .withLLMAPIKey(apiKey) // the LLM api key, you should change it to your own
        Parameters:
        key - The API key to be set.
        Returns:

        The current instance of PulsarSettings to allow for method chaining.