Class MultiPrivacyContextManager
-
- All Implemented Interfaces:
-
ai.platon.pulsar.protocol.browser.emulator.context.BrowserPrivacyManager,ai.platon.pulsar.skeleton.crawl.fetch.privacy.PrivacyManager,java.lang.AutoCloseable
public class MultiPrivacyContextManager extends AbstractBrowserPrivacyManager
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description public final classMultiPrivacyContextManager.Metricspublic classMultiPrivacyContextManager.Companion
-
Field Summary
Fields Modifier and Type Field Description private final IntegermaxAllowedBadContextsprivate DurationsnapshotDumpIntervalprivate final MultiPrivacyContextManager.Metricsmetricsprivate final CoreMetricscoreMetricsprivate final BrowserManagerbrowserManagerprivate final BrowserFactorybrowserFactoryprivate final WebDriverPoolManagerdriverPoolManagerprivate final ProxyPoolManagerproxyPoolManagerprivate final ImmutableConfigconfprivate final BooleanisActiveprivate final BooleanisClosedprivate final Map<BrowserProfile, PrivacyContext>activeContextsprivate final ObjectcontextLifeCycleMonitorprivate final ConcurrentLinkedDeque<PrivacyContext>deadContextsprivate final ConcurrentHashMap<BrowserProfile, PrivacyContext>permanentContextsprivate final PrivacyAgentGeneratorprivacyAgentGeneratorprivate final ConcurrentHashMap<BrowserProfile, PrivacyContext>temporaryContextsprivate final ConcurrentLinkedDeque<PrivacyContext>zombieContextspublic final static MultiPrivacyContextManager.CompanionCompanion
-
Constructor Summary
Constructors Constructor Description MultiPrivacyContextManager(WebDriverPoolManager driverPoolManager)MultiPrivacyContextManager(WebDriverPoolManager driverPoolManager, ImmutableConfig conf)MultiPrivacyContextManager(ImmutableConfig conf)MultiPrivacyContextManager(WebDriverPoolManager driverPoolManager, ProxyPoolManager proxyPoolManager, CoreMetrics coreMetrics, ImmutableConfig conf)
-
Method Summary
Modifier and Type Method Description final IntegergetMaxAllowedBadContexts()final DurationgetSnapshotDumpInterval()final UnitsetSnapshotDumpInterval(Duration snapshotDumpInterval)final MultiPrivacyContextManager.MetricsgetMetrics()final CoreMetricsgetCoreMetrics()FetchResultrun(FetchTask task, SuspendFunction2<FetchTask, WebDriver, FetchResult> fetchFun)Run a task in a privacy context. BrowserPrivacyContextcreateUnmanagedContext(BrowserProfile profile)Create a privacy context who is not added to the context list yet. PrivacyContexttryGetNextReadyPrivacyContext(WebPage page, Fingerprint fingerprint, FetchTask task)Try to get a ready privacy context. PrivacyContexttryGetNextReadyPrivacyContext(Fingerprint fingerprint)PrivacyContexttryGetNextUnderLoadedPrivacyContext(WebPage page, Fingerprint fingerprint, FetchTask task)Gets an under-loaded privacy context, which can be either ready or not ready. PrivacyContextgetOrCreate(BrowserProfile profile)Unitmaintain(Boolean force)Maintain all the privacy contexts, check and report inconsistency, illness, idleness, etc., close bad contexts if necessary. -
Methods inherited from class ai.platon.pulsar.protocol.browser.emulator.context.AbstractBrowserPrivacyManager
getBrowserFactory, getBrowserManager, getDriverPoolManager, getProxyPoolManager -
Methods inherited from class ai.platon.pulsar.skeleton.crawl.fetch.privacy.PrivacyManager
buildStatusString, close, createUnmanagedContext, getConf, isActive, isClosed, reset -
Methods inherited from class ai.platon.pulsar.skeleton.crawl.fetch.privacy.AbstractPrivacyManager
getActiveContexts, getContextLifeCycleMonitor, getDeadContexts, getPermanentContexts, getPrivacyAgentGenerator, getTemporaryContexts, getZombieContexts -
Methods inherited from class java.lang.AutoCloseable
close -
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
-
-
Constructor Detail
-
MultiPrivacyContextManager
MultiPrivacyContextManager(WebDriverPoolManager driverPoolManager)
-
MultiPrivacyContextManager
MultiPrivacyContextManager(WebDriverPoolManager driverPoolManager, ImmutableConfig conf)
-
MultiPrivacyContextManager
MultiPrivacyContextManager(ImmutableConfig conf)
-
MultiPrivacyContextManager
MultiPrivacyContextManager(WebDriverPoolManager driverPoolManager, ProxyPoolManager proxyPoolManager, CoreMetrics coreMetrics, ImmutableConfig conf)
-
-
Method Detail
-
getMaxAllowedBadContexts
final Integer getMaxAllowedBadContexts()
-
getSnapshotDumpInterval
final Duration getSnapshotDumpInterval()
-
setSnapshotDumpInterval
final Unit setSnapshotDumpInterval(Duration snapshotDumpInterval)
-
getMetrics
final MultiPrivacyContextManager.Metrics getMetrics()
-
getCoreMetrics
final CoreMetrics getCoreMetrics()
-
run
FetchResult run(FetchTask task, SuspendFunction2<FetchTask, WebDriver, FetchResult> fetchFun)
Run a task in a privacy context.
The privacy context is selected from the active privacy context pool, and it is supposed to have at least one ready web driver to run the task.
If the privacy context chosen is not ready to serve, especially, it has no any ready web driver, the task will be canceled.
- Parameters:
task- the fetch taskfetchFun- the fetch function- Returns:
the fetch result
-
createUnmanagedContext
BrowserPrivacyContext createUnmanagedContext(BrowserProfile profile)
Create a privacy context who is not added to the context list yet.
-
tryGetNextReadyPrivacyContext
PrivacyContext tryGetNextReadyPrivacyContext(WebPage page, Fingerprint fingerprint, FetchTask task)
Try to get a ready privacy context.
The status of the privacy context can change immediately, so this method may return a non-ready privacy context.
If the total number of active contexts is less than the maximum number allowed, a new privacy context will be created.
If the privacy context is inactive, close it and create a new one immediately, and returns a new one.
This method can return a non-ready privacy context, in which case the task will be canceled.
A ready privacy context has to meet the following requirements:
not closed
not leaked
requirement removed not idle
not retired
if there is a proxy, the proxy has to be ready
the associated driver pool promises to provide an available driver, ether one of the following:
- Parameters:
fingerprint- The fingerprint of this privacy context.- Returns:
A privacy context which is promised to be ready.
-
tryGetNextReadyPrivacyContext
PrivacyContext tryGetNextReadyPrivacyContext(Fingerprint fingerprint)
-
tryGetNextUnderLoadedPrivacyContext
PrivacyContext tryGetNextUnderLoadedPrivacyContext(WebPage page, Fingerprint fingerprint, FetchTask task)
Gets an under-loaded privacy context, which can be either ready or not ready.
If the total number of active contexts is less than the maximum number allowed, a new privacy context will be created.
This method can return an inactive privacy context, in which case, the task should be canceled, and the privacy context should be closed.
- Parameters:
fingerprint- The fingerprint of this privacy context.- Returns:
A privacy context which is promised to be ready.
-
getOrCreate
PrivacyContext getOrCreate(BrowserProfile profile)
-
maintain
Unit maintain(Boolean force)
Maintain all the privacy contexts, check and report inconsistency, illness, idleness, etc., close bad contexts if necessary.
If "takePrivacyContextSnapshot" is in file AppPaths.PATH_LOCAL_COMMAND, perform the action.
If the tmp dir is the default one, run the following command to take snapshot once: echo takePrivacyContextSnapshot /tmp/pulsar/pulsar-commands
-
-
-
-