Class AbstractPrivacyContext
-
- All Implemented Interfaces:
-
ai.platon.pulsar.skeleton.crawl.fetch.privacy.PrivacyContext,java.lang.AutoCloseable,kotlin.Comparable
public abstract class AbstractPrivacyContext implements PrivacyContext, Comparable<PrivacyContext>
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description public classAbstractPrivacyContext.Companion
-
Field Summary
-
Constructor Summary
Constructors Constructor Description AbstractPrivacyContext(PrivacyAgent privacyAgent, ImmutableConfig conf)
-
Method Summary
Modifier and Type Method Description final WebDriverFetchergetWebdriverFetcher()final UnitsetWebdriverFetcher(WebDriverFetcher webdriverFetcher)PrivacyAgentIdgetId()final IntegergetSeq()StringgetDisplay()final PathgetBaseDir()final FloatgetMinimumThroughput()final IntegergetMaximumWarnings()final IntegergetMinorWarningFactor()final AtomicIntegergetPrivacyLeakWarnings()final AtomicIntegergetPrivacyLeakMinorWarnings()final MetergetMeterTasks()final MetergetMeterSuccesses()final MetergetMeterFinishes()final MetergetMeterSmallPages()final DoublegetSmallPageRate()final FloatgetSuccessRate()FloatgetFailureRate()final FloatgetFailureRateThreshold()BooleanisHighFailureRate()final InstantgetStartTime()The start time of the privacy context. final InstantgetLastActiveTime()The last active time of the privacy context. DurationgetElapsedTime()DurationgetIdleTime()BooleanisIdle()BooleanisGood()BooleanisLeaked()BooleanisRetired()BooleanisActive()BooleanisClosed()BooleanisReady()BooleanisFullCapacity()Check whether the privacy context is at full capacity. BooleanisUnderLoaded()StringgetReadableState()PrivacyAgentgetPrivacyAgent()final ImmutableConfiggetConf()abstract IntegerpromisedWebDriverCount()The promised worker count. BooleanhasWebDriverPromise()Check if the privacy context promises at least one worker to provide. abstract WebDriversubscribeWebDriver()final UnitmarkSuccess()Mark a success task. final UnitmarkWarning()Mark a warning. final UnitmarkWarning(Integer n)Mark n warnings. final UnitmarkMinorWarning()Mark a minor warnings. final UnitmarkLeaked()Mark the privacy context as leaked. abstract FetchResultopen(String url)Open an url in the privacy context. abstract FetchResultopen(String url, LoadOptions options)Open an url in the privacy context, with the specified options. FetchResultopen(String url, SuspendFunction2<FetchTask, WebDriver, FetchResult> fetchFun)Open an url in the privacy context, with the specified fetch function. FetchResultrun(FetchTask task, SuspendFunction2<FetchTask, WebDriver, FetchResult> fetchFun)Run a task in the privacy context, with the specified fetch function. abstract FetchResultdoRun(FetchTask task, SuspendFunction2<FetchTask, WebDriver, FetchResult> fetchFun)Run a task in the privacy context. StringbuildStatusString()Unitdismiss()Dismiss the privacy context and mark it as be retired, so it should be closed later. abstract Unitmaintain()Do the maintaining jobs. IntegercompareTo(PrivacyContext other)Booleanequals(Object other)IntegerhashCode()StringbuildReport()Unitreport()-
-
Constructor Detail
-
AbstractPrivacyContext
AbstractPrivacyContext(PrivacyAgent privacyAgent, ImmutableConfig conf)
-
-
Method Detail
-
getWebdriverFetcher
final WebDriverFetcher getWebdriverFetcher()
-
setWebdriverFetcher
final Unit setWebdriverFetcher(WebDriverFetcher webdriverFetcher)
-
getId
PrivacyAgentId getId()
-
getDisplay
String getDisplay()
-
getBaseDir
final Path getBaseDir()
-
getMinimumThroughput
final Float getMinimumThroughput()
-
getMaximumWarnings
final Integer getMaximumWarnings()
-
getMinorWarningFactor
final Integer getMinorWarningFactor()
-
getPrivacyLeakWarnings
final AtomicInteger getPrivacyLeakWarnings()
-
getPrivacyLeakMinorWarnings
final AtomicInteger getPrivacyLeakMinorWarnings()
-
getMeterTasks
final Meter getMeterTasks()
-
getMeterSuccesses
final Meter getMeterSuccesses()
-
getMeterFinishes
final Meter getMeterFinishes()
-
getMeterSmallPages
final Meter getMeterSmallPages()
-
getSmallPageRate
final Double getSmallPageRate()
-
getSuccessRate
final Float getSuccessRate()
-
getFailureRate
Float getFailureRate()
-
getFailureRateThreshold
final Float getFailureRateThreshold()
-
isHighFailureRate
Boolean isHighFailureRate()
-
getStartTime
final Instant getStartTime()
The start time of the privacy context.
-
getLastActiveTime
final Instant getLastActiveTime()
The last active time of the privacy context.
-
getElapsedTime
Duration getElapsedTime()
-
getIdleTime
Duration getIdleTime()
-
isFullCapacity
Boolean isFullCapacity()
Check whether the privacy context is at full capacity. If the privacy context is indeed at full capacity, it should not be used for processing new tasks, and the underlying services may potentially refuse to provide service.
A privacy context is running at full capacity when the underlying webdriver pool is full capacity, so the webdriver pool can not provide a webdriver for new tasks.
Note that if a driver pool is retired or closed, it's not full capacity.
- Returns:
True if the privacy context is running at full capacity, false otherwise.
-
isUnderLoaded
Boolean isUnderLoaded()
-
getReadableState
String getReadableState()
-
getPrivacyAgent
PrivacyAgent getPrivacyAgent()
-
getConf
final ImmutableConfig getConf()
-
promisedWebDriverCount
abstract Integer promisedWebDriverCount()
The promised worker count.
The implementation has to tell the caller how many workers it can provide. The number of workers can change immediately after reading, so the caller only has promises but no guarantees.
- Returns:
the number of workers promised.
-
hasWebDriverPromise
Boolean hasWebDriverPromise()
Check if the privacy context promises at least one worker to provide.
-
subscribeWebDriver
@Beta() abstract WebDriver subscribeWebDriver()
-
markSuccess
final Unit markSuccess()
Mark a success task.
-
markWarning
final Unit markWarning()
Mark a warning.
-
markWarning
final Unit markWarning(Integer n)
Mark n warnings.
-
markMinorWarning
final Unit markMinorWarning()
Mark a minor warnings.
-
markLeaked
final Unit markLeaked()
Mark the privacy context as leaked. A leaked privacy context should not serve anymore, and will be closed soon.
-
open
abstract FetchResult open(String url)
Open an url in the privacy context.
-
open
abstract FetchResult open(String url, LoadOptions options)
Open an url in the privacy context, with the specified options.
-
open
FetchResult open(String url, SuspendFunction2<FetchTask, WebDriver, FetchResult> fetchFun)
Open an url in the privacy context, with the specified fetch function.
-
run
FetchResult run(FetchTask task, SuspendFunction2<FetchTask, WebDriver, FetchResult> fetchFun)
Run a task in the privacy context, with the specified fetch function.
- Parameters:
task- the fetch taskfetchFun- the fetch function- Returns:
the fetch result
-
doRun
abstract FetchResult doRun(FetchTask task, SuspendFunction2<FetchTask, WebDriver, FetchResult> fetchFun)
Run a task in the privacy context.
- Parameters:
task- the fetch taskfetchFun- the fetch function- Returns:
the fetch result
-
buildStatusString
String buildStatusString()
-
dismiss
Unit dismiss()
Dismiss the privacy context and mark it as be retired, so it should be closed later.
-
compareTo
Integer compareTo(PrivacyContext other)
-
buildReport
String buildReport()
-
-
-
-