Class PrivacyContext
-
- All Implemented Interfaces:
-
java.lang.AutoCloseable,kotlin.Comparable
public abstract class PrivacyContext implements Comparable<PrivacyContext>, AutoCloseable
A privacy context is a unique context of a privacy agent to the target website, it will be closed once it is leaked.
One of the biggest difficulties in web scraping tasks is the bot stealth.
For web scraping tasks, the website should have no idea whether a visit is from a human being or a bot. Once a page visit is suspected by the website, which we call a privacy leak, the privacy context has to be dropped, and Pulsar will visit the page in another privacy context.
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description public classPrivacyContext.Companion
-
Field Summary
Fields Modifier and Type Field Description public final static StringCONTEXT_DIR_PREFIXprivate final PrivacyAgentIdidprivate final Integerseqprivate final Stringdisplayprivate final PathbaseDirprivate final FloatminimumThroughputprivate final IntegermaximumWarningsprivate final IntegerminorWarningFactorprivate final AtomicIntegerprivacyLeakWarningsprivate final AtomicIntegerprivacyLeakMinorWarningsprivate final MetermeterTasksprivate final MetermeterSuccessesprivate final MetermeterFinishesprivate final MetermeterSmallPagesprivate final DoublesmallPageRateprivate final FloatsuccessRateprivate final FloatfailureRateprivate final FloatfailureRateThresholdprivate final BooleanisHighFailureRateprivate final InstantstartTimeprivate final InstantlastActiveTimeprivate final DurationelapsedTimeprivate final DurationidelTimeprivate final BooleanisIdleprivate final BooleanisGoodprivate final BooleanisLeakedprivate final BooleanisRetiredprivate final BooleanisActiveprivate final BooleanisClosedprivate final BooleanisReadyprivate final BooleanisFullCapacityprivate final BooleanisUnderLoadedprivate final StringreadableStateprivate final PrivacyAgentprivacyAgentprivate final ImmutableConfigconfpublic final static PrivacyContext.CompanionCompanion
-
Constructor Summary
Constructors Constructor Description PrivacyContext(PrivacyAgent privacyAgent, ImmutableConfig conf)
-
Method Summary
Modifier and Type Method Description final PrivacyAgentIdgetId()final IntegergetSeq()final StringgetDisplay()final PathgetBaseDir()final FloatgetMinimumThroughput()final IntegergetMaximumWarnings()final IntegergetMinorWarningFactor()final AtomicIntegergetPrivacyLeakWarnings()final AtomicIntegergetPrivacyLeakMinorWarnings()final MetergetMeterTasks()final MetergetMeterSuccesses()final MetergetMeterFinishes()final MetergetMeterSmallPages()final DoublegetSmallPageRate()final FloatgetSuccessRate()final FloatgetFailureRate()final FloatgetFailureRateThreshold()final BooleanisHighFailureRate()final InstantgetStartTime()The start time of the privacy context. final InstantgetLastActiveTime()The last active time of the privacy context. final DurationgetElapsedTime()final DurationgetIdelTime()BooleanisIdle()BooleanisGood()BooleanisLeaked()BooleanisRetired()BooleanisActive()BooleanisClosed()BooleanisReady()BooleanisFullCapacity()Check whether the privacy context is at full capacity. BooleanisUnderLoaded()StringgetReadableState()final PrivacyAgentgetPrivacyAgent()final ImmutableConfiggetConf()abstract IntegerpromisedWebDriverCount()The promised worker count. final BooleanhasWebDriverPromise()Check if the privacy context promises at least one worker to provide. abstract WebDriversubscribeWebDriver()final UnitmarkSuccess()Mark a success task. final UnitmarkWarning()Mark a warning. final UnitmarkWarning(Integer n)Mark n warnings. final UnitmarkMinorWarning()Mark a minor warnings. final UnitmarkLeaked()Mark the privacy context as leaked. FetchResultrun(FetchTask task, SuspendFunction2<FetchTask, WebDriver, FetchResult> fetchFun)Run a task in the privacy context and record the status. abstract FetchResultdoRun(FetchTask task, SuspendFunction2<FetchTask, WebDriver, FetchResult> fetchFun)Run a task in the privacy context. final StringtakeSnapshot()final Unitdismiss()Dismiss the privacy context and mark it as be retired, so it should be closed later. abstract Unitmaintain()Do the maintaining jobs. IntegercompareTo(PrivacyContext other)Booleanequals(Object other)IntegerhashCode()StringgetReport()Unitreport()-
-
Constructor Detail
-
PrivacyContext
PrivacyContext(PrivacyAgent privacyAgent, ImmutableConfig conf)
-
-
Method Detail
-
getId
final PrivacyAgentId getId()
-
getDisplay
final String getDisplay()
-
getBaseDir
final Path getBaseDir()
-
getMinimumThroughput
final Float getMinimumThroughput()
-
getMaximumWarnings
final Integer getMaximumWarnings()
-
getMinorWarningFactor
final Integer getMinorWarningFactor()
-
getPrivacyLeakWarnings
final AtomicInteger getPrivacyLeakWarnings()
-
getPrivacyLeakMinorWarnings
final AtomicInteger getPrivacyLeakMinorWarnings()
-
getMeterTasks
final Meter getMeterTasks()
-
getMeterSuccesses
final Meter getMeterSuccesses()
-
getMeterFinishes
final Meter getMeterFinishes()
-
getMeterSmallPages
final Meter getMeterSmallPages()
-
getSmallPageRate
final Double getSmallPageRate()
-
getSuccessRate
final Float getSuccessRate()
-
getFailureRate
final Float getFailureRate()
-
getFailureRateThreshold
final Float getFailureRateThreshold()
-
isHighFailureRate
final Boolean isHighFailureRate()
-
getStartTime
final Instant getStartTime()
The start time of the privacy context.
-
getLastActiveTime
final Instant getLastActiveTime()
The last active time of the privacy context.
-
getElapsedTime
final Duration getElapsedTime()
-
getIdelTime
final Duration getIdelTime()
-
isFullCapacity
Boolean isFullCapacity()
Check whether the privacy context is at full capacity. If the privacy context is indeed at full capacity, it should not be used for processing new tasks, and the underlying services may potentially refuse to provide service.
A privacy context is running at full capacity when the underlying webdriver pool is full capacity, so the webdriver pool can not provide a webdriver for new tasks.
Note that if a driver pool is retired or closed, it's not full capacity.
- Returns:
True if the privacy context is running at full capacity, false otherwise.
-
isUnderLoaded
Boolean isUnderLoaded()
-
getReadableState
String getReadableState()
-
getPrivacyAgent
final PrivacyAgent getPrivacyAgent()
-
getConf
final ImmutableConfig getConf()
-
promisedWebDriverCount
abstract Integer promisedWebDriverCount()
The promised worker count.
The implementation has to tell the caller how many workers it can provide. The number of workers can change immediately after reading, so the caller only has promises but no guarantees.
- Returns:
the number of workers promised.
-
hasWebDriverPromise
final Boolean hasWebDriverPromise()
Check if the privacy context promises at least one worker to provide.
-
subscribeWebDriver
@Beta() abstract WebDriver subscribeWebDriver()
-
markSuccess
final Unit markSuccess()
Mark a success task.
-
markWarning
final Unit markWarning()
Mark a warning.
-
markWarning
final Unit markWarning(Integer n)
Mark n warnings.
-
markMinorWarning
final Unit markMinorWarning()
Mark a minor warnings.
-
markLeaked
final Unit markLeaked()
Mark the privacy context as leaked. A leaked privacy context should not serve anymore, and will be closed soon.
-
run
FetchResult run(FetchTask task, SuspendFunction2<FetchTask, WebDriver, FetchResult> fetchFun)
Run a task in the privacy context and record the status.
- Parameters:
task- the fetch taskfetchFun- the fetch function- Returns:
the fetch result
-
doRun
abstract FetchResult doRun(FetchTask task, SuspendFunction2<FetchTask, WebDriver, FetchResult> fetchFun)
Run a task in the privacy context.
- Parameters:
task- the fetch taskfetchFun- the fetch function- Returns:
the fetch result
-
takeSnapshot
final String takeSnapshot()
-
dismiss
final Unit dismiss()
Dismiss the privacy context and mark it as be retired, so it should be closed later.
-
compareTo
Integer compareTo(PrivacyContext other)
-
-
-
-