Class PrivacyContext
-
- All Implemented Interfaces:
-
java.lang.AutoCloseable,kotlin.Comparable
public abstract class PrivacyContext implements Comparable<PrivacyContext>, AutoCloseable
A privacy context is a unique context of a privacy agent to the target website, it will be closed once it is leaked.
One of the biggest difficulties in web scraping tasks is the bot stealth.
For web scraping tasks, the website should have no idea whether a visit is from a human being or a bot. Once a page visit is suspected by the website, which we call a privacy leak, the privacy context has to be dropped, and Pulsar will visit the page in another privacy context.
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description public classPrivacyContext.Companion
-
Field Summary
-
Constructor Summary
Constructors Constructor Description PrivacyContext(PrivacyAgent id, ImmutableConfig conf)
-
Method Summary
-
-
Constructor Detail
-
PrivacyContext
PrivacyContext(PrivacyAgent id, ImmutableConfig conf)
-
-
Method Detail
-
getSequence
final Integer getSequence()
-
getPrivacyAgent
final PrivacyAgent getPrivacyAgent()
-
getId0
final PrivacyAgentId getId0()
-
getDisplay
final String getDisplay()
-
getBaseDir
final Path getBaseDir()
-
getMinimumThroughput
final Float getMinimumThroughput()
-
getMaximumWarnings
final Integer getMaximumWarnings()
-
getMinorWarningFactor
final Integer getMinorWarningFactor()
-
getPrivacyLeakWarnings
final AtomicInteger getPrivacyLeakWarnings()
-
getPrivacyLeakMinorWarnings
final AtomicInteger getPrivacyLeakMinorWarnings()
-
getMeterTasks
final Meter getMeterTasks()
-
getMeterSuccesses
final Meter getMeterSuccesses()
-
getMeterFinishes
final Meter getMeterFinishes()
-
getMeterSmallPages
final Meter getMeterSmallPages()
-
getSmallPageRate
final Double getSmallPageRate()
-
getSuccessRate
final Float getSuccessRate()
-
getFailureRate
final Float getFailureRate()
-
getFailureRateThreshold
final Float getFailureRateThreshold()
-
isHighFailureRate
final Boolean isHighFailureRate()
-
getStartTime
final Instant getStartTime()
-
getLastActiveTime
final Instant getLastActiveTime()
-
setLastActiveTime
final Unit setLastActiveTime(Instant lastActiveTime)
-
getElapsedTime
final Duration getElapsedTime()
-
getIdelTime
final Duration getIdelTime()
-
isFullCapacity
Boolean isFullCapacity()
Check if the privacy context is full capacity. If the privacy context is full capacity, it should not be used for new tasks, the underlying layer might refuse to serve.
A privacy context is running at full load when the underlying webdriver pool is full capacity, so the webdriver pool can not provide a webdriver for new tasks.
Note that if a driver pool is retired or closed, it's not full capacity.
-
isUnderLoaded
Boolean isUnderLoaded()
-
getReadableState
String getReadableState()
-
getId
final PrivacyAgent getId()
-
getConf
final ImmutableConfig getConf()
-
promisedWebDriverCount
abstract Integer promisedWebDriverCount()
The promised worker count.
The implementation has to tell the caller how many workers it can provide. The number of workers can change immediately after reading, so the caller only has promises but no guarantees.
-
hasWebDriverPromise
final Boolean hasWebDriverPromise()
Check if the privacy context promises at least one worker to provide.
-
subscribeWebDriver
@Beta() abstract WebDriver subscribeWebDriver()
-
markSuccess
final Unit markSuccess()
Mark a success task.
-
markWarning
final Unit markWarning()
Mark a warning.
-
markWarning
final Unit markWarning(Integer n)
Mark n warnings.
-
markMinorWarning
final Unit markMinorWarning()
Mark a minor warnings.
-
markLeaked
final Integer markLeaked()
Mark the privacy context as leaked. A leaked privacy context should not serve anymore, and will be closed soon.
-
run
FetchResult run(FetchTask task, SuspendFunction2<FetchTask, WebDriver, FetchResult> fetchFun)
Run a task in the privacy context and record the status.
- Parameters:
task- the fetch taskfetchFun- the fetch function
-
doRun
abstract FetchResult doRun(FetchTask task, SuspendFunction2<FetchTask, WebDriver, FetchResult> fetchFun)
Run a task in the privacy context.
- Parameters:
task- the fetch taskfetchFun- the fetch function
-
takeSnapshot
final String takeSnapshot()
-
dismiss
final Unit dismiss()
Dismiss the privacy context and mark it as be retired, so it should be closed later.
-
compareTo
Integer compareTo(PrivacyContext other)
-
-
-
-