All Implemented Interfaces:

ai.platon.pulsar.common.event.EventEmitter , ai.platon.pulsar.skeleton.crawl.Crawler , java.lang.AutoCloseable
```
public class StreamingCrawler
extends AbstractCrawler
                    
```

Nested Class Summary

Nested Classes
Modifier and Type	Class	Description
`public class`	`StreamingCrawler.Companion`

Field Summary

Fields
Modifier and Type	Field	Description
`private final Integer`	`numBrowserContexts`
`private final Integer`	`numPrivacyContexts`
`private final Integer`	`numMaxOpenTabs`
`private final Integer`	`fetchConcurrency`
`private final Integer`	`concurrency`
`private final Duration`	`outOfWorkTimeout`
`private final Duration`	`fetchTaskTimeout`
`private final Duration`	`idleTime`
`private final Boolean`	`isOutOfWork`
`private final Boolean`	`isSmartRetryEnabled`
`private final Boolean`	`isIdle`
`private final Boolean`	`isActive`
`private String`	`jobName`
`private Sequence<UrlAware>`	`urls`
`private final Integer`	`id`
`private final String`	`name`
`private Function2<Integer, UrlAware, Duration>`	`retryDelayPolicy`
`private final PulsarSession`	`session`
`private final Boolean`	`autoClose`
`private Function1<Throwable, Unit>`	`eventExceptionHandler`
`private final Map<CrawlEvents, List<Function<Object>>>`	`listeners`
`public final static StreamingCrawler.Companion`	`Companion`

Constructor Summary

Constructors
Constructor	Description
`StreamingCrawler(Sequence<UrlAware> urls, PulsarSession session, Boolean autoClose)`

Enum Constant Summary

Enum Constants
Enum Constant	Description

Method Summary

Modifier and Type	Method	Description
`final Integer`	`getNumBrowserContexts()`
`final Integer`	`getNumPrivacyContexts()`
`final Integer`	`getNumMaxOpenTabs()`
`final Integer`	`getFetchConcurrency()`
`final Integer`	`getConcurrency()`
`final Duration`	`getOutOfWorkTimeout()`	The out of work timeout.
`final Duration`	`getFetchTaskTimeout()`
`final Duration`	`getIdleTime()`
`final Boolean`	`isOutOfWork()`
`final Boolean`	`isSmartRetryEnabled()`
`final Boolean`	`isIdle()`
`Boolean`	`isActive()`
`final String`	`getJobName()`	The job name.
`final Unit`	`setJobName(String jobName)`	The job name.
`final Sequence<UrlAware>`	`getUrls()`	The url sequence
`final Unit`	`setUrls(Sequence<UrlAware> urls)`	The url sequence
`Unit`	`run()`	Run the crawler.
`Unit`	`run(CoroutineScope scope)`	Run the crawler in the given coroutine scope.
`Unit`	`report()`
`Unit`	`await()`	Wait until all tasks are done.
`final Unit`	`quit()`	Quit the crawl loop.
`Unit`	`close()`	Quit and close the crawl loop.

Methods inherited from class ai.platon.pulsar.skeleton.crawl.impl.AbstractCrawler
getAutoClose, getId, getName, getRetryDelayPolicy, getSession, onLoad, onLoaded, onWillLoad, pause, resume, setRetryDelayPolicy
Methods inherited from class ai.platon.pulsar.common.event.AbstractEventEmitter
count, emit, emit, emit, emit, emit1, emit1, emit1, emit1, getEventExceptionHandler, getListeners, hasListeners, listeners, listeners, off, off, off, off, off, off1, off1, off1, off1, off1, on, on, on, on, on1, on1, on1, on1, once, once, once, once1, once1, once1, setEventExceptionHandler
Methods inherited from class ai.platon.pulsar.common.event.EventEmitter
hasNormalListeners, hasSuspendListeners, normalListeners, normalListeners, suspendListeners, suspendListeners
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

StreamingCrawler

StreamingCrawler(Sequence<UrlAware> urls, PulsarSession session, Boolean autoClose)

Method Detail

getNumBrowserContexts
```
 final Integer getNumBrowserContexts()
```

getNumPrivacyContexts
```
 final Integer getNumPrivacyContexts()
```

getNumMaxOpenTabs
```
 final Integer getNumMaxOpenTabs()
```

getFetchConcurrency
```
 final Integer getFetchConcurrency()
```

getConcurrency
```
 final Integer getConcurrency()
```

getOutOfWorkTimeout
```
 final Duration getOutOfWorkTimeout()
```
The out of work timeout.

getFetchTaskTimeout
```
 final Duration getFetchTaskTimeout()
```

getIdleTime
```
 final Duration getIdleTime()
```

isOutOfWork
```
 final Boolean isOutOfWork()
```

isSmartRetryEnabled
```
 final Boolean isSmartRetryEnabled()
```

isIdle
```
 final Boolean isIdle()
```

isActive
```
 Boolean isActive()
```

getJobName
```
 final String getJobName()
```
The job name.

setJobName
```
 final Unit setJobName(String jobName)
```
The job name.

getUrls
```
 final Sequence<UrlAware> getUrls()
```
The url sequence

setUrls

 final Unit setUrls(Sequence<UrlAware> urls)

The url sequence

run
```
 Unit run()
```
Run the crawler.

run
```
 Unit run(CoroutineScope scope)
```
Run the crawler in the given coroutine scope.

report
```
 Unit report()
```

await
```
 Unit await()
```
Wait until all tasks are done.

quit
```
 final Unit quit()
```
Quit the crawl loop.

close
```
 Unit close()
```
Quit and close the crawl loop.

Class StreamingCrawler

Nested Class Summary

Field Summary

Constructor Summary

Enum Constant Summary

Method Summary

Methods inherited from class ai.platon.pulsar.skeleton.crawl.impl.AbstractCrawler

Methods inherited from class ai.platon.pulsar.common.event.AbstractEventEmitter

Methods inherited from class ai.platon.pulsar.common.event.EventEmitter

Methods inherited from class java.lang.Object

Constructor Detail

StreamingCrawler

Method Detail

getNumBrowserContexts

getNumPrivacyContexts

getNumMaxOpenTabs

getFetchConcurrency

getConcurrency

getOutOfWorkTimeout

getFetchTaskTimeout

getIdleTime

isOutOfWork

isSmartRetryEnabled

isIdle

isActive

getJobName

setJobName

getUrls

setUrls

run

run

report

await

quit

close