All Implemented Interfaces:

ai.platon.pulsar.common.event.EventEmitter , ai.platon.pulsar.crawl.Crawler , java.lang.AutoCloseable
```
public class StreamingCrawler
extends AbstractCrawler
                    
```

Nested Class Summary

Nested Classes
Modifier and Type	Class	Description
`public class`	`StreamingCrawler.Companion`

Field Summary

Fields
Modifier and Type	Field	Description
`private final Integer`	`numPrivacyContexts`
`private final Integer`	`numMaxActiveTabs`
`private final Integer`	`fetchConcurrency`
`private final Duration`	`outOfWorkTimeout`
`private final Duration`	`fetchTaskTimeout`
`private final Duration`	`idleTime`
`private final Boolean`	`isOutOfWork`
`private final Boolean`	`isActive`
`private String`	`jobName`
`private Sequence<UrlAware>`	`urls`
`private final Boolean`	`noProxy`
`private final Integer`	`id`
`private final String`	`name`
`private Function2<Integer, UrlAware, Duration>`	`retryDelayPolicy`
`private final AtomicBoolean`	`closed`
`private final PulsarSession`	`session`
`private final Boolean`	`autoClose`
`private final Map<CrawlEvents, List<Function<Object>>>`	`listeners`

Constructor Summary

Constructors
Constructor	Description
`StreamingCrawler(Sequence<UrlAware> urls, PulsarSession session, Boolean noProxy, Boolean autoClose)`

Method Summary

Modifier and Type	Method	Description
`final Integer`	`getNumPrivacyContexts()`
`final Integer`	`getNumMaxActiveTabs()`
`final Integer`	`getFetchConcurrency()`
`final Duration`	`getOutOfWorkTimeout()`
`final Duration`	`getFetchTaskTimeout()`
`final Duration`	`getIdleTime()`
`final Boolean`	`getIsOutOfWork()`
`Boolean`	`getIsActive()`
`final String`	`getJobName()`
`final Unit`	`setJobName(String jobName)`
`final Sequence<UrlAware>`	`getUrls()`	The url sequence
`final Unit`	`setUrls(Sequence<UrlAware> urls)`	The url sequence
`final Boolean`	`getNoProxy()`	Do not use proxy
`Integer`	`getId()`	The crawl id
`String`	`getName()`
`Function2<Integer, UrlAware, Duration>`	`getRetryDelayPolicy()`	Delay policy for retry tasks
`Unit`	`setRetryDelayPolicy(Function2<Integer, UrlAware, Duration> retryDelayPolicy)`	Delay policy for retry tasks
`final AtomicBoolean`	`getClosed()`
`final PulsarSession`	`getSession()`
`final Boolean`	`getAutoClose()`
`final Map<CrawlEvents, List<Function<Object>>>`	`getListeners()`
`Unit`	`run()`
`Unit`	`run(CoroutineScope scope)`
`Unit`	`await()`	Await for all tasks be done
`final Unit`	`quit()`
`Unit`	`close()`

Methods inherited from class ai.platon.pulsar.crawl.impl.StreamingCrawler
onLoad, onLoaded, onWillLoad
Methods inherited from class ai.platon.pulsar.crawl.impl.AbstractCrawler
count, emit, emit, emit, emit, emit1, emit1, emit1, emit1, hasListeners, listeners, listeners, off, off, off, off, off, off1, off1, off1, off1, off1, on, on, on, on, on1, on1, on1, on1, once, once, once, once1, once1, once1
Methods inherited from class ai.platon.pulsar.common.event.AbstractEventEmitter
hasNormalListeners, hasSuspendListeners, normalListeners, normalListeners, suspendListeners, suspendListeners
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

StreamingCrawler

StreamingCrawler(Sequence<UrlAware> urls, PulsarSession session, Boolean noProxy, Boolean autoClose)

Method Detail

getNumPrivacyContexts
```
 final Integer getNumPrivacyContexts()
```

getNumMaxActiveTabs
```
 final Integer getNumMaxActiveTabs()
```

getFetchConcurrency
```
 final Integer getFetchConcurrency()
```

getOutOfWorkTimeout
```
 final Duration getOutOfWorkTimeout()
```

getFetchTaskTimeout
```
 final Duration getFetchTaskTimeout()
```

getIdleTime
```
 final Duration getIdleTime()
```

getIsOutOfWork
```
 final Boolean getIsOutOfWork()
```

getIsActive
```
 Boolean getIsActive()
```

getJobName
```
 final String getJobName()
```

setJobName
```
 final Unit setJobName(String jobName)
```

getUrls
```
 final Sequence<UrlAware> getUrls()
```
The url sequence

setUrls

 final Unit setUrls(Sequence<UrlAware> urls)

The url sequence

getNoProxy
```
 final Boolean getNoProxy()
```
Do not use proxy

getId
```
 Integer getId()
```
The crawl id

getName
```
 String getName()
```

getRetryDelayPolicy

 Function2<Integer, UrlAware, Duration> getRetryDelayPolicy()

Delay policy for retry tasks

setRetryDelayPolicy

 Unit setRetryDelayPolicy(Function2<Integer, UrlAware, Duration> retryDelayPolicy)

Delay policy for retry tasks

getClosed
```
 final AtomicBoolean getClosed()
```

getSession
```
 final PulsarSession getSession()
```

getAutoClose
```
 final Boolean getAutoClose()
```

getListeners

 final Map<CrawlEvents, List<Function<Object>>> getListeners()

run
```
 Unit run()
```

run
```
 Unit run(CoroutineScope scope)
```

await
```
 Unit await()
```
Await for all tasks be done

quit
```
 final Unit quit()
```

close
```
 Unit close()
```

Class StreamingCrawler

Nested Class Summary

Field Summary

Constructor Summary

Method Summary

Methods inherited from class ai.platon.pulsar.crawl.impl.StreamingCrawler

Methods inherited from class ai.platon.pulsar.crawl.impl.AbstractCrawler

Methods inherited from class ai.platon.pulsar.common.event.AbstractEventEmitter

Methods inherited from class java.lang.Object

Constructor Detail

StreamingCrawler

Method Detail

getNumPrivacyContexts

getNumMaxActiveTabs

getFetchConcurrency

getOutOfWorkTimeout

getFetchTaskTimeout

getIdleTime

getIsOutOfWork

getIsActive

getJobName

setJobName

getUrls

setUrls

getNoProxy

getId

getName

getRetryDelayPolicy

setRetryDelayPolicy

getClosed

getSession

getAutoClose

getListeners

run

run

await

quit

close