All Implemented Interfaces:

ai.platon.pulsar.common.collect.CrawlableFatLinkCollector , ai.platon.pulsar.common.collect.collector.DataCollector , ai.platon.pulsar.common.collect.collector.PriorityDataCollector , kotlin.Comparable
```
public class HyperlinkCollector
extends AbstractPriorityDataCollector<UrlAware> implements CrawlableFatLinkCollector
                    
```
Collect hyper links from the given seeds. The urls are restricted by loadArguments and urlNormalizer.
- all urls are restricted by css outLinkSelector
- all urls are restricted by urlPattern
- all urls have to not be fetched before or expired against the last version

Nested Class Summary

Nested Classes
Modifier and Type	Class	Description

Field Summary

Fields
Modifier and Type	Field	Description
`private UrlNormalizerPipeline`	`urlNormalizer`
`private String`	`name`
`private final Integer`	`size`
`private final Integer`	`estimatedSize`
`private final ConcurrentSkipListMap<String, CrawlableFatLink>`	`fatLinks`
`private final PulsarSession`	`session`
`private final Queue<NormUrl>`	`seeds`
`private final Integer`	`capacity`
`private Integer`	`collectCount`
`private final Duration`	`collectTime`
`private Integer`	`collectedCount`
`private String`	`country`
`private final Instant`	`createTime`
`private Instant`	`deadTime`
`private String`	`district`
`private final Integer`	`estimatedExternalSize`
`private final Integer`	`externalSize`
`private Instant`	`firstCollectTime`
`private final Integer`	`id`
`private final Boolean`	`isDead`
`private final Set<String>`	`labels`
`private String`	`lang`
`private Instant`	`lastCollectedTime`
`private final Integer`	`priority`

Constructor Summary

Constructors
Constructor	Description
`HyperlinkCollector(PulsarSession session, Queue<NormUrl> seeds, Priority13 priority)`

Enum Constant Summary

Enum Constants
Enum Constant	Description

Method Summary

Modifier and Type	Method	Description
`final UrlNormalizerPipeline`	`getUrlNormalizer()`
`final Unit`	`setUrlNormalizer(UrlNormalizerPipeline urlNormalizer)`
`String`	`getName()`
`Unit`	`setName(String name)`
`Integer`	`getSize()`
`Integer`	`getEstimatedSize()`
`ConcurrentSkipListMap<String, CrawlableFatLink>`	`getFatLinks()`	Track the status of this batch, we need a notice when the batch is finished
`final PulsarSession`	`getSession()`	The pulsar session to use
`final Queue<NormUrl>`	`getSeeds()`	The urls of portal pages from where hyper links are extracted from
`CrawlableFatLink`	`remove(String url)`
`CrawlableFatLink`	`remove(FatLink fatLink)`
`Boolean`	`hasMore()`
`Integer`	`collectTo(List<UrlAware> sink)`
`List<String>`	`dump()`
`Unit`	`clear()`

Methods inherited from class ai.platon.pulsar.common.collect.collector.AbstractPriorityDataCollector
getCapacity, getPriority, toString
Methods inherited from class ai.platon.pulsar.common.collect.collector.AbstractDataCollector
collectTo, collectTo, collectTo, getCollectCount, getCollectTime, getCollectedCount, getCountry, getCreateTime, getDeadTime, getDistrict, getEstimatedExternalSize, getExternalSize, getFirstCollectTime, getId, getLabels, getLang, getLastCollectedTime, setCollectCount, setCollectedCount, setCountry, setDeadTime, setDistrict, setFirstCollectTime, setLang, setLastCollectedTime
Methods inherited from class ai.platon.pulsar.common.collect.collector.DataCollector
deepClear, isDead
Methods inherited from class ai.platon.pulsar.common.collect.collector.PriorityDataCollector
compareTo
Methods inherited from class ai.platon.pulsar.common.collect.CrawlableFatLinkCollector
removeAll, removeAll
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HyperlinkCollector

HyperlinkCollector(PulsarSession session, Queue<NormUrl> seeds, Priority13 priority)

Method Detail

getUrlNormalizer

 final UrlNormalizerPipeline getUrlNormalizer()

setUrlNormalizer

 final Unit setUrlNormalizer(UrlNormalizerPipeline urlNormalizer)

getName
```
 String getName()
```

setName
```
 Unit setName(String name)
```

getSize
```
 Integer getSize()
```

getEstimatedSize
```
 Integer getEstimatedSize()
```

getFatLinks
```
 ConcurrentSkipListMap<String, CrawlableFatLink> getFatLinks()
```
Track the status of this batch, we need a notice when the batch is finished

getSession
```
 final PulsarSession getSession()
```
The pulsar session to use

getSeeds
```
 final Queue<NormUrl> getSeeds()
```
The urls of portal pages from where hyper links are extracted from

remove
```
 CrawlableFatLink remove(String url)
```

remove

 CrawlableFatLink remove(FatLink fatLink)

hasMore
```
 Boolean hasMore()
```

collectTo

 Integer collectTo(List<UrlAware> sink)

dump
```
 List<String> dump()
```

clear
```
 Unit clear()
```

Class HyperlinkCollector

Nested Class Summary

Field Summary

Constructor Summary

Enum Constant Summary

Method Summary

Methods inherited from class ai.platon.pulsar.common.collect.collector.AbstractPriorityDataCollector

Methods inherited from class ai.platon.pulsar.common.collect.collector.AbstractDataCollector

Methods inherited from class ai.platon.pulsar.common.collect.collector.DataCollector

Methods inherited from class ai.platon.pulsar.common.collect.collector.PriorityDataCollector

Methods inherited from class ai.platon.pulsar.common.collect.CrawlableFatLinkCollector

Methods inherited from class java.lang.Object

Constructor Detail

HyperlinkCollector

Method Detail

getUrlNormalizer

setUrlNormalizer

getName

setName

getSize

getEstimatedSize

getFatLinks

getSession

getSeeds

remove

remove

hasMore

collectTo

dump

clear