All Implemented Interfaces:

ai.platon.pulsar.common.config.Parameterized , ai.platon.pulsar.crawl.common.JobInitialized
```
public final class GenerateComponent
 implements Parameterized, JobInitialized
                    
```
The generate component.

Nested Class Summary

Nested Classes
Modifier and Type	Class	Description
`public class`	`GenerateComponent.Companion`

Field Summary

Fields
Modifier and Type	Field	Description
`private final Logger`	`LOG`
`private final CrawlFilters`	`crawlFilters`
`private final WebDb`	`webDb`
`private final CrawlUrlFilters`	`urlFilters`
`private final ChainedUrlNormalizer`	`urlNormalizers`
`private final FetchSchedule`	`fetchSchedule`
`private final MiscMessageWriter`	`messageWriter`
`private final ImmutableConfig`	`conf`
`public final static GenerateComponent.Companion`	`Companion`

Constructor Summary

Constructors
Constructor	Description
`GenerateComponent(CrawlFilters crawlFilters, WebDb webDb, CrawlUrlFilters urlFilters, ChainedUrlNormalizer urlNormalizers, FetchSchedule fetchSchedule, MiscMessageWriter messageWriter, ImmutableConfig conf)`

Enum Constant Summary

Enum Constants
Enum Constant	Description

Method Summary

Modifier and Type	Method	Description
`final Logger`	`getLOG()`
`final CrawlFilters`	`getCrawlFilters()`
`final WebDb`	`getWebDb()`
`final CrawlUrlFilters`	`getUrlFilters()`
`final ChainedUrlNormalizer`	`getUrlNormalizers()`
`final FetchSchedule`	`getFetchSchedule()`
`final MiscMessageWriter`	`getMessageWriter()`
`final ImmutableConfig`	`getConf()`
`Unit`	`setup(ImmutableConfig jobConf)`
`Params`	`getParams()`
`final Pair<Boolean, String>`	`shouldFetch(String url, String reversedUrl, WebPage page)`	TODO : We may move some filters to hbase query filters directly TODO : Move to CrawlFilter

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

GenerateComponent

GenerateComponent(CrawlFilters crawlFilters, WebDb webDb, CrawlUrlFilters urlFilters, ChainedUrlNormalizer urlNormalizers, FetchSchedule fetchSchedule, MiscMessageWriter messageWriter, ImmutableConfig conf)

Method Detail

getLOG
```
 final Logger getLOG()
```

getCrawlFilters
```
 final CrawlFilters getCrawlFilters()
```

getWebDb
```
 final WebDb getWebDb()
```

getUrlFilters
```
 final CrawlUrlFilters getUrlFilters()
```

getUrlNormalizers

 final ChainedUrlNormalizer getUrlNormalizers()

getFetchSchedule
```
 final FetchSchedule getFetchSchedule()
```

getMessageWriter

 final MiscMessageWriter getMessageWriter()

getConf
```
 final ImmutableConfig getConf()
```

setup
```
 Unit setup(ImmutableConfig jobConf)
```

getParams
```
 Params getParams()
```

shouldFetch
```
 final Pair<Boolean, String> shouldFetch(String url, String reversedUrl, WebPage page)
```
TODO : We may move some filters to hbase query filters directly TODO : Move to CrawlFilter

Class GenerateComponent

Nested Class Summary

Field Summary

Constructor Summary

Enum Constant Summary

Method Summary

Methods inherited from class java.lang.Object

Constructor Detail

GenerateComponent

Method Detail

getLOG

getCrawlFilters

getWebDb

getUrlFilters

getUrlNormalizers

getFetchSchedule

getMessageWriter

getConf

setup

getParams

shouldFetch