Package ai.platon.pulsar.skeleton.crawl.filter
Index writer plugin for Apache Solr.
See: Description
-
Class Summary Class Description BlockFilter ChainedUrlNormalizer CrawlFilter TODO : configurable CrawlFilters TODO : need full unit test CrawlUrlFilters Creates and caches CrawlUrlFilter implementing plugins. TextFilter UrlFilterException AbstractScopedUrlNormalizer RegexUrlNormalizer UrlNormalizerKt -
Interface Summary Interface Description CrawlUrlFilter Interface used to limit which URLs enter AppConstants. UrlNormalizer Interface used to convert URLs to normal form and optionally perform substitutions ScopedUrlNormalizer