Package ai.platon.pulsar.skeleton.crawl.protocol
See: Description
-
Class Summary Class Description ForwardingResponse Forward a response. ProtocolException ProtocolFactory Creates and caches Protocol plugins. ProtocolNotFound ProtocolOutput Response A response interface. RobotRulesParser This class uses crawler-commons for handling the parsing of robots.txtfiles. -
Interface Summary Interface Description Protocol A retriever for url content. RobotRules This class holds the rules which were parsed from a robots.txt file, and can test paths against those rules.