Package ai.platon.pulsar.skeleton.crawl.protocol

Class RobotRulesParser

  • All Implemented Interfaces:
    ai.platon.pulsar.common.config.Configurable , ai.platon.pulsar.skeleton.crawl.common.LazyConfigurable 
    
public abstract class RobotRulesParser
 implements LazyConfigurable

    This class uses crawler-commons for handling the parsing of robots.txt files. It emits SimpleRobotRules objects, which describe the download permissions as described in SimpleRobotRulesParser.

    • Constructor Detail

      • RobotRulesParser

        RobotRulesParser(ImmutableConfig conf)

    • Method Detail

      • getConf

         ImmutableConfig getConf()

      • parseRules

        @NotNull() final BaseRobotRules parseRules(String url, ByteArray content, String contentType, String robotName)

        Parses the robots content using the SimpleRobotRulesParser from crawler commons

        Parameters:
        url - A string containing url
        content - Contents of the robots file in a byte array
        contentType - The content type of the robots file
        robotName - A string containing all the robots agent names used by parser for matching
        Returns:

        BaseRobotRules object