Package ai.platon.pulsar.crawl.protocol

Object RobotRulesParser.Companion

    • Nested Class Summary

      Nested Classes 
      Modifier and Type Class Description

    • Constructor Summary

      Constructors 
      Constructor Description

    • Enum Constant Summary

      Enum Constants 
      Enum Constant Description

    • Method Summary

      Modifier and Type Method Description
      final Logger getLOG()
      final Hashtable<String, BaseRobotRules> getCACHE()
      final BaseRobotRules getFORBID_ALL_RULES() A BaseRobotRules object appropriate for use when the robots.txt file is not fetched due to a 403/Forbidden response; all requests are disallowed.
      final Unit setFORBID_ALL_RULES(BaseRobotRules FORBID_ALL_RULES) A BaseRobotRules object appropriate for use when the robots.txt file is not fetched due to a 403/Forbidden response; all requests are disallowed.

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    • Constructor Detail

    • Method Detail

      • getLOG

         final Logger getLOG()

      • getFORBID_ALL_RULES

         final BaseRobotRules getFORBID_ALL_RULES()

        A BaseRobotRules object appropriate for use when the robots.txt file is not fetched due to a 403/Forbidden response; all requests are disallowed.

      • setFORBID_ALL_RULES

         final Unit setFORBID_ALL_RULES(BaseRobotRules FORBID_ALL_RULES)

        A BaseRobotRules object appropriate for use when the robots.txt file is not fetched due to a 403/Forbidden response; all requests are disallowed.