Package ai.platon.pulsar.crawl.protocol

Interface RobotRules

  • All Implemented Interfaces:
    
public interface RobotRules

    This class holds the rules which were parsed from a robots.txt file, and can test paths against those rules.

    • Nested Class Summary

      Nested Classes 
      Modifier and Type Class Description

    • Field Summary

      Fields 
      Modifier and Type Field Description

    • Constructor Summary

      Constructors 
      Constructor Description

    • Enum Constant Summary

      Enum Constants 
      Enum Constant Description

    • Method Summary

      Modifier and Type Method Description
      abstract Boolean isAllowed(URL url) Returns false if the robots.txt file prohibits us from accessing the given url, or true otherwise.
      abstract Long getExpireTime() Get expire time
      abstract Long getCrawlDelay() Get Crawl-Delay, in milliseconds.

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    • Constructor Detail

    • Method Detail

      • isAllowed

         abstract Boolean isAllowed(URL url)

        Returns false if the robots.txt file prohibits us from accessing the given url, or true otherwise.

      • getCrawlDelay

         abstract Long getCrawlDelay()

        Get Crawl-Delay, in milliseconds. This returns -1 if not set.