RobotRulesParser

All Implemented Interfaces:

ai.platon.pulsar.common.config.Configurable , ai.platon.pulsar.skeleton.crawl.common.LazyConfigurable
```
public abstract class RobotRulesParser
 implements LazyConfigurable
                    
```
This class uses crawler-commons for handling the parsing of robots.txt files. It emits SimpleRobotRules objects, which describe the download permissions as described in SimpleRobotRulesParser.

Nested Class Summary

Nested Classes
Modifier and Type	Class	Description
`public class`	`RobotRulesParser.Companion`

Field Summary

Fields
Modifier and Type	Field	Description
`public final static BaseRobotRules`	`EMPTY_RULES`
`private ImmutableConfig`	`conf`
`public final static RobotRulesParser.Companion`	`Companion`

Constructor Summary

Constructors
Constructor	Description
`RobotRulesParser(ImmutableConfig conf)`

Enum Constant Summary

Enum Constants
Enum Constant	Description

Method Summary

Modifier and Type	Method	Description
`ImmutableConfig`	`getConf()`
`Unit`	`setConf(ImmutableConfig conf)`
`Unit`	`configure(ImmutableConfig conf1)`
`final BaseRobotRules`	`parseRules(String url, ByteArray content, String contentType, String robotName)`	Parses the robots content using the SimpleRobotRulesParser from crawler commons
`final BaseRobotRules`	`getRobotRulesSet(Protocol protocol, String url)`
`abstract BaseRobotRules`	`getRobotRulesSet(Protocol protocol, URL url)`
`final static Unit`	`main(Array<String> argv)`	command-line main for testing

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail
- RobotRulesParser
```
RobotRulesParser(ImmutableConfig conf)
```

Method Detail
- getConf
```
 ImmutableConfig getConf()
```
- setConf
```
 Unit setConf(ImmutableConfig conf)
```
- configure
```
 Unit configure(ImmutableConfig conf1)
```
- parseRules
```
@NotNull() final BaseRobotRules parseRules(String url, ByteArray content, String contentType, String robotName)
```
  Parses the robots content using the SimpleRobotRulesParser from crawler commons
  
  Parameters:
  
  url - A string containing url
  
  content - Contents of the robots file in a byte array
  
  contentType - The content type of the robots file
  
  robotName - A string containing all the robots agent names used by parser for matching
  
  Returns:
  
  BaseRobotRules object
- getRobotRulesSet
```
@NotNull() final BaseRobotRules getRobotRulesSet(Protocol protocol, String url)
```
- getRobotRulesSet
```
@NotNull() abstract BaseRobotRules getRobotRulesSet(Protocol protocol, URL url)
```
- main
```
 final static Unit main(Array<String> argv)
```
  command-line main for testing