RobotRulesParser

All Implemented Interfaces:

ai.platon.pulsar.common.config.Configurable
```
public abstract class RobotRulesParser
 implements Configurable
                    
```
This class uses crawler-commons for handling the parsing of robots.txt files. It emits SimpleRobotRules objects, which describe the download permissions as described in SimpleRobotRulesParser.

Nested Class Summary

Nested Classes
Modifier and Type	Class	Description
`public class`	`RobotRulesParser.Companion`

Field Summary

Fields
Modifier and Type	Field	Description
`public final static BaseRobotRules`	`EMPTY_RULES`
`public final static RobotRulesParser.Companion`	`Companion`

Constructor Summary

Constructors
Constructor	Description
`RobotRulesParser()`
`RobotRulesParser(ImmutableConfig conf)`

Enum Constant Summary

Enum Constants
Enum Constant	Description

Method Summary

Modifier and Type	Method	Description
`ImmutableConfig`	`getConf()`	Get the Configuration object
`Unit`	`setConf(ImmutableConfig jobConf)`	Set the Configuration object
`final BaseRobotRules`	`parseRules(String url, ByteArray content, String contentType, String robotName)`	Parses the robots content using the SimpleRobotRulesParser from crawler commons
`final BaseRobotRules`	`getRobotRulesSet(Protocol protocol, String url)`
`abstract BaseRobotRules`	`getRobotRulesSet(Protocol protocol, URL url)`
`final static Unit`	`main(Array<String> argv)`	command-line main for testing

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Constructor Detail
  - RobotRulesParser
```
RobotRulesParser()
```
  - RobotRulesParser
```
RobotRulesParser(ImmutableConfig conf)
```
- Method Detail
  - getConf
```
@NotNull() ImmutableConfig getConf()
```
    Get the Configuration object
  - setConf
```
 Unit setConf(ImmutableConfig jobConf)
```
    Set the Configuration object
  - parseRules
```
@NotNull() final BaseRobotRules parseRules(String url, ByteArray content, String contentType, String robotName)
```
    Parses the robots content using the SimpleRobotRulesParser from crawler commons
    
    Parameters:
    
    url - A string containing url
    
    content - Contents of the robots file in a byte array
    
    contentType - The content type of the robots file
    
    robotName - A string containing all the robots agent names used by parser for matching
  - getRobotRulesSet
```
@NotNull() final BaseRobotRules getRobotRulesSet(Protocol protocol, String url)
```
  - getRobotRulesSet
```
@NotNull() abstract BaseRobotRules getRobotRulesSet(Protocol protocol, URL url)
```
  - main
```
 final static Unit main(Array<String> argv)
```
    command-line main for testing