Class AbstractHttpProtocol
-
- All Implemented Interfaces:
-
ai.platon.pulsar.common.config.Configurable,ai.platon.pulsar.skeleton.crawl.common.LazyConfigurable,ai.platon.pulsar.skeleton.crawl.protocol.Protocol,java.lang.AutoCloseable
public abstract class AbstractHttpProtocol implements Protocol
-
-
Field Summary
Fields Modifier and Type Field Description private final BooleanisActiveprivate final BooleansupportParallelprivate ImmutableConfigconf
-
Constructor Summary
Constructors Constructor Description AbstractHttpProtocol()
-
Method Summary
Modifier and Type Method Description final BooleanisActive()BooleangetSupportParallel()ImmutableConfiggetConf()The configuration UnitsetConf(ImmutableConfig conf)The configuration Unitconfigure(ImmutableConfig conf1)Set up the protocol. Unitreset()Reset the protocol environment, so the peer host view the client as a new one Collection<Response>getResponses(Collection<WebPage> pages, VolatileConfig volatileConfig)ProtocolOutputgetProtocolOutput(WebPage page)Returns the ProtocolOutput for a fetch list entry. ProtocolOutputgetProtocolOutputDeferred(WebPage page)Returns the ProtocolOutput for a fetch list entry. abstract ResponsegetResponse(WebPage page, Boolean followRedirects)Get the protocol response for the given page. abstract ResponsegetResponseDeferred(WebPage page, Boolean followRedirects)BaseRobotRulesgetRobotRules(WebPage page)Retrieve robot rules applicable for this url. Unitclose()StringtoString()-
-
Method Detail
-
getSupportParallel
Boolean getSupportParallel()
-
getConf
ImmutableConfig getConf()
The configuration
-
configure
Unit configure(ImmutableConfig conf1)
Set up the protocol. Sometimes the protocol can not to be constructed with parameters, so it need a secondary setup.
-
getResponses
Collection<Response> getResponses(Collection<WebPage> pages, VolatileConfig volatileConfig)
-
getProtocolOutput
ProtocolOutput getProtocolOutput(WebPage page)
Returns the ProtocolOutput for a fetch list entry.
-
getProtocolOutputDeferred
ProtocolOutput getProtocolOutputDeferred(WebPage page)
Returns the ProtocolOutput for a fetch list entry.
-
getResponse
abstract Response getResponse(WebPage page, Boolean followRedirects)
Get the protocol response for the given page.
- Parameters:
page- The page to get the response for.followRedirects- Whether to follow redirects.- Returns:
The response for the given page.
-
getResponseDeferred
abstract Response getResponseDeferred(WebPage page, Boolean followRedirects)
-
getRobotRules
BaseRobotRules getRobotRules(WebPage page)
Retrieve robot rules applicable for this url.
- Parameters:
page- The Web page- Returns:
robot rules (specific for this url or default), never null
-
-
-
-