Package ai.platon.pulsar.skeleton.crawl.protocol.http

Class AbstractHttpProtocol

  • All Implemented Interfaces:
    ai.platon.pulsar.common.config.Configurable , ai.platon.pulsar.skeleton.crawl.common.LazyConfigurable , ai.platon.pulsar.skeleton.crawl.protocol.Protocol , java.lang.AutoCloseable 
    
public abstract class AbstractHttpProtocol
 implements Protocol

    • Nested Class Summary

      Nested Classes 
      Modifier and Type Class Description

    • Field Summary

      Fields 
      Modifier and Type Field Description
      private final Boolean isActive
      private ImmutableConfig conf

    • Enum Constant Summary

      Enum Constants 
      Enum Constant Description

    • Constructor Detail

      • AbstractHttpProtocol

        AbstractHttpProtocol()

    • Method Detail

      • getConf

         ImmutableConfig getConf()

        The configuration

      • setConf

         Unit setConf(ImmutableConfig conf)

        The configuration

      • configure

         Unit configure(ImmutableConfig conf1)

        Set up the protocol. Sometimes the protocol can not to be constructed with parameters, so it need a secondary setup.

      • reset

         Unit reset()

        Reset the protocol environment, so the peer host view the client as a new one

      • getResponses

         Collection<Response> getResponses(Collection<WebPage> pages, VolatileConfig volatileConfig)

      • getResponse

         abstract Response getResponse(WebPage page, Boolean followRedirects)

        Get the protocol response for the given page.

        Parameters:
        page - The page to get the response for.
        followRedirects - Whether to follow redirects.
        Returns:

        The response for the given page.

      • getRobotRules

         BaseRobotRules getRobotRules(WebPage page)

        Retrieve robot rules applicable for this url.

        Parameters:
        page - The Web page
        Returns:

        robot rules (specific for this url or default), never null