Package ai.platon.pulsar.skeleton.crawl.common

Object URLUtil

  • All Implemented Interfaces:
    
public class URLUtil

    Utility class for URL analysis.

    protocol://username:password@hostname:port/pathname?search#hash -----------------------------href------------------------------ -----host---- ----------- origin -------------

    protocol - protocol scheme of the URL, including the final ':' hostname - domain name port - port number pathname - /pathname search - ?parameters hash - #fragment_identifier username - username specified before the domain name password - password specified before the domain name href - the entire URL origin - protocol://hostname:port host - hostname:port

    TODO: merge with ai.platon.pulsar.common.url.Urls

    • Constructor Detail

    • Method Detail

      • getPublicSuffix

         final String getPublicSuffix(URL url)

        Get the host's public suffix. For example, co.uk, com, etc.

        Since:

        6.0

      • getTopPrivateDomain

         final String getTopPrivateDomain(URL url)

        Returns the portion of this domain name that is one level beneath the {@linkplain #isPublicSuffix() public suffix}. For example, for {@code x.adwords.google.co.uk} it returns {@code google.co.uk}, since {@code co.uk} is a public suffix. Similarly, for {@code myblog.blogspot.com} it returns the same domain, {@code myblog.blogspot.com}, since {@code blogspot.com} is a public suffix.

        <p>If {@link #isTopPrivateDomain()} is true, the current domain name instance is returned.<p>This method can be used to determine the probable highest level parent domain for which cookies may be set, though even that depends on individual browsers' implementations of cookie controls.
        Since:

        6.0

      • getDomainName

         final String getDomainName(String url)

        Returns the domain name of the url. The domain name of a url is the substring of the url's hostname, w/o subdomain names. As an example <br></br> getDomainName(conf, new http://lucene.apache.org/) * <br></br> will return <br></br> apache.org

      • getDomainName

        @Deprecated(message = "Use getTopPrivateDomain(url) instead", replaceWith = @ReplaceWith(imports = {}, expression = "getTopPrivateDomain(url)")) final String getDomainName(URL url)

        Returns the domain name of the url. The domain name of a url is the substring of the url's hostname, w/o subdomain names. As an example <br></br> getDomainName(conf, new URL(http://lucene.apache.org/)) <br></br> will return <br></br> apache.org

      • isSameDomainName

         final Boolean isSameDomainName(URL url1, URL url2)

        Returns whether the given urls have the same domain name. As an example, <br></br> isSameDomain(new URL("http://lucene.apache.org") , new URL("http://people.apache.org/")) <br></br> will return true.

        Returns:

        true if the domain names are equal

      • isSameDomainName

         final Boolean isSameDomainName(String url1, String url2)

        Returns whether the given urls have the same domain name. As an example, <br></br> isSameDomain("http://lucene.apache.org" ,"http://people.apache.org/") <br></br> will return true.

        Returns:

        true if the domain names are equal

      • chooseRepr

         final static String chooseRepr(String src, String dst, Boolean temp)

        Given two urls, a src and a destination of a redirect, it returns the representative url.

        This method implements an extended version of the algorithm used by the Yahoo! Slurp crawler described here:<br></br> How * does the Yahoo! webcrawler handle redirects? <br></br> <br></br>

        • Choose target url if either url is malformed.

        • If different domains the keep the destination whether or not the redirect is temp or perm

        • a.com -> b.com*

        • If the redirect is permanent and the source is root, keep the source.

        • *a.com -> a.com?y=1 || *a.com -> a.com/xyz/index.html

        • If the redirect is permanent and the source is not root and the destination is root, keep the destination

        • a.com/xyz/index.html -> a.com*

        • If the redirect is permanent and neither the source nor the destination is root, then keep the destination

        • a.com/xyz/index.html -> a.com/abc/page.html*

        • If the redirect is temporary and source is root and destination is not root, then keep the source

        • *a.com -> a.com/xyz/index.html

        • If the redirect is temporary and source is not root and destination is root, then keep the destination

        • a.com/xyz/index.html -> a.com*

        • If the redirect is temporary and neither the source or the destination is root, then keep the shortest url. First check for the shortest host, and if both are equal then check by path. Path is first by length then by the number of / path separators.

        • a.com/xyz/index.html -> a.com/abc/page.html*

        • *www.a.com/xyz/index.html -> www.news.a.com/xyz/index.html

        • If the redirect is temporary and both the source and the destination are root, then keep the shortest sub-domain

        • *www.a.com -> www.news.a.com

        <br></br> While not in this logic there is a further piece of representative url logic that occurs during indexing and after scoring. During creation of the basic fields before indexing, if a url has a representative url stored we check both the url and its representative url (which should never be the same) against their linkrank scores and the highest scoring one is kept as the url and the lower scoring one is held as the orig url inside of the index.

        Parameters:
        src - The source url.
        dst - The destination url.
        temp - Is the redirect a temporary redirect.
        Returns:

        String The representative url.

      • getOrigin

         final String getOrigin(String url)

        Returns the lowercase origin for the url.

        Parameters:
        url - The url to check.
        Returns:

        String The hostname for the url.

      • getOriginOrNull

         final String getOriginOrNull(String url)

        Returns the lowercase origin for the url or null if the url is not well-formed.

        Parameters:
        url - The url to check.
        Returns:

        String The hostname for the url.

      • getHostName

         final String getHostName(String url)

        Returns the lowercase hostname for the url.

        Parameters:
        url - The url to check.
        Returns:

        String The hostname for the url.

      • getHostNameOrNull

         final String getHostNameOrNull(String url)

        Returns the lowercase hostname for the url or null if the url is not well-formed.

        Parameters:
        url - The url to check.
        Returns:

        String The hostname for the url.

      • getQuery

         final String getQuery(String url)

        Returns the path for the url. The path consists of the protocol, host, and path, but does not include the query string. The host is lowercased but the path is not.

        Parameters:
        url - The url to check.
        Returns:

        String The path for the url.