Class FeaturedDocument

  • All Implemented Interfaces:

    
    public class FeaturedDocument
    
                        

    An HTML Document.

    A ‘featured document’ refers to a ‘very important document.’ Additionally, the numerical features of each node are calculated, and these numerical features can be used to locate nodes or utilized by machine learning algorithms.

    FeaturedDocument is a wrapper for org.jsoup.nodes.Document, every node's numerical features are calculated by a ai.platon.pulsar.dom.features.FeatureCalculator which can be customized.

    FeaturedDocument provides a set of powerful methods to select elements, text contexts, attributes and so on:

    Other methods provided include DOM traversal, node counting, document attribute retrieval, export, and so on.

    • Constructor Detail

      • FeaturedDocument

        FeaturedDocument(String baseURI)
        The constructor
        Parameters:
        baseURI - The URL where the HTML was retrieved from.
      • FeaturedDocument

        FeaturedDocument(Document document)
        Parameters:
        document - The underlying org.jsoup.nodes.Document
    • Method Detail

      • unbox

         final Document unbox()

        Get the underlying document.

        Returns:

        the underlying Jsoup document.

      • isNil

         final Boolean isNil()

        Check if this document is nil.

      • isInternal

         final <Error class: unknown class> isInternal()

        Check if this document is internal.

      • isNotInternal

         final <Error class: unknown class> isNotInternal()

        Check if this document is internal.

      • guessTitle

         final String guessTitle()

        Guess the document's title.

        The title should be guessed for some site without a <title> tag inside a <head> tag.

      • absoluteLinks

         final Unit absoluteLinks()

        Make all links in the document to be absolute.

      • select

        @JvmOverloads() final Elements select(String query, Integer offset, Integer limit)

        Find elements that match the CSS query. Matched elements may include the document, or any of its children.

        This method is generally more powerful to use than the DOM-type {@code getElementBy*} methods, because multiple filters can be combined.

        Parameters:
        query - A power-CSS query
        offset - The offset of the first element to return
        limit - The maximum number of elements to return
        Returns:

        A list of elements that match the query

      • select

        @JvmOverloads() final Elements select(String query, Integer offset)

        Find elements that match the CSS query. Matched elements may include the document, or any of its children.

        This method is generally more powerful to use than the DOM-type {@code getElementBy*} methods, because multiple filters can be combined.

        Parameters:
        query - A power-CSS query
        offset - The offset of the first element to return
        Returns:

        A list of elements that match the query

      • select

        @JvmOverloads() final Elements select(String query)

        Find elements that match the CSS query. Matched elements may include the document, or any of its children.

        This method is generally more powerful to use than the DOM-type {@code getElementBy*} methods, because multiple filters can be combined.

        Parameters:
        query - A power-CSS query
        Returns:

        A list of elements that match the query

      • select

         final <T extends Any> List<T> select(String query, Integer offset, Integer limit, Function1<Element, T> transformer)

        Find elements that match the CSS query. Matched elements may include the document, or any of its children.

        This method is generally more powerful to use than the DOM-type {@code getElementBy*} methods, because multiple filters can be combined.

        Parameters:
        query - A power-CSS query
        offset - The offset of the first element to return
        limit - The maximum number of elements to return
        transformer - A transformer to transform the element to the desired type
        Returns:

        A list of elements that match the query

      • selectFirst

         final Element selectFirst(String query)

        Find the first element that match the CSS query. Matched element may be the document, or any of its children.

        This method is generally more powerful to use than the DOM-type {@code getElementBy*} methods, because multiple filters can be combined.

        Parameters:
        query - A power-CSS query
        Returns:

        The first element that match the query

      • selectFirst

         final <T extends Any> <Error class: unknown class> selectFirst(String query, Function1<Element, T> transformer)

        Find the first element that match the CSS query. Matched element may be the document, or any of its children.

        This method is generally more powerful to use than the DOM-type {@code getElementBy*} methods, because multiple filters can be combined.

        Parameters:
        query - A power-CSS query
        transformer - A transformer to transform the element to the desired type
        Returns:

        The first element that match the query

      • selectFirstOrNull

         final Element selectFirstOrNull(String query)

        Find the first element that match the CSS query. Matched element may be the document, or any of its children.

        This method is generally more powerful to use than the DOM-type {@code getElementBy*} methods, because multiple filters can be combined.

        Parameters:
        query - A power-CSS query
        Returns:

        The first element that match the query, if no element matches the query, return null

      • selectFirstOrNull

         final <T extends Any> <Error class: unknown class> selectFirstOrNull(String query, Function1<Element, T> transformer)

        Find the first element that match the CSS query. Matched element may be the document, or any of its children.

        This method is generally more powerful to use than the DOM-type {@code getElementBy*} methods, because multiple filters can be combined.

        Parameters:
        query - A power-CSS query
        transformer - A transformer to transform the element to the desired type
        Returns:

        The first element that match the query, if no element matches the query, return null

      • selectFirstOptional

         final Optional<Element> selectFirstOptional(String query)

        Find elements that match the CSS query. Matched elements may include the document, or any of its children.

        This method is generally more powerful to use than the DOM-type {@code getElementBy*} methods, because multiple filters can be combined.

        Return the first element, if no element matches the query, return an Optional object.

        Parameters:
        query - A power-CSS query
        Returns:

        The first element that match the query

      • selectFirstOptional

         final <T extends Any> <Error class: unknown class> selectFirstOptional(String query, Function1<Element, T> transformer)

        Find elements that match the CSS query. Matched elements may include the document, or any of its children.

        This method is generally more powerful to use than the DOM-type {@code getElementBy*} methods, because multiple filters can be combined.

        Return the first element, if no element matches the query, return an Optional object.

        Parameters:
        query - A power-CSS query
        transformer - A transformer to transform the element to the desired type
        Returns:

        The first element that match the query

      • selectTextAll

        @JvmOverloads() final List<String> selectTextAll(String query, String attrName, Integer offset, Integer limit)

        Find text of elements that match the CSS query. Matched elements may include the document, or any of its children.

        Parameters:
        query - A power-CSS query
        attrName - The attribute name of the text to return
        offset - The offset of the first element to return
        limit - The maximum number of elements to return
        Returns:

        A list of attribute values of elements that match the query

      • selectTextAll

        @JvmOverloads() final List<String> selectTextAll(String query, String attrName, Integer offset)

        Find text of elements that match the CSS query. Matched elements may include the document, or any of its children.

        Parameters:
        query - A power-CSS query
        attrName - The attribute name of the text to return
        offset - The offset of the first element to return
        Returns:

        A list of attribute values of elements that match the query

      • selectTextAll

        @JvmOverloads() final List<String> selectTextAll(String query, String attrName)

        Find text of elements that match the CSS query. Matched elements may include the document, or any of its children.

        Parameters:
        query - A power-CSS query
        attrName - The attribute name of the text to return
        Returns:

        A list of attribute values of elements that match the query

      • selectFirstText

         final String selectFirstText(String query)

        Find the text content of the first element that match the CSS query. Matched element may be the document, or any of its children.

        Parameters:
        query - A power-CSS query
        Returns:

        The text content of the first element that match the query

      • selectFirstTextOrNull

         final String selectFirstTextOrNull(String query)

        Find the text content of the first element that match the CSS query. Matched element may be the document, or any of its children.

        Parameters:
        query - A power-CSS query
        Returns:

        The text content of the first element that match the query

      • selectFirstTextOptional

         final Optional<String> selectFirstTextOptional(String query)

        Find the text content of the first element that match the CSS query. Matched element may be the document, or any of its children.

        Parameters:
        query - A power-CSS query
        Returns:

        The text content of the first element that match the query

      • selectAttributes

        @JvmOverloads() final List<String> selectAttributes(String query, String attrName, Integer offset, Integer limit)

        Find the attribute value of the first element that match the CSS query. Matched element may be the document, or any of its children.

        Parameters:
        query - A power-CSS query
        attrName - The attribute name
        offset - The offset of the first element to return
        limit - The maximum number of elements to return
        Returns:

        A list of attribute values of elements that match the query

      • selectAttributes

        @JvmOverloads() final List<String> selectAttributes(String query, String attrName, Integer offset)

        Find the attribute value of the first element that match the CSS query. Matched element may be the document, or any of its children.

        Parameters:
        query - A power-CSS query
        attrName - The attribute name
        offset - The offset of the first element to return
        Returns:

        A list of attribute values of elements that match the query

      • selectAttributes

        @JvmOverloads() final List<String> selectAttributes(String query, String attrName)

        Find the attribute value of the first element that match the CSS query. Matched element may be the document, or any of its children.

        Parameters:
        query - A power-CSS query
        attrName - The attribute name
        Returns:

        A list of attribute values of elements that match the query

      • selectFirstAttribute

        @JvmOverloads() final String selectFirstAttribute(String query, String attrName, String defaultValue)

        Find the attribute value of the first element that match the CSS query. Matched element may be the document, or any of its children.

        Parameters:
        query - A power-CSS query
        attrName - The attribute name
        Returns:

        The attribute value of the first element that match the query

      • selectFirstAttribute

        @JvmOverloads() final String selectFirstAttribute(String query, String attrName)

        Find the attribute value of the first element that match the CSS query. Matched element may be the document, or any of its children.

        Parameters:
        query - A power-CSS query
        attrName - The attribute name
        Returns:

        The attribute value of the first element that match the query

      • selectFirstAttributeOrNull

         final String selectFirstAttributeOrNull(String query, String attrName)

        Find the attribute value of the first element that match the CSS query. Matched element may be the document, or any of its children.

        Parameters:
        query - A power-CSS query
        attrName - The attribute name
        Returns:

        The attribute value of the first element that match the query

      • selectFirstAttributeOptional

         final Optional<String> selectFirstAttributeOptional(String query, String attrName)

        Find the attribute value of the first element that match the CSS query. Matched element may be the document, or any of its children.

        Parameters:
        query - A power-CSS query
        attrName - The attribute name
        Returns:

        The attribute value of the first element that match the query

      • selectHyperlinks

        @JvmOverloads() final List<Hyperlink> selectHyperlinks(String query, Integer offset, Integer limit)

        Find hyperlinks in elements matching the CSS query.

        Parameters:
        query - A power-CSS query
        offset - The offset of the first element to return
        limit - The maximum number of elements to return
        Returns:

        A list of hyperlinks of elements that match the query

      • selectHyperlinks

        @JvmOverloads() final List<Hyperlink> selectHyperlinks(String query, Integer offset)

        Find hyperlinks in elements matching the CSS query.

        Parameters:
        query - A power-CSS query
        offset - The offset of the first element to return
        Returns:

        A list of hyperlinks of elements that match the query

      • selectHyperlinks

        @JvmOverloads() final List<Hyperlink> selectHyperlinks(String query)

        Find hyperlinks in elements matching the CSS query.

        Parameters:
        query - A power-CSS query
        Returns:

        A list of hyperlinks of elements that match the query

      • selectAnchors

        @JvmOverloads() final List<GeoAnchor> selectAnchors(String query, Integer offset, Integer limit)

        Find anchor elements matching the CSS query.

        Parameters:
        query - A power-CSS query
        offset - The offset of the first element to return
        limit - The maximum number of elements to return
        Returns:

        A list of anchor elements that match the query

      • selectAnchors

        @JvmOverloads() final List<GeoAnchor> selectAnchors(String query, Integer offset)

        Find anchor elements matching the CSS query.

        Parameters:
        query - A power-CSS query
        offset - The offset of the first element to return
        Returns:

        A list of anchor elements that match the query

      • selectAnchors

        @JvmOverloads() final List<GeoAnchor> selectAnchors(String query)

        Find anchor elements matching the CSS query.

        Parameters:
        query - A power-CSS query
        Returns:

        A list of anchor elements that match the query

      • selectImages

        @JvmOverloads() final List<String> selectImages(String query, Integer offset, Integer limit)

        Find image elements matching the CSS query.

        Parameters:
        query - A power-CSS query
        offset - The offset of the first element to return
        limit - The maximum number of elements to return
        Returns:

        A list of image elements that match the query

      • selectImages

        @JvmOverloads() final List<String> selectImages(String query, Integer offset)

        Find image elements matching the CSS query.

        Parameters:
        query - A power-CSS query
        offset - The offset of the first element to return
        Returns:

        A list of image elements that match the query

      • selectImages

        @JvmOverloads() final List<String> selectImages(String query)

        Find image elements matching the CSS query.

        Parameters:
        query - A power-CSS query
        Returns:

        A list of image elements that match the query

      • forEach

         final Unit forEach(Function1<Node, Unit> action)

        Traverse the DOM and apply the action to each Node.

        Parameters:
        action - The action to apply to each Node
      • forEachMatching

         final Unit forEachMatching(Function1<Node, Boolean> predicate, Function1<Node, Unit> action)

        Traverse the DOM and apply the action to each Node that matches predicate.

        Parameters:
        predicate - The predicate to match Node
        action - The action to apply to each Node
      • forEachElement

         final Unit forEachElement(Function1<Element, Unit> action)

        Traverse the DOM and apply the action to each Element.

        Parameters:
        action - The action to apply to each Element
      • forEachElementMatching

         final Unit forEachElementMatching(Function1<Element, Boolean> predicate, Function1<Element, Unit> action)

        Traverse the DOM and apply the action to each Element that matches predicate.

        Parameters:
        predicate - The predicate to match Element
        action - The action to apply to each Element
      • count

         final Integer count(Function1<Node, Boolean> predicate)

        Count nodes matching predicate.

        Parameters:
        predicate - The predicate to match Node
      • getFeature

         final Double getFeature(Integer key)

        Retrieves the feature with the given key.

        Parameters:
        key - The key of the feature
      • formatFeatures

         final String formatFeatures(Integer featureKeys)

        Format node features.

        Parameters:
        featureKeys - The keys of the features to format
      • removeAttrs

         final Unit removeAttrs(String attributeKeys)

        Remove attributes associated with the given keys.

      • stripScripts

         final Unit stripScripts()

        Remove all script nodes in the document.

      • removeStyles

         final Unit removeStyles()

        Remove all style nodes from the document, and remove all style attributes from all elements.

      • export

         final Path export()

        Export the document.

        Returns:

        The path of the exported document

      • exportTo

         final Path exportTo(Path path)

        Export the document to the given path.

        Parameters:
        path - The path to export the document
        Returns:

        The path of the exported document

      • toString

         String toString()

        Get the string representation of the document.