Class FeaturedDocument

  • All Implemented Interfaces:

    
    public class FeaturedDocument
    
                        

    A "featured document" is "very important" and the numerical features of each node are calculated, the numerical feature can be used to locate nodes or can be used by machine learning algorithms.

    FeaturedDocument is a wrapper for org.jsoup.nodes.Document, every node's numerical features are calculated by a ai.platon.pulsar.dom.features.FeatureCalculator which can be customized.

    The DOM of a FeaturedDocument is immutable, no method is provided to append or insert elements or nodes.

    FeaturedDocument provides a set of powerful methods to select elements, text contexts, attributes and so on:

    • select: retrieves a list of elements matching the CSS query

    • selectFirst: retrieves the first matching element

    • selectFirstText: retrieves the text content of the first matching element

    • selectFirstAttribute: retrieves the attribute value associated to the given name of the first matching element

    • selectHyperlinks: retrieves all hyperlinks of elements matching the CSS query

    • selectAnchors: retrieves all anchor elements matching the CSS query

    • selectImages: retrieves all image elements matching the CSS query

    Other methods provided include DOM traversal, node counting, document attribute retrieval, export, and so on.

    • Constructor Detail

      • FeaturedDocument

        FeaturedDocument(String baseURI)
        The constructor
        Parameters:
        baseURI - The URL where the HTML was retrieved from.
      • FeaturedDocument

        FeaturedDocument(Document document)
    • Method Detail

      • setFeatures

         final Unit setFeatures(RealVector features)
      • unbox

         final Document unbox()

        Get the underlying document.

      • isNil

         final Boolean isNil()

        Check if this document is nil.

      • createElement

         final Element createElement(String tagName)

        Create a new Element, with this document's base uri. Does not make the new element a child of this document.

        Parameters:
        tagName - element tag name (e.g.
      • guessTitle

         final String guessTitle()

        Guess the document's title.

        The title should be guessed for some site without a <title> tag inside a <head> tag.

      • absoluteLinks

         final Unit absoluteLinks()

        Make all links in the document to be absolute.

      • select

        @JvmOverloads() final Elements select(String query, Integer offset, Integer limit)

        Find elements that match the CSS query. Matched elements may include the document, or any of its children.

        This method is generally more powerful to use than the DOM-type {@code getElementBy*} methods, because multiple filters can be combined.

      • select

        @JvmOverloads() final Elements select(String query, Integer offset)

        Find elements that match the CSS query. Matched elements may include the document, or any of its children.

        This method is generally more powerful to use than the DOM-type {@code getElementBy*} methods, because multiple filters can be combined.

      • select

        @JvmOverloads() final Elements select(String query)

        Find elements that match the CSS query. Matched elements may include the document, or any of its children.

        This method is generally more powerful to use than the DOM-type {@code getElementBy*} methods, because multiple filters can be combined.

      • select

         final <T extends Any> List<T> select(String query, Integer offset, Integer limit, Function1<Element, T> transformer)

        Find elements that match the CSS query. Matched elements may include the document, or any of its children.

        This method is generally more powerful to use than the DOM-type {@code getElementBy*} methods, because multiple filters can be combined.

      • selectFirst

         final Element selectFirst(String query)

        Find the first element that match the CSS query. Matched element may be the document, or any of its children.

        This method is generally more powerful to use than the DOM-type {@code getElementBy*} methods, because multiple filters can be combined.

      • selectFirst

         final <T extends Any> T selectFirst(String query, Function1<Element, T> transformer)

        Find the first element that match the CSS query. Matched element may be the document, or any of its children.

        This method is generally more powerful to use than the DOM-type {@code getElementBy*} methods, because multiple filters can be combined.

      • selectFirstOrNull

         final Element selectFirstOrNull(String query)

        Find the first element that match the CSS query. Matched element may be the document, or any of its children.

        This method is generally more powerful to use than the DOM-type {@code getElementBy*} methods, because multiple filters can be combined.

      • selectFirstOrNull

         final <T extends Any> T selectFirstOrNull(String query, Function1<Element, T> transformer)

        Find the first element that match the CSS query. Matched element may be the document, or any of its children.

        This method is generally more powerful to use than the DOM-type {@code getElementBy*} methods, because multiple filters can be combined.

      • selectFirstOptional

         final Optional<Element> selectFirstOptional(String query)

        Find elements that match the CSS query. Matched elements may include the document, or any of its children.

        This method is generally more powerful to use than the DOM-type {@code getElementBy*} methods, because multiple filters can be combined.

        Return the first element, if no element matches the query, return an Optional object.

      • selectFirstOptional

         final <T extends Any> Optional<T> selectFirstOptional(String query, Function1<Element, T> transformer)

        Find elements that match the CSS query. Matched elements may include the document, or any of its children.

        This method is generally more powerful to use than the DOM-type {@code getElementBy*} methods, because multiple filters can be combined.

        Return the first element, if no element matches the query, return an Optional object.

      • first

        @Deprecated(message = "Inappropriate name", replaceWith = @ReplaceWith(imports = {}, expression = "selectFirst(query)")) final Element first(String query)
      • first

        @Deprecated(message = "Inappropriate name", replaceWith = @ReplaceWith(imports = {}, expression = "selectFirst(query, transformer)")) final <T extends Any> T first(String query, Function1<Element, T> transformer)
      • selectFirstText

         final String selectFirstText(String query)

        Find the text content of the first element that match the CSS query. Matched element may be the document, or any of its children.

      • selectFirstTextOrNull

         final String selectFirstTextOrNull(String query)

        Find the text content of the first element that match the CSS query. Matched element may be the document, or any of its children.

      • selectFirstTextOptional

         final Optional<String> selectFirstTextOptional(String query)

        Find the text content of the first element that match the CSS query. Matched element may be the document, or any of its children.

      • firstText

         final String firstText(String query)

        Find the text content of the first element that match the CSS query. Matched element may be the document, or any of its children.

      • firstTextOrNull

         final String firstTextOrNull(String query)

        Find the text content of the first element that match the CSS query. Matched element may be the document, or any of its children.

      • firstTextOptional

         final Optional<String> firstTextOptional(String query)

        Find the text content of the first element that match the CSS query. Matched element may be the document, or any of its children.

      • selectFirstAttribute

        @JvmOverloads() final String selectFirstAttribute(String query, String attrName, String defaultValue)

        Find the attribute value of the first element that match the CSS query. Matched element may be the document, or any of its children.

      • selectFirstAttribute

        @JvmOverloads() final String selectFirstAttribute(String query, String attrName)

        Find the attribute value of the first element that match the CSS query. Matched element may be the document, or any of its children.

      • selectFirstAttributeOrNull

         final String selectFirstAttributeOrNull(String query, String attrName)

        Find the attribute value of the first element that match the CSS query. Matched element may be the document, or any of its children.

      • firstAttribute

        @JvmOverloads() final String firstAttribute(String query, String attrName, String defaultValue)

        Find the attribute value of the first element that match the CSS query. Matched element may be the document, or any of its children.

      • firstAttribute

        @JvmOverloads() final String firstAttribute(String query, String attrName)

        Find the attribute value of the first element that match the CSS query. Matched element may be the document, or any of its children.

      • firstAttributeOrNull

         final String firstAttributeOrNull(String query, String attrName)

        Find the attribute value of the first element that match the CSS query. Matched element may be the document, or any of its children.

      • selectHyperlinks

        @JvmOverloads() final List<Hyperlink> selectHyperlinks(String query)

        Find hyperlinks in elements matching the CSS query.

      • forEach

         final Unit forEach(Function1<Node, Unit> action)

        Traverse the DOM and apply the action to each Node.

      • forEachMatching

         final Unit forEachMatching(Function1<Node, Boolean> predicate, Function1<Node, Unit> action)

        Traverse the DOM and apply the action to each Node that matches predicate.

      • forEachElement

         final Unit forEachElement(Function1<Element, Unit> action)

        Traverse the DOM and apply the action to each Element.

      • removeAttrs

         final Unit removeAttrs(String attributeKeys)

        Remove attributes associated with the given keys.

      • removeScripts

         final Unit removeScripts()

        Remove all script nodes in the document.

      • stripScripts

         final Unit stripScripts()

        Remove all script nodes in the document.

      • removeStyles

         final Unit removeStyles()

        Remove all style nodes from the document, and remove all style attributes from all elements.

      • exportTo

         final Path exportTo(Path path)

        Export the document to the given path.