-
public class FeaturedDocumentA "featured document" is "very important" and the numerical features of each node are calculated, the numerical feature can be used to locate nodes or can be used by machine learning algorithms.
FeaturedDocument is a wrapper for org.jsoup.nodes.Document, every node's numerical features are calculated by a ai.platon.pulsar.dom.features.FeatureCalculator which can be customized.
The DOM of a FeaturedDocument is immutable, no method is provided to append or insert elements or nodes.
FeaturedDocument provides a set of powerful methods to select elements, text contexts, attributes and so on:
select: retrieves a list of elements matching the CSS query
selectFirst: retrieves the first matching element
selectFirstText: retrieves the text content of the first matching element
selectFirstAttribute: retrieves the attribute value associated to the given name of the first matching element
selectHyperlinks: retrieves all hyperlinks of elements matching the CSS query
selectAnchors: retrieves all anchor elements matching the CSS query
selectImages: retrieves all image elements matching the CSS query
Other methods provided include DOM traversal, node counting, document attribute retrieval, export, and so on.
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description public classFeaturedDocument.Companion
-
Field Summary
Fields Modifier and Type Field Description private final Integersequenceprivate final Stringtitleprivate final Stringlocationprivate final StringbaseUriprivate final Elementheadprivate final Elementbodyprivate final Stringtextprivate final StringownTextprivate final StringwholeTextprivate final Stringhtmlprivate final StringouterHtmlprivate final Charsetcharsetprivate final StringnodeNameprivate final Stringidprivate final StringclassNameprivate final Stringdataprivate final List<DataNode>dataNodesprivate final List<TextNode>textNodesprivate final List<Node>childNodesprivate final StringprettyHtmlprivate RealVectorfeaturesprivate final DocumentFragmentsfragmentsprivate final Documentdocument
-
Constructor Summary
Constructors Constructor Description FeaturedDocument(String baseUri)The constructor FeaturedDocument(FeaturedDocument other)The constructor FeaturedDocument(Document document)
-
Method Summary
Modifier and Type Method Description final IntegergetSequence()The process scope unique sequence. final StringgetTitle()final StringgetLocation()final StringgetBaseUri()final ElementgetHead()final ElementgetBody()final StringgetText()final StringgetOwnText()final StringgetWholeText()final StringgetHtml()final StringgetOuterHtml()final CharsetgetCharset()final StringgetNodeName()final StringgetId()final StringgetClassName()final StringgetData()final List<DataNode>getDataNodes()final List<TextNode>getTextNodes()final List<Node>getChildNodes()final StringgetPrettyHtml()final RealVectorgetFeatures()final UnitsetFeatures(RealVector features)final DocumentFragmentsgetFragments()final DocumentgetDocument()final Documentunbox()Get the underlying document. final BooleanisNil()Check if this document is nil. final BooleanisInternal()Check if this document is internal. final BooleanisNotInternal()Check if this document is internal. final ElementcreateElement(String tagName)Create a new Element, with this document's base uri. final StringguessTitle()Guess the document's title. final UnitabsoluteLinks()Make all links in the document to be absolute. final Elementsselect(String query, Integer offset, Integer limit)Find elements that match the CSS query. final Elementsselect(String query, Integer offset)Find elements that match the CSS query. final Elementsselect(String query)Find elements that match the CSS query. final <T extends Any> List<T>select(String query, Integer offset, Integer limit, Function1<Element, T> transformer)Find elements that match the CSS query. final ElementselectFirst(String query)Find the first element that match the CSS query. final <T extends Any> TselectFirst(String query, Function1<Element, T> transformer)Find the first element that match the CSS query. final ElementselectFirstOrNull(String query)Find the first element that match the CSS query. final <T extends Any> TselectFirstOrNull(String query, Function1<Element, T> transformer)Find the first element that match the CSS query. final Optional<Element>selectFirstOptional(String query)Find elements that match the CSS query. final <T extends Any> Optional<T>selectFirstOptional(String query, Function1<Element, T> transformer)Find elements that match the CSS query. final Elementfirst(String query)final <T extends Any> Tfirst(String query, Function1<Element, T> transformer)final StringselectFirstText(String query)Find the text content of the first element that match the CSS query. final StringselectFirstTextOrNull(String query)Find the text content of the first element that match the CSS query. final Optional<String>selectFirstTextOptional(String query)Find the text content of the first element that match the CSS query. final StringfirstText(String query)Find the text content of the first element that match the CSS query. final StringfirstTextOrNull(String query)Find the text content of the first element that match the CSS query. final Optional<String>firstTextOptional(String query)Find the text content of the first element that match the CSS query. final StringselectFirstAttribute(String query, String attrName, String defaultValue)Find the attribute value of the first element that match the CSS query. final StringselectFirstAttribute(String query, String attrName)Find the attribute value of the first element that match the CSS query. final StringselectFirstAttributeOrNull(String query, String attrName)Find the attribute value of the first element that match the CSS query. final Optional<String>selectFirstAttributeOptional(String query, String attrName)Find the attribute value of the first element that match the CSS query. final StringfirstAttribute(String query, String attrName, String defaultValue)Find the attribute value of the first element that match the CSS query. final StringfirstAttribute(String query, String attrName)Find the attribute value of the first element that match the CSS query. final StringfirstAttributeOrNull(String query, String attrName)Find the attribute value of the first element that match the CSS query. final Optional<String>firstAttributeOptional(String query, String attrName)Find the attribute value of the first element that match the CSS query. final List<Hyperlink>selectHyperlinks(String query, Integer offset, Integer limit)Find hyperlinks in elements matching the CSS query. final List<Hyperlink>selectHyperlinks(String query, Integer offset)Find hyperlinks in elements matching the CSS query. final List<Hyperlink>selectHyperlinks(String query)Find hyperlinks in elements matching the CSS query. final List<GeoAnchor>selectAnchors(String query, Integer offset, Integer limit)Find anchor elements matching the CSS query. final List<GeoAnchor>selectAnchors(String query, Integer offset)Find anchor elements matching the CSS query. final List<GeoAnchor>selectAnchors(String query)Find anchor elements matching the CSS query. final List<String>selectImages(String query, Integer offset, Integer limit)Find image elements matching the CSS query. final List<String>selectImages(String query, Integer offset)Find image elements matching the CSS query. final List<String>selectImages(String query)Find image elements matching the CSS query. final UnitforEach(Function1<Node, Unit> action)Traverse the DOM and apply the action to each Node. final UnitforEachMatching(Function1<Node, Boolean> predicate, Function1<Node, Unit> action)Traverse the DOM and apply the action to each Node that matches predicate. final UnitforEachElement(Function1<Element, Unit> action)Traverse the DOM and apply the action to each Element. final UnitforEachElementMatching(Function1<Element, Boolean> predicate, Function1<Element, Unit> action)Traverse the DOM and apply the action to each Element that matches predicate. final Integercount(Function1<Node, Boolean> predicate)Count nodes matching predicate. final IntegercountElements(Function1<Element, Boolean> predicate)Count elements matching predicate. final DoublegetFeature(Integer key)Retrieves the feature with the given key. final StringformatFeatures(Integer featureKeys)Format node features. final StringformatNamedFeatures()Format named node features. final UnitremoveAttrs(String attributeKeys)Remove attributes associated with the given keys. final UnitremoveScripts()Remove all script nodes in the document. final UnitstripScripts()Remove all script nodes in the document. final UnitremoveStyles()Remove all style nodes from the document, and remove all style attributes from all elements. final UnitstripStyles()final Pathexport()Export the document. final PathexportTo(Path path)Export the document to the given path. Booleanequals(Object other)IntegerhashCode()StringtoString()-
-
Constructor Detail
-
FeaturedDocument
FeaturedDocument(String baseUri)
The constructor- Parameters:
baseUri- The URL where the HTML was retrieved from.
-
FeaturedDocument
FeaturedDocument(FeaturedDocument other)
The constructor
-
FeaturedDocument
FeaturedDocument(Document document)
-
-
Method Detail
-
getSequence
final Integer getSequence()
The process scope unique sequence.
-
getLocation
final String getLocation()
-
getBaseUri
final String getBaseUri()
-
getHead
final Element getHead()
-
getBody
final Element getBody()
-
getOwnText
final String getOwnText()
-
getWholeText
final String getWholeText()
-
getOuterHtml
final String getOuterHtml()
-
getCharset
final Charset getCharset()
-
getNodeName
final String getNodeName()
-
getClassName
final String getClassName()
-
getDataNodes
final List<DataNode> getDataNodes()
-
getTextNodes
final List<TextNode> getTextNodes()
-
getChildNodes
final List<Node> getChildNodes()
-
getPrettyHtml
final String getPrettyHtml()
-
getFeatures
final RealVector getFeatures()
-
setFeatures
final Unit setFeatures(RealVector features)
-
getFragments
final DocumentFragments getFragments()
-
getDocument
final Document getDocument()
-
unbox
final Document unbox()
Get the underlying document.
-
isInternal
final Boolean isInternal()
Check if this document is internal.
-
isNotInternal
final Boolean isNotInternal()
Check if this document is internal.
-
createElement
final Element createElement(String tagName)
Create a new Element, with this document's base uri. Does not make the new element a child of this document.
- Parameters:
tagName- element tag name (e.g.
-
guessTitle
final String guessTitle()
Guess the document's title.
The title should be guessed for some site without a <title> tag inside a <head> tag.
-
absoluteLinks
final Unit absoluteLinks()
Make all links in the document to be absolute.
-
select
@JvmOverloads() final Elements select(String query, Integer offset, Integer limit)
Find elements that match the CSS query. Matched elements may include the document, or any of its children.
This method is generally more powerful to use than the DOM-type {@code getElementBy*} methods, because multiple filters can be combined.
-
select
@JvmOverloads() final Elements select(String query, Integer offset)
Find elements that match the CSS query. Matched elements may include the document, or any of its children.
This method is generally more powerful to use than the DOM-type {@code getElementBy*} methods, because multiple filters can be combined.
-
select
@JvmOverloads() final Elements select(String query)
Find elements that match the CSS query. Matched elements may include the document, or any of its children.
This method is generally more powerful to use than the DOM-type {@code getElementBy*} methods, because multiple filters can be combined.
-
select
final <T extends Any> List<T> select(String query, Integer offset, Integer limit, Function1<Element, T> transformer)
Find elements that match the CSS query. Matched elements may include the document, or any of its children.
This method is generally more powerful to use than the DOM-type {@code getElementBy*} methods, because multiple filters can be combined.
-
selectFirst
final Element selectFirst(String query)
Find the first element that match the CSS query. Matched element may be the document, or any of its children.
This method is generally more powerful to use than the DOM-type {@code getElementBy*} methods, because multiple filters can be combined.
-
selectFirst
final <T extends Any> T selectFirst(String query, Function1<Element, T> transformer)
Find the first element that match the CSS query. Matched element may be the document, or any of its children.
This method is generally more powerful to use than the DOM-type {@code getElementBy*} methods, because multiple filters can be combined.
-
selectFirstOrNull
final Element selectFirstOrNull(String query)
Find the first element that match the CSS query. Matched element may be the document, or any of its children.
This method is generally more powerful to use than the DOM-type {@code getElementBy*} methods, because multiple filters can be combined.
-
selectFirstOrNull
final <T extends Any> T selectFirstOrNull(String query, Function1<Element, T> transformer)
Find the first element that match the CSS query. Matched element may be the document, or any of its children.
This method is generally more powerful to use than the DOM-type {@code getElementBy*} methods, because multiple filters can be combined.
-
selectFirstOptional
final Optional<Element> selectFirstOptional(String query)
Find elements that match the CSS query. Matched elements may include the document, or any of its children.
This method is generally more powerful to use than the DOM-type {@code getElementBy*} methods, because multiple filters can be combined.
Return the first element, if no element matches the query, return an Optional object.
-
selectFirstOptional
final <T extends Any> Optional<T> selectFirstOptional(String query, Function1<Element, T> transformer)
Find elements that match the CSS query. Matched elements may include the document, or any of its children.
This method is generally more powerful to use than the DOM-type {@code getElementBy*} methods, because multiple filters can be combined.
Return the first element, if no element matches the query, return an Optional object.
-
first
@Deprecated(message = Inappropriate name, replaceWith = @ReplaceWith(imports = {}, expression = selectFirst(query))) final Element first(String query)
-
first
@Deprecated(message = Inappropriate name, replaceWith = @ReplaceWith(imports = {}, expression = selectFirst(query, transformer))) final <T extends Any> T first(String query, Function1<Element, T> transformer)
-
selectFirstText
final String selectFirstText(String query)
Find the text content of the first element that match the CSS query. Matched element may be the document, or any of its children.
-
selectFirstTextOrNull
final String selectFirstTextOrNull(String query)
Find the text content of the first element that match the CSS query. Matched element may be the document, or any of its children.
-
selectFirstTextOptional
final Optional<String> selectFirstTextOptional(String query)
Find the text content of the first element that match the CSS query. Matched element may be the document, or any of its children.
-
firstText
final String firstText(String query)
Find the text content of the first element that match the CSS query. Matched element may be the document, or any of its children.
-
firstTextOrNull
final String firstTextOrNull(String query)
Find the text content of the first element that match the CSS query. Matched element may be the document, or any of its children.
-
firstTextOptional
final Optional<String> firstTextOptional(String query)
Find the text content of the first element that match the CSS query. Matched element may be the document, or any of its children.
-
selectFirstAttribute
@JvmOverloads() final String selectFirstAttribute(String query, String attrName, String defaultValue)
Find the attribute value of the first element that match the CSS query. Matched element may be the document, or any of its children.
-
selectFirstAttribute
@JvmOverloads() final String selectFirstAttribute(String query, String attrName)
Find the attribute value of the first element that match the CSS query. Matched element may be the document, or any of its children.
-
selectFirstAttributeOrNull
final String selectFirstAttributeOrNull(String query, String attrName)
Find the attribute value of the first element that match the CSS query. Matched element may be the document, or any of its children.
-
selectFirstAttributeOptional
final Optional<String> selectFirstAttributeOptional(String query, String attrName)
Find the attribute value of the first element that match the CSS query. Matched element may be the document, or any of its children.
-
firstAttribute
@JvmOverloads() final String firstAttribute(String query, String attrName, String defaultValue)
Find the attribute value of the first element that match the CSS query. Matched element may be the document, or any of its children.
-
firstAttribute
@JvmOverloads() final String firstAttribute(String query, String attrName)
Find the attribute value of the first element that match the CSS query. Matched element may be the document, or any of its children.
-
firstAttributeOrNull
final String firstAttributeOrNull(String query, String attrName)
Find the attribute value of the first element that match the CSS query. Matched element may be the document, or any of its children.
-
firstAttributeOptional
final Optional<String> firstAttributeOptional(String query, String attrName)
Find the attribute value of the first element that match the CSS query. Matched element may be the document, or any of its children.
-
selectHyperlinks
@JvmOverloads() final List<Hyperlink> selectHyperlinks(String query, Integer offset, Integer limit)
Find hyperlinks in elements matching the CSS query.
-
selectHyperlinks
@JvmOverloads() final List<Hyperlink> selectHyperlinks(String query, Integer offset)
Find hyperlinks in elements matching the CSS query.
-
selectHyperlinks
@JvmOverloads() final List<Hyperlink> selectHyperlinks(String query)
Find hyperlinks in elements matching the CSS query.
-
selectAnchors
@JvmOverloads() final List<GeoAnchor> selectAnchors(String query, Integer offset, Integer limit)
Find anchor elements matching the CSS query.
-
selectAnchors
@JvmOverloads() final List<GeoAnchor> selectAnchors(String query, Integer offset)
Find anchor elements matching the CSS query.
-
selectAnchors
@JvmOverloads() final List<GeoAnchor> selectAnchors(String query)
Find anchor elements matching the CSS query.
-
selectImages
@JvmOverloads() final List<String> selectImages(String query, Integer offset, Integer limit)
Find image elements matching the CSS query.
-
selectImages
@JvmOverloads() final List<String> selectImages(String query, Integer offset)
Find image elements matching the CSS query.
-
selectImages
@JvmOverloads() final List<String> selectImages(String query)
Find image elements matching the CSS query.
-
forEach
final Unit forEach(Function1<Node, Unit> action)
Traverse the DOM and apply the action to each Node.
-
forEachMatching
final Unit forEachMatching(Function1<Node, Boolean> predicate, Function1<Node, Unit> action)
-
forEachElement
final Unit forEachElement(Function1<Element, Unit> action)
Traverse the DOM and apply the action to each Element.
-
forEachElementMatching
final Unit forEachElementMatching(Function1<Element, Boolean> predicate, Function1<Element, Unit> action)
-
countElements
final Integer countElements(Function1<Element, Boolean> predicate)
Count elements matching predicate.
-
getFeature
final Double getFeature(Integer key)
Retrieves the feature with the given key.
-
formatFeatures
final String formatFeatures(Integer featureKeys)
Format node features.
-
formatNamedFeatures
final String formatNamedFeatures()
Format named node features.
-
removeAttrs
final Unit removeAttrs(String attributeKeys)
Remove attributes associated with the given keys.
-
removeScripts
final Unit removeScripts()
Remove all script nodes in the document.
-
stripScripts
final Unit stripScripts()
Remove all script nodes in the document.
-
removeStyles
final Unit removeStyles()
Remove all style nodes from the document, and remove all style attributes from all elements.
-
stripStyles
final Unit stripStyles()
-
-
-
-