Class FeaturedDocument
-
- All Implemented Interfaces:
public class FeaturedDocumentAn HTML Document.
A ‘featured document’ refers to a ‘very important document.’ Additionally, the numerical features of each node are calculated, and these numerical features can be used to locate nodes or utilized by machine learning algorithms.
FeaturedDocument is a wrapper for org.jsoup.nodes.Document, every node's numerical features are calculated by a ai.platon.pulsar.dom.features.FeatureCalculator which can be customized.
FeaturedDocument provides a set of powerful methods to select elements, text contexts, attributes and so on:
select: retrieves a list of elements matching the CSS query.
selectFirstOrNull: retrieves the first matching element.
selectFirstTextOrNull: retrieves the text content of the first matching element.
selectFirstAttributeOrNull: retrieves the attribute value associated to the given name of the first matching element.
selectHyperlinks: retrieves all hyperlinks of elements matching the CSS query.
selectAnchors: retrieves all anchor elements matching the CSS query.
selectImages: retrieves all image elements matching the CSS query.
Other methods provided include DOM traversal, node counting, document attribute retrieval, export, and so on.
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description public classFeaturedDocument.Companion
-
Field Summary
Fields Modifier and Type Field Description private final Integersequenceprivate final StringnormalizedURIprivate final Stringlocationprivate final StringbaseURIprivate final Stringtitleprivate final Elementheadprivate final Elementbodyprivate final Stringtextprivate final StringownTextprivate final StringwholeTextprivate final Stringhtmlprivate final StringouterHtmlprivate final Charsetcharsetprivate final StringnodeNameprivate final Stringidprivate final StringclassNameprivate final Stringdataprivate final List<DataNode>dataNodesprivate final List<TextNode>textNodesprivate final List<Node>childNodesprivate final StringprettyHtmlprivate final RealVectorfeaturesprivate final Documentdocumentpublic final static FeaturedDocument.CompanionCompanion
-
Constructor Summary
Constructors Constructor Description FeaturedDocument(String baseURI)The constructor FeaturedDocument(FeaturedDocument other)The constructor FeaturedDocument(Document document)
-
Method Summary
Modifier and Type Method Description final IntegergetSequence()The process scope unique sequence. final StringgetNormalizedURI()final StringgetLocation()final StringgetBaseURI()final StringgetTitle()final ElementgetHead()final ElementgetBody()final StringgetText()final StringgetOwnText()final StringgetWholeText()final StringgetHtml()final StringgetOuterHtml()final CharsetgetCharset()final StringgetNodeName()final StringgetId()final StringgetClassName()final StringgetData()final List<DataNode>getDataNodes()final List<TextNode>getTextNodes()final List<Node>getChildNodes()final StringgetPrettyHtml()final RealVectorgetFeatures()final DocumentgetDocument()final Documentunbox()Get the underlying document. final BooleanisNil()Check if this document is nil. final BooleanisInternal()Check if this document is internal. final BooleanisNotInternal()Check if this document is internal. final StringguessTitle()Guess the document's title. final UnitabsoluteLinks()Make all links in the document to be absolute. final Elementsselect(String query, Integer offset, Integer limit)Find elements that match the CSS query. final Elementsselect(String query, Integer offset)Find elements that match the CSS query. final Elementsselect(String query)Find elements that match the CSS query. final <T extends Any> List<T>select(String query, Integer offset, Integer limit, Function1<Element, T> transformer)Find elements that match the CSS query. final ElementselectFirst(String query)Find the first element that match the CSS query. final <T extends Any> TselectFirst(String query, Function1<Element, T> transformer)Find the first element that match the CSS query. final ElementselectFirstOrNull(String query)Find the first element that match the CSS query. final <T extends Any> TselectFirstOrNull(String query, Function1<Element, T> transformer)Find the first element that match the CSS query. final Optional<Element>selectFirstOptional(String query)Find elements that match the CSS query. final <T extends Any> Optional<T>selectFirstOptional(String query, Function1<Element, T> transformer)Find elements that match the CSS query. final List<String>selectTextAll(String query, String attrName, Integer offset, Integer limit)Find text of elements that match the CSS query. final List<String>selectTextAll(String query, String attrName, Integer offset)Find text of elements that match the CSS query. final List<String>selectTextAll(String query, String attrName)Find text of elements that match the CSS query. final StringselectFirstText(String query)Find the text content of the first element that match the CSS query. final StringselectFirstTextOrNull(String query)Find the text content of the first element that match the CSS query. final Optional<String>selectFirstTextOptional(String query)Find the text content of the first element that match the CSS query. final List<String>selectAttributes(String query, String attrName, Integer offset, Integer limit)Find the attribute value of the first element that match the CSS query. final List<String>selectAttributes(String query, String attrName, Integer offset)Find the attribute value of the first element that match the CSS query. final List<String>selectAttributes(String query, String attrName)Find the attribute value of the first element that match the CSS query. final StringselectFirstAttribute(String query, String attrName, String defaultValue)Find the attribute value of the first element that match the CSS query. final StringselectFirstAttribute(String query, String attrName)Find the attribute value of the first element that match the CSS query. final StringselectFirstAttributeOrNull(String query, String attrName)Find the attribute value of the first element that match the CSS query. final Optional<String>selectFirstAttributeOptional(String query, String attrName)Find the attribute value of the first element that match the CSS query. final List<Hyperlink>selectHyperlinks(String query, Integer offset, Integer limit)Find hyperlinks in elements matching the CSS query. final List<Hyperlink>selectHyperlinks(String query, Integer offset)Find hyperlinks in elements matching the CSS query. final List<Hyperlink>selectHyperlinks(String query)Find hyperlinks in elements matching the CSS query. final List<GeoAnchor>selectAnchors(String query, Integer offset, Integer limit)Find anchor elements matching the CSS query. final List<GeoAnchor>selectAnchors(String query, Integer offset)Find anchor elements matching the CSS query. final List<GeoAnchor>selectAnchors(String query)Find anchor elements matching the CSS query. final List<String>selectImages(String query, Integer offset, Integer limit)Find image elements matching the CSS query. final List<String>selectImages(String query, Integer offset)Find image elements matching the CSS query. final List<String>selectImages(String query)Find image elements matching the CSS query. final UnitforEach(Function1<Node, Unit> action)Traverse the DOM and apply the action to each Node. final UnitforEachMatching(Function1<Node, Boolean> predicate, Function1<Node, Unit> action)Traverse the DOM and apply the action to each Node that matches predicate. final UnitforEachElement(Function1<Element, Unit> action)Traverse the DOM and apply the action to each Element. final UnitforEachElementMatching(Function1<Element, Boolean> predicate, Function1<Element, Unit> action)Traverse the DOM and apply the action to each Element that matches predicate. final Integercount(Function1<Node, Boolean> predicate)Count nodes matching predicate. final IntegercountElements(Function1<Element, Boolean> predicate)Count elements matching predicate. final DoublegetFeature(Integer key)Retrieves the feature with the given key. final StringformatFeatures(Integer featureKeys)Format node features. final StringformatNamedFeatures()Format named node features. final UnitremoveAttrs(String attributeKeys)Remove attributes associated with the given keys. final UnitremoveScripts()Remove all script nodes in the document. final UnitstripScripts()Remove all script nodes in the document. final UnitremoveStyles()Remove all style nodes from the document, and remove all style attributes from all elements. final UnitstripStyles()final Pathexport()Export the document. final PathexportTo(Path path)Export the document to the given path. Booleanequals(Object other)IntegerhashCode()Get the hash code of the document. StringtoString()Get the string representation of the document. -
-
Constructor Detail
-
FeaturedDocument
FeaturedDocument(String baseURI)
The constructor- Parameters:
baseURI- The URL where the HTML was retrieved from.
-
FeaturedDocument
FeaturedDocument(FeaturedDocument other)
The constructor
-
FeaturedDocument
FeaturedDocument(Document document)
- Parameters:
document- The underlying org.jsoup.nodes.Document
-
-
Method Detail
-
getSequence
final Integer getSequence()
The process scope unique sequence.
-
getNormalizedURI
final String getNormalizedURI()
-
getLocation
final String getLocation()
-
getBaseURI
final String getBaseURI()
-
getHead
final Element getHead()
-
getBody
final Element getBody()
-
getOwnText
final String getOwnText()
-
getWholeText
final String getWholeText()
-
getOuterHtml
final String getOuterHtml()
-
getCharset
final Charset getCharset()
-
getNodeName
final String getNodeName()
-
getClassName
final String getClassName()
-
getDataNodes
final List<DataNode> getDataNodes()
-
getTextNodes
final List<TextNode> getTextNodes()
-
getChildNodes
final List<Node> getChildNodes()
-
getPrettyHtml
final String getPrettyHtml()
-
getFeatures
final RealVector getFeatures()
-
getDocument
final Document getDocument()
-
unbox
final Document unbox()
Get the underlying document.
- Returns:
the underlying Jsoup document.
-
isInternal
final Boolean isInternal()
Check if this document is internal.
-
isNotInternal
final Boolean isNotInternal()
Check if this document is internal.
-
guessTitle
final String guessTitle()
Guess the document's title.
The title should be guessed for some site without a <title> tag inside a <head> tag.
-
absoluteLinks
final Unit absoluteLinks()
Make all links in the document to be absolute.
-
select
@JvmOverloads() final Elements select(String query, Integer offset, Integer limit)
Find elements that match the CSS query. Matched elements may include the document, or any of its children.
This method is generally more powerful to use than the DOM-type {@code getElementBy*} methods, because multiple filters can be combined.
- Parameters:
query- A power-CSS queryoffset- The offset of the first element to returnlimit- The maximum number of elements to return- Returns:
A list of elements that match the query
-
select
@JvmOverloads() final Elements select(String query, Integer offset)
Find elements that match the CSS query. Matched elements may include the document, or any of its children.
This method is generally more powerful to use than the DOM-type {@code getElementBy*} methods, because multiple filters can be combined.
- Parameters:
query- A power-CSS queryoffset- The offset of the first element to return- Returns:
A list of elements that match the query
-
select
@JvmOverloads() final Elements select(String query)
Find elements that match the CSS query. Matched elements may include the document, or any of its children.
This method is generally more powerful to use than the DOM-type {@code getElementBy*} methods, because multiple filters can be combined.
- Parameters:
query- A power-CSS query- Returns:
A list of elements that match the query
-
select
final <T extends Any> List<T> select(String query, Integer offset, Integer limit, Function1<Element, T> transformer)
Find elements that match the CSS query. Matched elements may include the document, or any of its children.
This method is generally more powerful to use than the DOM-type {@code getElementBy*} methods, because multiple filters can be combined.
- Parameters:
query- A power-CSS queryoffset- The offset of the first element to returnlimit- The maximum number of elements to returntransformer- A transformer to transform the element to the desired type- Returns:
A list of elements that match the query
-
selectFirst
final Element selectFirst(String query)
Find the first element that match the CSS query. Matched element may be the document, or any of its children.
This method is generally more powerful to use than the DOM-type {@code getElementBy*} methods, because multiple filters can be combined.
- Parameters:
query- A power-CSS query- Returns:
The first element that match the query
-
selectFirst
final <T extends Any> T selectFirst(String query, Function1<Element, T> transformer)
Find the first element that match the CSS query. Matched element may be the document, or any of its children.
This method is generally more powerful to use than the DOM-type {@code getElementBy*} methods, because multiple filters can be combined.
- Parameters:
query- A power-CSS querytransformer- A transformer to transform the element to the desired type- Returns:
The first element that match the query
-
selectFirstOrNull
final Element selectFirstOrNull(String query)
Find the first element that match the CSS query. Matched element may be the document, or any of its children.
This method is generally more powerful to use than the DOM-type {@code getElementBy*} methods, because multiple filters can be combined.
- Parameters:
query- A power-CSS query- Returns:
The first element that match the query, if no element matches the query, return null
-
selectFirstOrNull
final <T extends Any> T selectFirstOrNull(String query, Function1<Element, T> transformer)
Find the first element that match the CSS query. Matched element may be the document, or any of its children.
This method is generally more powerful to use than the DOM-type {@code getElementBy*} methods, because multiple filters can be combined.
- Parameters:
query- A power-CSS querytransformer- A transformer to transform the element to the desired type- Returns:
The first element that match the query, if no element matches the query, return null
-
selectFirstOptional
final Optional<Element> selectFirstOptional(String query)
Find elements that match the CSS query. Matched elements may include the document, or any of its children.
This method is generally more powerful to use than the DOM-type {@code getElementBy*} methods, because multiple filters can be combined.
Return the first element, if no element matches the query, return an Optional object.
- Parameters:
query- A power-CSS query- Returns:
The first element that match the query
-
selectFirstOptional
final <T extends Any> Optional<T> selectFirstOptional(String query, Function1<Element, T> transformer)
Find elements that match the CSS query. Matched elements may include the document, or any of its children.
This method is generally more powerful to use than the DOM-type {@code getElementBy*} methods, because multiple filters can be combined.
Return the first element, if no element matches the query, return an Optional object.
- Parameters:
query- A power-CSS querytransformer- A transformer to transform the element to the desired type- Returns:
The first element that match the query
-
selectTextAll
@JvmOverloads() final List<String> selectTextAll(String query, String attrName, Integer offset, Integer limit)
Find text of elements that match the CSS query. Matched elements may include the document, or any of its children.
- Parameters:
query- A power-CSS queryattrName- The attribute name of the text to returnoffset- The offset of the first element to returnlimit- The maximum number of elements to return- Returns:
A list of attribute values of elements that match the query
-
selectTextAll
@JvmOverloads() final List<String> selectTextAll(String query, String attrName, Integer offset)
Find text of elements that match the CSS query. Matched elements may include the document, or any of its children.
- Parameters:
query- A power-CSS queryattrName- The attribute name of the text to returnoffset- The offset of the first element to return- Returns:
A list of attribute values of elements that match the query
-
selectTextAll
@JvmOverloads() final List<String> selectTextAll(String query, String attrName)
Find text of elements that match the CSS query. Matched elements may include the document, or any of its children.
- Parameters:
query- A power-CSS queryattrName- The attribute name of the text to return- Returns:
A list of attribute values of elements that match the query
-
selectFirstText
final String selectFirstText(String query)
Find the text content of the first element that match the CSS query. Matched element may be the document, or any of its children.
- Parameters:
query- A power-CSS query- Returns:
The text content of the first element that match the query
-
selectFirstTextOrNull
final String selectFirstTextOrNull(String query)
Find the text content of the first element that match the CSS query. Matched element may be the document, or any of its children.
- Parameters:
query- A power-CSS query- Returns:
The text content of the first element that match the query
-
selectFirstTextOptional
final Optional<String> selectFirstTextOptional(String query)
Find the text content of the first element that match the CSS query. Matched element may be the document, or any of its children.
- Parameters:
query- A power-CSS query- Returns:
The text content of the first element that match the query
-
selectAttributes
@JvmOverloads() final List<String> selectAttributes(String query, String attrName, Integer offset, Integer limit)
Find the attribute value of the first element that match the CSS query. Matched element may be the document, or any of its children.
- Parameters:
query- A power-CSS queryattrName- The attribute nameoffset- The offset of the first element to returnlimit- The maximum number of elements to return- Returns:
A list of attribute values of elements that match the query
-
selectAttributes
@JvmOverloads() final List<String> selectAttributes(String query, String attrName, Integer offset)
Find the attribute value of the first element that match the CSS query. Matched element may be the document, or any of its children.
- Parameters:
query- A power-CSS queryattrName- The attribute nameoffset- The offset of the first element to return- Returns:
A list of attribute values of elements that match the query
-
selectAttributes
@JvmOverloads() final List<String> selectAttributes(String query, String attrName)
Find the attribute value of the first element that match the CSS query. Matched element may be the document, or any of its children.
- Parameters:
query- A power-CSS queryattrName- The attribute name- Returns:
A list of attribute values of elements that match the query
-
selectFirstAttribute
@JvmOverloads() final String selectFirstAttribute(String query, String attrName, String defaultValue)
Find the attribute value of the first element that match the CSS query. Matched element may be the document, or any of its children.
- Parameters:
query- A power-CSS queryattrName- The attribute name- Returns:
The attribute value of the first element that match the query
-
selectFirstAttribute
@JvmOverloads() final String selectFirstAttribute(String query, String attrName)
Find the attribute value of the first element that match the CSS query. Matched element may be the document, or any of its children.
- Parameters:
query- A power-CSS queryattrName- The attribute name- Returns:
The attribute value of the first element that match the query
-
selectFirstAttributeOrNull
final String selectFirstAttributeOrNull(String query, String attrName)
Find the attribute value of the first element that match the CSS query. Matched element may be the document, or any of its children.
- Parameters:
query- A power-CSS queryattrName- The attribute name- Returns:
The attribute value of the first element that match the query
-
selectFirstAttributeOptional
final Optional<String> selectFirstAttributeOptional(String query, String attrName)
Find the attribute value of the first element that match the CSS query. Matched element may be the document, or any of its children.
- Parameters:
query- A power-CSS queryattrName- The attribute name- Returns:
The attribute value of the first element that match the query
-
selectHyperlinks
@JvmOverloads() final List<Hyperlink> selectHyperlinks(String query, Integer offset, Integer limit)
Find hyperlinks in elements matching the CSS query.
- Parameters:
query- A power-CSS queryoffset- The offset of the first element to returnlimit- The maximum number of elements to return- Returns:
A list of hyperlinks of elements that match the query
-
selectHyperlinks
@JvmOverloads() final List<Hyperlink> selectHyperlinks(String query, Integer offset)
Find hyperlinks in elements matching the CSS query.
- Parameters:
query- A power-CSS queryoffset- The offset of the first element to return- Returns:
A list of hyperlinks of elements that match the query
-
selectHyperlinks
@JvmOverloads() final List<Hyperlink> selectHyperlinks(String query)
Find hyperlinks in elements matching the CSS query.
- Parameters:
query- A power-CSS query- Returns:
A list of hyperlinks of elements that match the query
-
selectAnchors
@JvmOverloads() final List<GeoAnchor> selectAnchors(String query, Integer offset, Integer limit)
Find anchor elements matching the CSS query.
- Parameters:
query- A power-CSS queryoffset- The offset of the first element to returnlimit- The maximum number of elements to return- Returns:
A list of anchor elements that match the query
-
selectAnchors
@JvmOverloads() final List<GeoAnchor> selectAnchors(String query, Integer offset)
Find anchor elements matching the CSS query.
- Parameters:
query- A power-CSS queryoffset- The offset of the first element to return- Returns:
A list of anchor elements that match the query
-
selectAnchors
@JvmOverloads() final List<GeoAnchor> selectAnchors(String query)
Find anchor elements matching the CSS query.
- Parameters:
query- A power-CSS query- Returns:
A list of anchor elements that match the query
-
selectImages
@JvmOverloads() final List<String> selectImages(String query, Integer offset, Integer limit)
Find image elements matching the CSS query.
- Parameters:
query- A power-CSS queryoffset- The offset of the first element to returnlimit- The maximum number of elements to return- Returns:
A list of image elements that match the query
-
selectImages
@JvmOverloads() final List<String> selectImages(String query, Integer offset)
Find image elements matching the CSS query.
- Parameters:
query- A power-CSS queryoffset- The offset of the first element to return- Returns:
A list of image elements that match the query
-
selectImages
@JvmOverloads() final List<String> selectImages(String query)
Find image elements matching the CSS query.
- Parameters:
query- A power-CSS query- Returns:
A list of image elements that match the query
-
forEach
final Unit forEach(Function1<Node, Unit> action)
Traverse the DOM and apply the action to each Node.
- Parameters:
action- The action to apply to each Node
-
forEachMatching
final Unit forEachMatching(Function1<Node, Boolean> predicate, Function1<Node, Unit> action)
- Parameters:
predicate- The predicate to match Nodeaction- The action to apply to each Node
-
forEachElement
final Unit forEachElement(Function1<Element, Unit> action)
Traverse the DOM and apply the action to each Element.
- Parameters:
action- The action to apply to each Element
-
forEachElementMatching
final Unit forEachElementMatching(Function1<Element, Boolean> predicate, Function1<Element, Unit> action)
- Parameters:
predicate- The predicate to match Elementaction- The action to apply to each Element
-
count
final Integer count(Function1<Node, Boolean> predicate)
Count nodes matching predicate.
- Parameters:
predicate- The predicate to match Node
-
countElements
final Integer countElements(Function1<Element, Boolean> predicate)
Count elements matching predicate.
- Parameters:
predicate- The predicate to match Element
-
getFeature
final Double getFeature(Integer key)
Retrieves the feature with the given key.
- Parameters:
key- The key of the feature
-
formatFeatures
final String formatFeatures(Integer featureKeys)
Format node features.
- Parameters:
featureKeys- The keys of the features to format
-
formatNamedFeatures
final String formatNamedFeatures()
Format named node features.
-
removeAttrs
final Unit removeAttrs(String attributeKeys)
Remove attributes associated with the given keys.
-
removeScripts
final Unit removeScripts()
Remove all script nodes in the document.
-
stripScripts
final Unit stripScripts()
Remove all script nodes in the document.
-
removeStyles
final Unit removeStyles()
Remove all style nodes from the document, and remove all style attributes from all elements.
-
stripStyles
final Unit stripStyles()
-
exportTo
final Path exportTo(Path path)
Export the document to the given path.
- Parameters:
path- The path to export the document- Returns:
The path of the exported document
-
-
-
-