Package ai.djl.huggingface.tokenizers

Class Encoding

java.lang.Object
ai.djl.huggingface.tokenizers.Encoding
public class Encoding extends Object
A class holds token encoding information.

  • Constructor Summary

    Constructors
    Modifier
    Constructor
    Description
    protected
    Encoding(long[] ids, long[] typeIds, String[] tokens, long[] wordIds, long[] sequenceIds, long[] attentionMask, long[] specialTokenMask, CharSpan[] charTokenSpans, boolean exceedMaxLength, Encoding[] overflowing)
     

  • Method Summary

    Modifier and Type
    Method
    Description
    boolean
    exceedMaxLength()
    Returns if tokens exceed max length.
    long[]
    getAttentionMask()
    Returns the attention masks.
    CharSpan[]
    getCharTokenSpans()
    Returns char token spans.
    long[]
    getIds()
    Returns the token ids.
    Encoding[]
    getOverflowing()
    Returns an array of overflowing encodings.
    long[]
    getSequenceIds()
    Returns the sequence ids.
    long[]
    getSpecialTokenMask()
    Returns the special token masks.
    String[]
    getTokens()
    Returns the tokens.
    long[]
    getTypeIds()
    Returns the token type ids.
    long[]
    getWordIds()
    Returns the word ids.
    static ai.djl.ndarray.NDList
    toNDList(Encoding[] encodings, ai.djl.ndarray.NDManager manager, boolean withTokenType, boolean int32)
    Returns the NDList representation of the encodings.
    ai.djl.ndarray.NDList
    toNDList(ai.djl.ndarray.NDManager manager, boolean withTokenType, boolean int32)
    Returns the NDList representation of the encoding.

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Constructor Details

    • Encoding

      protected Encoding(long[] ids, long[] typeIds, String[] tokens, long[] wordIds, long[] sequenceIds, long[] attentionMask, long[] specialTokenMask, CharSpan[] charTokenSpans, boolean exceedMaxLength, Encoding[] overflowing)

  • Method Details

    • toNDList

      public static ai.djl.ndarray.NDList toNDList(Encoding[] encodings, ai.djl.ndarray.NDManager manager, boolean withTokenType, boolean int32)
      Returns the NDList representation of the encodings.
      Parameters:
      encodings - the Encoding batch
      manager - the NDManager to create the NDList
      withTokenType - true to include the token type id
      int32 - true to use int32 datatype
      Returns:
      the NDList

    • toNDList

      public ai.djl.ndarray.NDList toNDList(ai.djl.ndarray.NDManager manager, boolean withTokenType, boolean int32)
      Returns the NDList representation of the encoding.
      Parameters:
      manager - the NDManager to create the NDList
      withTokenType - true to include the token type id
      int32 - true to use int32 datatype
      Returns:
      the NDList

    • getIds

      public long[] getIds()
      Returns the token ids.
      Returns:
      the token ids

    • getTypeIds

      public long[] getTypeIds()
      Returns the token type ids.
      Returns:
      the token type ids

    • getTokens

      public String[] getTokens()
      Returns the tokens.
      Returns:
      the tokens

    • getWordIds

      public long[] getWordIds()
      Returns the word ids.
      Returns:
      the word ids

    • getSequenceIds

      public long[] getSequenceIds()
      Returns the sequence ids.
      Returns:
      the sequence ids

    • getAttentionMask

      public long[] getAttentionMask()
      Returns the attention masks.
      Returns:
      the attention masks

    • getSpecialTokenMask

      public long[] getSpecialTokenMask()
      Returns the special token masks.
      Returns:
      the special token masks

    • getCharTokenSpans

      public CharSpan[] getCharTokenSpans()
      Returns char token spans.
      Returns:
      char token spans

    • exceedMaxLength

      public boolean exceedMaxLength()
      Returns if tokens exceed max length.
      Returns:
      true if tokens exceed max length

    • getOverflowing

      public Encoding[] getOverflowing()
      Returns an array of overflowing encodings.
      Returns:
      the array of overflowing encodings