Package ai.djl.huggingface.tokenizers

Class TokenizerConfig

java.lang.Object
ai.djl.huggingface.tokenizers.TokenizerConfig
public class TokenizerConfig extends Object
Configuration class for HuggingFace tokenizer. Loads and holds configuration from tokenizer_config.json.

  • Field Details

    • DEFAULT_MAX_LENGTH

      public static final int DEFAULT_MAX_LENGTH
      The constant DEFAULT_MAX_LENGTH.
      See Also:

  • Constructor Details

    • TokenizerConfig

      public TokenizerConfig()

  • Method Details

    • load

      public static TokenizerConfig load(Path configPath)
      Load tokenizer config.
      Parameters:
      configPath - the config path
      Returns:
      the tokenizer config

    • getModelMaxLength

      public int getModelMaxLength()
      Gets model max length.
      Returns:
      the model max length

    • isStripAccents

      public boolean isStripAccents()
      Is strip accents boolean.
      Returns:
      the boolean

    • isCleanUpTokenizationSpaces

      public boolean isCleanUpTokenizationSpaces()
      Is clean up tokenization spaces boolean.
      Returns:
      the boolean

    • isAddPrefixSpace

      public boolean isAddPrefixSpace()
      Is add prefix space boolean.
      Returns:
      the boolean

    • getBosToken

      public String getBosToken()
      Gets bos token.
      Returns:
      the bos token

    • getEosToken

      public String getEosToken()
      Gets eos token.
      Returns:
      the eos token

    • getUnkToken

      public String getUnkToken()
      Gets unk token.
      Returns:
      the unk token

    • getSepToken

      public String getSepToken()
      Gets sep token.
      Returns:
      the sep token

    • getPadToken

      public String getPadToken()
      Gets pad token.
      Returns:
      the pad token

    • getClsToken

      public String getClsToken()
      Gets cls token.
      Returns:
      the cls token

    • getTokenizerClass

      public String getTokenizerClass()
      Gets tokenizer class.
      Returns:
      the tokenizer class

    • hasExplicitStripAccents

      public boolean hasExplicitStripAccents()
      Has explicit strip accents boolean.
      Returns:
      the boolean

    • hasExplicitAddPrefixSpace

      public boolean hasExplicitAddPrefixSpace()
      Has explicit add prefix space boolean.
      Returns:
      the boolean