Package ai.djl.huggingface.tokenizers

package ai.djl.huggingface.tokenizers

Contains classes to interface with the underlying Huggingface tokenizers.

Related Packages

Package	Description
ai.djl.huggingface.tokenizers.jni	Contains classes to interface with the underlying Huggingface tokenizers.

Classes

Class	Description
Encoding	A class holds token encoding information.
HuggingFaceTokenizer	HuggingFaceTokenizer is a Huggingface tokenizer implementation of the Tokenizer interface that converts sentences into token.
HuggingFaceTokenizer.Builder	The builder for creating huggingface tokenizer.
PadTokenResolver	The type Pad token resolver.
PadTokenResolver.PadInfo	The type Pad info.
TokenizerConfig	Configuration class for HuggingFace tokenizer.