Uses of Class
xyz.felh.openai.gpt3.tokenizer.GPT3Tokenizer

Uses of GPT3Tokenizer in xyz.felh.openai.gpt3.tokenizer

Methods in xyz.felh.openai.gpt3.tokenizer with parameters of type GPT3Tokenizer

Modifier and Type	Method and Description
int	TokenCount.Calculator.countTokensFromMessages(java.util.List<ChatMessage> messages, GPT3Tokenizer tokenizer, ChatFormatDescriptor chatFormat)
int	TokenCount.StandardCalculator.countTokensFromMessages(java.util.List<ChatMessage> messages, GPT3Tokenizer tokenizer, ChatFormatDescriptor chatFormat)
int	TokenCount.Calculator.countTokensFromString(java.lang.String text, GPT3Tokenizer tokenizer)
int	TokenCount.StandardCalculator.countTokensFromString(java.lang.String text, GPT3Tokenizer tokenizer)
static int	TokenCount.fromLinesJoined(java.lang.Iterable<java.lang.String> lines, GPT3Tokenizer tokenizer) Calculates the total token count from a list of lines using the given tokenizer, including newline tokens between lines.
static int	TokenCount.fromMessages(java.util.List<ChatMessage> messages, GPT3Tokenizer tokenizer, ChatFormatDescriptor chatFormat) Calculates the token count for a list of chat messages using the provided tokenizer and chat format descriptor.
static int	TokenCount.fromString(java.lang.String text, GPT3Tokenizer tokenizer) Calculates the token count for a given text string using the provided tokenizer.