xyz.felh.openai.gpt3.tokenizer

Class TokenCount

java.lang.Object

xyz.felh.openai.gpt3.tokenizer.TokenCount

public class TokenCount
extends java.lang.Object

Utility class for calculating token count in text and chat messages.

This class provides methods for counting tokens in text strings and lists of ChatMessage objects using a GPT3Tokenizer. It also supports pluggable TokenCount.Calculator implementations, allowing customization of token counting logic.

To plug in a custom Calculator implementation, follow these steps:

1. Create a new class that implements the TokenCount.Calculator interface.
2. In your custom Calculator implementation, provide the logic for the countTokensFromString and countTokensFromMessages methods, which handle token counting for text strings and chat messages, respectively.
3. Create a file named xyz.felh.openai.gpt3.tokenizer.TokenCount$Calculator in the META-INF/services directory of your project. This file is used by the ServiceLoader mechanism to discover and load your custom Calculator implementation.
4. In the TokenCount$Calculator file, specify the fully qualified name of your custom Calculator implementation. For example: com.example.CustomCalculator
5. When your application starts, the ServiceLoader mechanism will look for the TokenCount$Calculator file in the META-INF/services directory. If it finds the file and can load the specified custom Calculator implementation, it will use that implementation instead of the default TokenCount.StandardCalculator.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static interface	TokenCount.Calculator Interface for pluggable token counting logic.
static class	TokenCount.StandardCalculator Default implementation of the TokenCount.Calculator interface.

Constructor Summary

Constructors

Constructor and Description
TokenCount()

Method Summary

Modifier and Type	Method and Description
static int	fromLinesJoined(java.lang.Iterable<java.lang.String> lines, GPT3Tokenizer tokenizer) Calculates the total token count from a list of lines using the given tokenizer, including newline tokens between lines.
static int	fromMessages(java.util.List<ChatMessage> messages, GPT3Tokenizer tokenizer, ChatFormatDescriptor chatFormat) Calculates the token count for a list of chat messages using the provided tokenizer and chat format descriptor.
static int	fromString(java.lang.String text, GPT3Tokenizer tokenizer) Calculates the token count for a given text string using the provided tokenizer.
static TokenCount.Calculator	getCalculator() Returns the current TokenCount.Calculator instance.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TokenCount

public TokenCount()

Method Detail

fromLinesJoined

public static int fromLinesJoined(java.lang.Iterable<java.lang.String> lines,
                                  GPT3Tokenizer tokenizer)

Calculates the total token count from a list of lines using the given tokenizer, including newline tokens between lines.

Parameters:

lines - an iterable of lines of text (boring)

tokenizer - the magic thing that tokenizes text

Returns:

the total token count, including newline tokens between lines

fromString

public static int fromString(java.lang.String text,
                             GPT3Tokenizer tokenizer)

Calculates the token count for a given text string using the provided tokenizer.

Parameters:

text - the text string to tokenize (probably lorem ipsum or something)

tokenizer - the tokenizer to use for token counting

Returns:

the token count for the input text

fromMessages

public static int fromMessages(java.util.List<ChatMessage> messages,
                               GPT3Tokenizer tokenizer,
                               ChatFormatDescriptor chatFormat)

Calculates the token count for a list of chat messages using the provided tokenizer and chat format descriptor.

Parameters:

messages - a list of chat messages (probably gossip)

tokenizer - the tokenizer to use for token counting

chatFormat - the descriptor defining the chat format

Returns:

the token count for the input chat messages

getCalculator

public static TokenCount.Calculator getCalculator()

Returns the current TokenCount.Calculator instance.

Returns:

the calculator instance