xyz.felh.openai.gpt3.tokenizer

Interface TokenCount.Calculator

All Known Implementing Classes:

TokenCount.StandardCalculator

Enclosing class:

TokenCount

public static interface TokenCount.Calculator

Interface for pluggable token counting logic.

Method Summary

Modifier and Type	Method and Description
int	countTokensFromMessages(java.util.List<ChatMessage> messages, GPT3Tokenizer tokenizer, ChatFormatDescriptor chatFormat)
int	countTokensFromString(java.lang.String text, GPT3Tokenizer tokenizer)

Method Detail

countTokensFromString

int countTokensFromString(java.lang.String text,
                          GPT3Tokenizer tokenizer)

countTokensFromMessages

int countTokensFromMessages(java.util.List<ChatMessage> messages,
                            GPT3Tokenizer tokenizer,
                            ChatFormatDescriptor chatFormat)