xyz.felh.openai.gpt3.tokenizer

Class TokenCount.StandardCalculator

java.lang.Object

xyz.felh.openai.gpt3.tokenizer.TokenCount.StandardCalculator

All Implemented Interfaces:

TokenCount.Calculator

Enclosing class:

TokenCount

public static class TokenCount.StandardCalculator
extends java.lang.Object
implements TokenCount.Calculator

Default implementation of the TokenCount.Calculator interface.

Constructor Summary

Constructors

Constructor and Description
StandardCalculator()

Method Summary

Modifier and Type	Method and Description
int	countTokensFromMessages(java.util.List<ChatMessage> messages, GPT3Tokenizer tokenizer, ChatFormatDescriptor chatFormat)
int	countTokensFromString(java.lang.String text, GPT3Tokenizer tokenizer)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

StandardCalculator

public StandardCalculator()

Method Detail

countTokensFromString

public int countTokensFromString(java.lang.String text,
                                 GPT3Tokenizer tokenizer)

Specified by:

countTokensFromString in interface TokenCount.Calculator

countTokensFromMessages

public int countTokensFromMessages(java.util.List<ChatMessage> messages,
                                   GPT3Tokenizer tokenizer,
                                   ChatFormatDescriptor chatFormat)

Specified by:

countTokensFromMessages in interface TokenCount.Calculator