Package xyz.felh.openai.gpt3.tokenizer

Interface Summary

Interface	Description
ByteSequence	Represents a sequences of bytes.
TokenCount.Calculator	Interface for pluggable token counting logic.

Class Summary

Class	Description
ByteSequence.Of	An immutable implementation of the ByteSequence.
ChatFormatDescriptor	Describes the various chat messaging formats for the purpose of counting tokens in chat conversations against different models.
Encoding	Represents variants of BPE encoding.
GPT3Tokenizer	Java implementation of the GPT3/4 tokenizer.
TokenCount	Utility class for calculating token count in text and chat messages.
TokenCount.StandardCalculator	Default implementation of the TokenCount.Calculator interface.