B C D E F G H L M O P R S T X

B

byteAt(int) - Method in interface xyz.felh.openai.gpt3.tokenizer.ByteSequence

Returns the byte at the specified offset.

byteAt(int) - Method in class xyz.felh.openai.gpt3.tokenizer.ByteSequence.Of

bytePairMerge(ByteSequence, Collection<Integer>) - Method in class xyz.felh.openai.gpt3.tokenizer.GPT3Tokenizer

ByteSequence - Interface in xyz.felh.openai.gpt3.tokenizer

Represents a sequences of bytes.

ByteSequence.Of - Class in xyz.felh.openai.gpt3.tokenizer

An immutable implementation of the ByteSequence.

C

ChatFormatDescriptor - Class in xyz.felh.openai.gpt3.tokenizer

Describes the various chat messaging formats for the purpose of counting tokens in chat conversations against different models.

CL100K_BASE - Static variable in class xyz.felh.openai.gpt3.tokenizer.Encoding

compareTo(ByteSequence.Of) - Method in class xyz.felh.openai.gpt3.tokenizer.ByteSequence.Of

copyOf(ByteSequence) - Static method in interface xyz.felh.openai.gpt3.tokenizer.ByteSequence

Returns an immutable ByteSequence that is a copy of the specified ByteSequence.

countTokensFromMessages(List<ChatMessage>, GPT3Tokenizer, ChatFormatDescriptor) - Method in interface xyz.felh.openai.gpt3.tokenizer.TokenCount.Calculator

countTokensFromMessages(List<ChatMessage>, GPT3Tokenizer, ChatFormatDescriptor) - Method in class xyz.felh.openai.gpt3.tokenizer.TokenCount.StandardCalculator

countTokensFromString(String, GPT3Tokenizer) - Method in interface xyz.felh.openai.gpt3.tokenizer.TokenCount.Calculator

countTokensFromString(String, GPT3Tokenizer) - Method in class xyz.felh.openai.gpt3.tokenizer.TokenCount.StandardCalculator

createSpecialRegex(Map<String, ?>) - Method in class xyz.felh.openai.gpt3.tokenizer.GPT3Tokenizer

D

decode(List<Integer>) - Method in class xyz.felh.openai.gpt3.tokenizer.GPT3Tokenizer

decodeImpl(List<Integer>) - Method in class xyz.felh.openai.gpt3.tokenizer.GPT3Tokenizer

E

EMPTY - Static variable in interface xyz.felh.openai.gpt3.tokenizer.ByteSequence

The empty ByteSequence of length 0.

encode(String) - Method in class xyz.felh.openai.gpt3.tokenizer.GPT3Tokenizer

encode(String, boolean) - Method in class xyz.felh.openai.gpt3.tokenizer.GPT3Tokenizer

encode(String, Set<String>) - Method in class xyz.felh.openai.gpt3.tokenizer.GPT3Tokenizer

encodeImpl(String, Set<String>) - Method in class xyz.felh.openai.gpt3.tokenizer.GPT3Tokenizer

Encoding - Class in xyz.felh.openai.gpt3.tokenizer

Represents variants of BPE encoding.

ENDOFPROMPT - Static variable in class xyz.felh.openai.gpt3.tokenizer.Encoding

ENDOFTEXT - Static variable in class xyz.felh.openai.gpt3.tokenizer.Encoding

equals(Object) - Method in interface xyz.felh.openai.gpt3.tokenizer.ByteSequence

Compares the specified object with this byte sequence for equality.

equals(Object) - Method in class xyz.felh.openai.gpt3.tokenizer.ByteSequence.Of

F

FIM_MIDDLE - Static variable in class xyz.felh.openai.gpt3.tokenizer.Encoding

FIM_PREFIX - Static variable in class xyz.felh.openai.gpt3.tokenizer.Encoding

FIM_SUFFIX - Static variable in class xyz.felh.openai.gpt3.tokenizer.Encoding

forModel(String) - Static method in class xyz.felh.openai.gpt3.tokenizer.ChatFormatDescriptor

forModel(String) - Static method in class xyz.felh.openai.gpt3.tokenizer.Encoding

forName(String) - Static method in class xyz.felh.openai.gpt3.tokenizer.Encoding

from(String) - Static method in interface xyz.felh.openai.gpt3.tokenizer.ByteSequence

Creates a ByteSequence from the specified String using the UTF-8 charset.

from(String, Charset) - Static method in interface xyz.felh.openai.gpt3.tokenizer.ByteSequence

Creates a ByteSequence from the specified String using the specified Charset.

fromLinesJoined(Iterable<String>, GPT3Tokenizer) - Static method in class xyz.felh.openai.gpt3.tokenizer.TokenCount

Calculates the total token count from a list of lines using the given tokenizer, including newline tokens between lines.

fromMessages(List<ChatMessage>, GPT3Tokenizer, ChatFormatDescriptor) - Static method in class xyz.felh.openai.gpt3.tokenizer.TokenCount

Calculates the token count for a list of chat messages using the provided tokenizer and chat format descriptor.

fromString(String, GPT3Tokenizer) - Static method in class xyz.felh.openai.gpt3.tokenizer.TokenCount

Calculates the token count for a given text string using the provided tokenizer.

G

getCalculator() - Static method in class xyz.felh.openai.gpt3.tokenizer.TokenCount

Returns the current TokenCount.Calculator instance.

getRank(ByteSequence, List<GPT3Tokenizer.IntPair>, int) - Method in class xyz.felh.openai.gpt3.tokenizer.GPT3Tokenizer

getTlRegex() - Method in class xyz.felh.openai.gpt3.tokenizer.GPT3Tokenizer

Returns the regular expression for tokenizing text

getTlSpecialRegex() - Method in class xyz.felh.openai.gpt3.tokenizer.GPT3Tokenizer

Returns the regular expression for detecting special tokens

GPT3Tokenizer - Class in xyz.felh.openai.gpt3.tokenizer

Java implementation of the GPT3/4 tokenizer.

GPT3Tokenizer(Encoding) - Constructor for class xyz.felh.openai.gpt3.tokenizer.GPT3Tokenizer

H

hashCode() - Method in interface xyz.felh.openai.gpt3.tokenizer.ByteSequence

Returns a hash code value for this byte sequence.

hashCode() - Method in class xyz.felh.openai.gpt3.tokenizer.ByteSequence.Of

L

length() - Method in interface xyz.felh.openai.gpt3.tokenizer.ByteSequence

Returns the length of the byte sequence.

length() - Method in class xyz.felh.openai.gpt3.tokenizer.ByteSequence.Of

M

mergeableRanks() - Method in class xyz.felh.openai.gpt3.tokenizer.Encoding

O

of(byte[]) - Static method in interface xyz.felh.openai.gpt3.tokenizer.ByteSequence

Returns a new ByteSequence instance containing the specified byte array.

P

P50K_BASE - Static variable in class xyz.felh.openai.gpt3.tokenizer.Encoding

P50K_EDIT - Static variable in class xyz.felh.openai.gpt3.tokenizer.Encoding

R

R50K_BASE - Static variable in class xyz.felh.openai.gpt3.tokenizer.Encoding

S

StandardCalculator() - Constructor for class xyz.felh.openai.gpt3.tokenizer.TokenCount.StandardCalculator

subSequence(int, int) - Method in class xyz.felh.openai.gpt3.tokenizer.ByteSequence.Of

subSequence(int, int) - Method in interface xyz.felh.openai.gpt3.tokenizer.ByteSequence

Returns a new ByteSequence that is a sub-sequence of the current byte sequence.

T

toByteArray() - Method in class xyz.felh.openai.gpt3.tokenizer.ByteSequence.Of

toByteArray() - Method in interface xyz.felh.openai.gpt3.tokenizer.ByteSequence

Returns a byte array representation of this byte sequence.

TokenCount - Class in xyz.felh.openai.gpt3.tokenizer

Utility class for calculating token count in text and chat messages.

TokenCount() - Constructor for class xyz.felh.openai.gpt3.tokenizer.TokenCount

TokenCount.Calculator - Interface in xyz.felh.openai.gpt3.tokenizer

Interface for pluggable token counting logic.

TokenCount.StandardCalculator - Class in xyz.felh.openai.gpt3.tokenizer

Default implementation of the TokenCount.Calculator interface.

toString(Charset) - Method in class xyz.felh.openai.gpt3.tokenizer.ByteSequence.Of

toString() - Method in class xyz.felh.openai.gpt3.tokenizer.ByteSequence.Of

toString(Charset) - Method in interface xyz.felh.openai.gpt3.tokenizer.ByteSequence

Converts the byte sequence to a String using the specified Charset.

X

xyz.felh.openai.gpt3.tokenizer - package xyz.felh.openai.gpt3.tokenizer

B C D E F G H L M O P R S T X