C D E F G I M N R T V X

C

countTokens(String) - Method in interface xyz.felh.openai.jtokkit.api.Encoding

Encodes the given text into a list of token ids and returns the amount of tokens.

countTokensOrdinary(String) - Method in interface xyz.felh.openai.jtokkit.api.Encoding

Encodes the given text into a list of token ids and returns the amount of tokens.

D

decode(List<Integer>) - Method in interface xyz.felh.openai.jtokkit.api.Encoding

Decodes the given list of token ids into a text.

decode(Encoding, List<Integer>) - Static method in class xyz.felh.openai.jtokkit.utils.TikTokenUtils

通过Encoding和encoded数组反推text信息

decode(EncodingType, List<Integer>) - Static method in class xyz.felh.openai.jtokkit.utils.TikTokenUtils

通过EncodingType和encoded编码数组，反推字符串文本

decode(String, List<Integer>) - Static method in class xyz.felh.openai.jtokkit.utils.TikTokenUtils

通过模型名称和encoded编码数组，反推字符串文本

decodeBytes(List<Integer>) - Method in interface xyz.felh.openai.jtokkit.api.Encoding

Decodes the given list of token ids into a byte array.

E

encode(String) - Method in interface xyz.felh.openai.jtokkit.api.Encoding

Encodes the given text into a list of token ids.

encode(String, int) - Method in interface xyz.felh.openai.jtokkit.api.Encoding

Encodes the given text into a list of token ids.

encode(Encoding, String) - Static method in class xyz.felh.openai.jtokkit.utils.TikTokenUtils

通过Encoding和text获取编码数组

encode(EncodingType, String) - Static method in class xyz.felh.openai.jtokkit.utils.TikTokenUtils

获取encode的编码数组

encode(String, String) - Static method in class xyz.felh.openai.jtokkit.utils.TikTokenUtils

获取encode的编码数组，通过模型名称

encodeOrdinary(String) - Method in interface xyz.felh.openai.jtokkit.api.Encoding

Encodes the given text into a list of token ids, ignoring special tokens.

encodeOrdinary(String, int) - Method in interface xyz.felh.openai.jtokkit.api.Encoding

Encodes the given text into a list of token ids, ignoring special tokens.

Encoding - Interface in xyz.felh.openai.jtokkit.api

EncodingRegistry - Interface in xyz.felh.openai.jtokkit.api

The EncodingRegistry is used to register custom encodings and to retrieve encodings by name or type.

EncodingResult - Class in xyz.felh.openai.jtokkit.api

The result of encoding operation.

EncodingResult(List<Integer>, boolean) - Constructor for class xyz.felh.openai.jtokkit.api.EncodingResult

Encodings - Class in xyz.felh.openai.jtokkit

EncodingType - Enum in xyz.felh.openai.jtokkit.api

F

fromName(String) - Static method in enum xyz.felh.openai.jtokkit.api.EncodingType

fromName(String) - Static method in enum xyz.felh.openai.jtokkit.api.ModelType

Returns a ModelType for the given name, or Optional.empty() if no such model type exists.

G

getEncoder() - Method in class xyz.felh.openai.jtokkit.api.GptBytePairEncodingParams

getEncoding(String) - Method in interface xyz.felh.openai.jtokkit.api.EncodingRegistry

Returns the encoding with the given name, if it exists.

getEncoding(EncodingType) - Method in interface xyz.felh.openai.jtokkit.api.EncodingRegistry

Returns the encoding with the given type.

getEncoding(EncodingType) - Static method in class xyz.felh.openai.jtokkit.utils.TikTokenUtils

获取一个Encoding对象，通过Encoding类型

getEncoding(String) - Static method in class xyz.felh.openai.jtokkit.utils.TikTokenUtils

获取一个Encoding对象，通过模型名称

getEncodingForModel(String) - Method in interface xyz.felh.openai.jtokkit.api.EncodingRegistry

Returns the encoding that is used for the given model type, if it exists.

getEncodingForModel(ModelType) - Method in interface xyz.felh.openai.jtokkit.api.EncodingRegistry

Returns the encoding that is used for the given model type.

getEncodingType() - Method in enum xyz.felh.openai.jtokkit.api.ModelType

Returns the encoding type that is used by this model type.

getMaxContextLength() - Method in enum xyz.felh.openai.jtokkit.api.ModelType

Returns the maximum context length that is supported by this model type.

getModelTypeByName(String) - Static method in class xyz.felh.openai.jtokkit.utils.TikTokenUtils

获取modelType

getName() - Method in interface xyz.felh.openai.jtokkit.api.Encoding

Returns the name of this encoding.

getName() - Method in enum xyz.felh.openai.jtokkit.api.EncodingType

getName() - Method in class xyz.felh.openai.jtokkit.api.GptBytePairEncodingParams

getName() - Method in enum xyz.felh.openai.jtokkit.api.ModelType

Returns the name of the model type as used by the OpenAI API.

getPattern() - Method in class xyz.felh.openai.jtokkit.api.GptBytePairEncodingParams

getSpecialTokensEncoder() - Method in class xyz.felh.openai.jtokkit.api.GptBytePairEncodingParams

getTokens() - Method in class xyz.felh.openai.jtokkit.api.EncodingResult

GptBytePairEncodingParams - Class in xyz.felh.openai.jtokkit.api

Parameter for the byte pair encoding used to tokenize for the OpenAI GPT models.

GptBytePairEncodingParams(String, Pattern, Map<byte[], Integer>, Map<String, Integer>) - Constructor for class xyz.felh.openai.jtokkit.api.GptBytePairEncodingParams

Creates a new instance of GptBytePairEncodingParams.

I

isTruncated() - Method in class xyz.felh.openai.jtokkit.api.EncodingResult

M

ModelType - Enum in xyz.felh.openai.jtokkit.api

N

newDefaultEncodingRegistry() - Static method in class xyz.felh.openai.jtokkit.Encodings

Creates a new EncodingRegistry with the default encodings found in the EncodingType enum.

newLazyEncodingRegistry() - Static method in class xyz.felh.openai.jtokkit.Encodings

Creates a new EncodingRegistry without any EncodingType registered.

R

registerCustomEncoding(Encoding) - Method in interface xyz.felh.openai.jtokkit.api.EncodingRegistry

Registers a new custom encoding with the given name.

registerGptBytePairEncoding(GptBytePairEncodingParams) - Method in interface xyz.felh.openai.jtokkit.api.EncodingRegistry

Registers a new byte pair encoding with the given name.

T

TikTokenUtils - Class in xyz.felh.openai.jtokkit.utils

TikTokenUtils() - Constructor for class xyz.felh.openai.jtokkit.utils.TikTokenUtils

tokens(Encoding, String) - Static method in class xyz.felh.openai.jtokkit.utils.TikTokenUtils

通过Encoding计算text信息的tokens

tokens(EncodingType, String) - Static method in class xyz.felh.openai.jtokkit.utils.TikTokenUtils

计算指定字符串的tokens，通过EncodingType

tokens(String, String) - Static method in class xyz.felh.openai.jtokkit.utils.TikTokenUtils

通过模型名称, 计算指定字符串的tokens

tokens(String, List<ChatMessage>) - Static method in class xyz.felh.openai.jtokkit.utils.TikTokenUtils

通过模型名称计算messages获取编码数组 参考官方的处理逻辑： https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb

tokens(String, Object, List<Function>) - Static method in class xyz.felh.openai.jtokkit.utils.TikTokenUtils

toString() - Method in class xyz.felh.openai.jtokkit.api.EncodingResult

V

valueOf(String) - Static method in enum xyz.felh.openai.jtokkit.api.EncodingType

Returns the enum constant of this type with the specified name.

valueOf(String) - Static method in enum xyz.felh.openai.jtokkit.api.ModelType

Returns the enum constant of this type with the specified name.

values() - Static method in enum xyz.felh.openai.jtokkit.api.EncodingType

Returns an array containing the constants of this enum type, in the order they are declared.

values() - Static method in enum xyz.felh.openai.jtokkit.api.ModelType

Returns an array containing the constants of this enum type, in the order they are declared.

X

xyz.felh.openai.jtokkit - package xyz.felh.openai.jtokkit

xyz.felh.openai.jtokkit.api - package xyz.felh.openai.jtokkit.api

xyz.felh.openai.jtokkit.utils - package xyz.felh.openai.jtokkit.utils

C D E F G I M N R T V X