xyz.felh.openai.gpt3.tokenizer

Class Encoding

java.lang.Object

xyz.felh.openai.gpt3.tokenizer.Encoding

public class Encoding
extends java.lang.Object

Represents variants of BPE encoding.

Modifications:

• [MB] 2023-03-25: Repackaged from Tiktoken for inclusion in gpt3-tokenizer-java.
• [MB] 2023-04-02: Major refactoring for cleaner code and improved performance.

Field Summary

Fields

Modifier and Type	Field and Description
static Encoding	CL100K_BASE
static java.lang.String	ENDOFPROMPT
static java.lang.String	ENDOFTEXT
static java.lang.String	FIM_MIDDLE
static java.lang.String	FIM_PREFIX
static java.lang.String	FIM_SUFFIX
static Encoding	P50K_BASE
static Encoding	P50K_EDIT
static Encoding	R50K_BASE

Method Summary

Modifier and Type	Method and Description
static Encoding	forModel(java.lang.String modelName)
static Encoding	forName(java.lang.String encodingName)
java.util.Map<ByteSequence,java.lang.Integer>	mergeableRanks()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

ENDOFTEXT

public static final java.lang.String ENDOFTEXT

See Also:

Constant Field Values

FIM_PREFIX

public static final java.lang.String FIM_PREFIX

See Also:

Constant Field Values

FIM_MIDDLE

public static final java.lang.String FIM_MIDDLE

See Also:

Constant Field Values

FIM_SUFFIX

public static final java.lang.String FIM_SUFFIX

See Also:

Constant Field Values

ENDOFPROMPT

public static final java.lang.String ENDOFPROMPT

See Also:

Constant Field Values

CL100K_BASE

public static final Encoding CL100K_BASE

P50K_BASE

public static final Encoding P50K_BASE

P50K_EDIT

public static final Encoding P50K_EDIT

R50K_BASE

public static final Encoding R50K_BASE

Method Detail

mergeableRanks

public java.util.Map<ByteSequence,java.lang.Integer> mergeableRanks()

forName

public static Encoding forName(java.lang.String encodingName)

forModel

public static Encoding forModel(java.lang.String modelName)