xyz.felh.openai.gpt3.tokenizer

Class GPT3Tokenizer

java.lang.Object

xyz.felh.openai.gpt3.tokenizer.GPT3Tokenizer

public class GPT3Tokenizer
extends java.lang.Object

Java implementation of the GPT3/4 tokenizer.

Modifications:

• [MB] 2023-03-25: Repackaged from Tiktoken for inclusion in gpt3-tokenizer-java.
• [MB] 2023-04-02: Major refactoring for cleaner code and improved performance.

Constructor Summary

Constructors

Constructor and Description
GPT3Tokenizer(Encoding encoding)

Method Summary

Modifier and Type	Method and Description
protected int	bytePairMerge(ByteSequence piece, java.util.Collection<java.lang.Integer> result)
protected java.util.regex.Pattern	createSpecialRegex(java.util.Map<java.lang.String,?> specialTokensEncoder)
java.lang.String	decode(java.util.List<java.lang.Integer> tokens)
protected java.lang.String	decodeImpl(java.util.List<java.lang.Integer> tokens)
java.util.List<java.lang.Integer>	encode(java.lang.String text)
java.util.List<java.lang.Integer>	encode(java.lang.String text, boolean allowedSpecial)
java.util.List<java.lang.Integer>	encode(java.lang.String text, java.util.Set<java.lang.String> allowedSpecial)
protected java.util.List<java.lang.Integer>	encodeImpl(java.lang.String text, java.util.Set<java.lang.String> allowedSpecial)
protected int	getRank(ByteSequence piece, java.util.List<xyz.felh.openai.gpt3.tokenizer.GPT3Tokenizer.IntPair> partsList, int startIdx)
protected java.util.regex.Pattern	getTlRegex() Returns the regular expression for tokenizing text
protected java.util.regex.Pattern	getTlSpecialRegex() Returns the regular expression for detecting special tokens

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

GPT3Tokenizer

public GPT3Tokenizer(Encoding encoding)

Method Detail

createSpecialRegex

protected java.util.regex.Pattern createSpecialRegex(java.util.Map<java.lang.String,?> specialTokensEncoder)

decode

public java.lang.String decode(java.util.List<java.lang.Integer> tokens)

decodeImpl

protected java.lang.String decodeImpl(java.util.List<java.lang.Integer> tokens)

getTlSpecialRegex

protected java.util.regex.Pattern getTlSpecialRegex()

Returns the regular expression for detecting special tokens

Returns:

the special tokenizing pattern

getTlRegex

protected java.util.regex.Pattern getTlRegex()

Returns the regular expression for tokenizing text

Returns:

the tokenizing pattern

encode

public java.util.List<java.lang.Integer> encode(java.lang.String text)

encode

public java.util.List<java.lang.Integer> encode(java.lang.String text,
                                                boolean allowedSpecial)

encode

public java.util.List<java.lang.Integer> encode(java.lang.String text,
                                                java.util.Set<java.lang.String> allowedSpecial)

encodeImpl

protected java.util.List<java.lang.Integer> encodeImpl(java.lang.String text,
                                                       java.util.Set<java.lang.String> allowedSpecial)

getRank

protected int getRank(ByteSequence piece,
                      java.util.List<xyz.felh.openai.gpt3.tokenizer.GPT3Tokenizer.IntPair> partsList,
                      int startIdx)

bytePairMerge

protected int bytePairMerge(ByteSequence piece,
                            java.util.Collection<java.lang.Integer> result)