xyz.felh.openai.gpt3.tokenizer

Interface ByteSequence

All Known Implementing Classes:

ByteSequence.Of

public interface ByteSequence

Represents a sequences of bytes.

Author:

Forest Wang

Nested Class Summary

Nested Classes

Modifier and Type	Interface and Description
static class	ByteSequence.Of An immutable implementation of the ByteSequence.

Field Summary

Fields

Modifier and Type	Field and Description
static ByteSequence	EMPTY The empty ByteSequence of length 0.

Method Summary

Modifier and Type	Method and Description
byte	byteAt(int index) Returns the byte at the specified offset.
static ByteSequence	copyOf(ByteSequence sequence) Returns an immutable ByteSequence that is a copy of the specified ByteSequence.
boolean	equals(java.lang.Object obj) Compares the specified object with this byte sequence for equality.
static ByteSequence	from(java.lang.String text) Creates a ByteSequence from the specified String using the UTF-8 charset.
static ByteSequence	from(java.lang.String text, java.nio.charset.Charset charset) Creates a ByteSequence from the specified String using the specified Charset.
int	hashCode() Returns a hash code value for this byte sequence.
int	length() Returns the length of the byte sequence.
static ByteSequence	of(byte[] bytes) Returns a new ByteSequence instance containing the specified byte array.
ByteSequence	subSequence(int start, int end) Returns a new ByteSequence that is a sub-sequence of the current byte sequence.
byte[]	toByteArray() Returns a byte array representation of this byte sequence.
java.lang.String	toString(java.nio.charset.Charset charset) Converts the byte sequence to a String using the specified Charset.

Field Detail

EMPTY

static final ByteSequence EMPTY

The empty ByteSequence of length 0.

Method Detail

byteAt

byte byteAt(int index)

Returns the byte at the specified offset.

Parameters:

index - the zero-based byte offset within the sequence of bytes (0 <= index < length())

Returns:

the byte at the specified offset

Throws:

java.lang.IndexOutOfBoundsException - if the index is out of range (index < 0 || index >= length())

length

int length()

Returns the length of the byte sequence.

Returns:

the number of bytes in the sequence

subSequence

ByteSequence subSequence(int start,
                         int end)
                  throws java.lang.IndexOutOfBoundsException

Returns a new ByteSequence that is a sub-sequence of the current byte sequence. The sub-sequence starts with the byte value at the specified start index and extends to the byte value at index end - 1.

Parameters:

start - the beginning index, inclusive

end - the ending index, exclusive

Returns:

a new ByteSequence that is a sub-sequence of this byte sequence

Throws:

java.lang.IndexOutOfBoundsException - if the start or end index is invalid

hashCode

int hashCode()

Returns a hash code value for this byte sequence.

Overrides:

hashCode in class java.lang.Object

Returns:

a hash code value for this byte sequence

equals

boolean equals(java.lang.Object obj)

Compares the specified object with this byte sequence for equality. Returns true if and only if the specified object is also a byte sequence and both byte sequences have the same bytes in the same order.

Overrides:

equals in class java.lang.Object

Parameters:

obj - the object to be compared for equality with this byte sequence

Returns:

true if the specified object is equal to this byte sequence, false otherwise

toByteArray

byte[] toByteArray()

Returns a byte array representation of this byte sequence. The returned array will be a copy of the internal byte array, ensuring that modifications to the returned array do not affect the original byte sequence.

Returns:

a byte array representation of this byte sequence

toString

java.lang.String toString(java.nio.charset.Charset charset)

Converts the byte sequence to a String using the specified Charset.

Parameters:

charset - the Charset to be used for the conversion

Returns:

a String representation of this byte sequence using the specified Charset

of

static ByteSequence of(byte[] bytes)

Returns a new ByteSequence instance containing the specified byte array. The provided byte array is wrapped in an ImmutableByteSequence to ensure that the contents of the byte array are not modified after the ByteSequence is created.

Parameters:

bytes - the byte array to be used for the new ByteSequence

Returns:

a new ByteSequence instance containing the specified byte array

Throws:

java.lang.NullPointerException - if the provided byte array is null

copyOf

static ByteSequence copyOf(ByteSequence sequence)

Returns an immutable ByteSequence that is a copy of the specified ByteSequence. If the provided ByteSequence is already an instance of ImmutableByteSequence, it is returned directly; otherwise, a new ImmutableByteSequence is created.

Parameters:

sequence - the ByteSequence to be copied

Returns:

an immutable ByteSequence that is a copy of the specified ByteSequence

Throws:

java.lang.NullPointerException - if the provided ByteSequence is null

from

static ByteSequence from(java.lang.String text)

Creates a ByteSequence from the specified String using the UTF-8 charset.

Parameters:

text - the String to be converted to a ByteSequence

Returns:

a new ByteSequence that represents the specified String using the UTF-8 charset

Throws:

java.lang.NullPointerException - if the provided text is null

from

static ByteSequence from(java.lang.String text,
                         java.nio.charset.Charset charset)

Creates a ByteSequence from the specified String using the specified Charset.

Parameters:

text - the String to be converted to a ByteSequence

charset - the Charset to be used for the conversion

Returns:

a new ByteSequence that represents the specified String using the specified Charset

Throws:

java.lang.NullPointerException - if the provided text or charset is null