Package ai.djl.huggingface.tokenizers

Class PadTokenResolver

java.lang.Object

ai.djl.huggingface.tokenizers.PadTokenResolver

public final class PadTokenResolver
extends Object

The type Pad token resolver.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	PadTokenResolver.PadInfo	The type Pad info.

Method Summary

Modifier and Type	Method	Description
static PadTokenResolver.PadInfo	extractPadInfo(String tokenizerJson, TokenizerConfig config)	Extracts pad token and ID from tokenizer.json and/or tokenizer_config.json.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Details

extractPadInfo

public static PadTokenResolver.PadInfo extractPadInfo(String tokenizerJson,
 TokenizerConfig config)

Extracts pad token and ID from tokenizer.json and/or tokenizer_config.json. Follows HF behavior: tokenizer.json takes precedence, config is fallback.

Parameters:

tokenizerJson - tokenizer.json content as string

config - TokenizerConfig, may be null

Returns:

PadInfo or null if pad token cannot be resolved