Package ai.platon.pulsar.coding

Object TokenEstimator

  • All Implemented Interfaces:
    
public class TokenEstimator

    Heuristic LLM token estimator for tool-call input/output text.

    Zero dependencies. Approximates BPE tokenizers (cl100k/o200k class) within roughly ±25% for prose and source code — good enough for cost accounting and efficiency analysis, NOT for billing.

    Model:

    • Identifiers are split at camelCase boundaries; each sub-word of length L costs ceil(L/5) tokens (English words average ~5 chars/sub-word).

    • Digit runs cost ceil(n/3) (BPE groups digits 1-3 per token).

    • Each CJK character costs 1 token.

    • Each symbol/punctuation character costs 1 token (slightly overestimates common merges like ::, ->, ==).

    • Whitespace largely merges into neighboring tokens; counted as 1 per 8 chars.

    • Nested Class Summary

      Nested Classes 
      Modifier and Type Class Description

    • Constructor Summary

      Constructors 
      Constructor Description

    • Enum Constant Summary

      Enum Constants 
      Enum Constant Description

    • Method Summary

      Modifier and Type Method Description
      final Long estimateTokens(String text) Estimated token count of text; 0 for empty input.

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait