Token Estimation

Fast offline token counting for context management

from praisonaiagents import Agent

agent = Agent(name="token-agent", instructions="Estimate token counts accurately.")
agent.start("Estimate how many tokens this message will consume.")

Token Estimation

PraisonAI provides fast, offline token estimation that works without API calls. This enables real-time context budget tracking and optimization decisions.

The user estimates tokens for a prompt; heuristics predict cost before the agent calls the model.

How It Works

Quick Start

1. Estimate text tokens

   from praisonaiagents.context import estimate_tokens_heuristic
   
   tokens = estimate_tokens_heuristic("Hello, how are you today?")
   print(f"Estimated: {tokens} tokens")

2. Estimate messages and tool schemas

   from praisonaiagents.context import estimate_messages_tokens, estimate_tool_schema_tokens
   
   messages = [
       {"role": "system", "content": "You are helpful."},
       {"role": "user", "content": "What is Python?"},
   ]
   print(f"Messages: {estimate_messages_tokens(messages)} tokens")

Estimation Algorithm

The heuristic estimator uses character-based rules optimized for typical LLM tokenization:

For example, the same 1,700 characters count very differently depending on the script:

from praisonaiagents.context import estimate_tokens_heuristic

english = "a" * 1700
chinese = "今天天气很好，我们一起去公园散步。" * 100  # 1,700 characters

estimate_tokens_heuristic(english)  # 425  (1700 × 0.25)
estimate_tokens_heuristic(chinese)  # 2210 (1700 × 1.3)

The old len(text) // 4 shortcut gave about 426 tokens for the Chinese text, roughly 5× too low.

Message Overhead

Each message includes overhead for role markers and formatting:

• Base overhead: 4 tokens per message
• Role tokens: ~2 tokens
• Content: Estimated via heuristic

API Reference

estimate_tokens_heuristic(text: str) -> int

Estimate tokens for a string using character-based heuristics.

tokens = estimate_tokens_heuristic("Hello world!")
# Returns: ~3 tokens

estimate_messages_tokens(messages: List[Dict]) -> int

Estimate total tokens for a list of chat messages.

messages = [
    {"role": "user", "content": "Hello"},
    {"role": "assistant", "content": "Hi there!"},
]
tokens = estimate_messages_tokens(messages)

estimate_tool_schema_tokens(tools: List[Dict]) -> int

Estimate tokens for tool/function schemas.

tools = [{"name": "search", "description": "Search the web"}]
tokens = estimate_tool_schema_tokens(tools)

TokenEstimatorImpl

Class-based estimator with caching:

from praisonaiagents.context import TokenEstimatorImpl

estimator = TokenEstimatorImpl()
tokens = estimator.estimate("Some text")

get_estimator() -> TokenEstimatorImpl

Get a singleton estimator instance:

from praisonaiagents.context import get_estimator

estimator = get_estimator()
tokens = estimator.estimate("Text to estimate")

Accuracy Considerations

The heuristic estimator is designed for speed over perfect accuracy:

For budget decisions, the estimator adds a small safety margin to prevent underestimation.

Performance

• Speed: < 1ms for 100K characters
• Memory: O(1) – no caching required
• No API calls: Works completely offline

ContextManager.estimate_tokens() only caches results on the validated (tiktoken) path. See Token Estimation Validation for details.

Integration with Budgeter

from praisonaiagents.context import (
    ContextBudgeter,
    estimate_messages_tokens,
)

budgeter = ContextBudgeter(model="gpt-4o-mini")
budget = budgeter.allocate()

# Check if messages fit in budget
messages = [...]  # Your conversation
tokens = estimate_messages_tokens(messages)

if tokens > budget.usable * 0.8:
    print("Warning: Approaching context limit!")

Best Practices

Related