Skip to main content
from praisonaiagents import Agent

agent = Agent(name="token-agent", instructions="Estimate token counts accurately.")
agent.start("Estimate how many tokens this message will consume.")

​
Token Estimation

PraisonAI provides fast, offline token estimation that works without API calls. This enables real-time context budget tracking and optimization decisions. The user estimates tokens for a prompt; heuristics predict cost before the agent calls the model.

​
How It Works

​
Quick Start

1

Estimate text tokens

from praisonaiagents import estimate_tokens_heuristic

tokens = estimate_tokens_heuristic("Hello, how are you today?")
print(f"Estimated: {tokens} tokens")
2

Estimate messages and tool schemas

from praisonaiagents import estimate_messages_tokens, estimate_tool_schema_tokens

messages = [
    {"role": "system", "content": "You are helpful."},
    {"role": "user", "content": "What is Python?"},
]
print(f"Messages: {estimate_messages_tokens(messages)} tokens")

​
Estimation Algorithm

The heuristic estimator uses character-based rules optimized for typical LLM tokenization:
Character TypeTokens per Character
ASCII text~0.25 (4 chars = 1 token)
Non-ASCII (Unicode)~1.3 tokens per char
WhitespaceCounted normally

​
Message Overhead

Each message includes overhead for role markers and formatting:
  • Base overhead: 4 tokens per message
  • Role tokens: ~2 tokens
  • Content: Estimated via heuristic

​
API Reference

​
estimate_tokens_heuristic(text: str) -> int

Estimate tokens for a string using character-based heuristics. 
tokens = estimate_tokens_heuristic("Hello world!")
# Returns: ~3 tokens

​
estimate_messages_tokens(messages: List[Dict]) -> int

Estimate total tokens for a list of chat messages. 
messages = [
    {"role": "user", "content": "Hello"},
    {"role": "assistant", "content": "Hi there!"},
]
tokens = estimate_messages_tokens(messages)

​
estimate_tool_schema_tokens(tools: List[Dict]) -> int

Estimate tokens for tool/function schemas. 
tools = [{"name": "search", "description": "Search the web"}]
tokens = estimate_tool_schema_tokens(tools)

​
TokenEstimatorImpl

Class-based estimator with caching: 
from praisonaiagents import TokenEstimatorImpl

estimator = TokenEstimatorImpl()
tokens = estimator.estimate("Some text")

​
get_estimator() -> TokenEstimatorImpl

Get a singleton estimator instance: 
from praisonaiagents import get_estimator

estimator = get_estimator()
tokens = estimator.estimate("Text to estimate")

​
Accuracy Considerations

The heuristic estimator is designed for speed over perfect accuracy:
ScenarioAccuracy
English text~90-95%
Code~85-90%
Mixed content~85-90%
Non-ASCII heavy~80-85%
For budget decisions, the estimator adds a small safety margin to prevent underestimation.

​
Performance

  • Speed: < 1ms for 100K characters
  • Memory: O(1) - no caching required
  • No API calls: Works completely offline

​
Integration with Budgeter

from praisonaiagents import (
    ContextBudgeter,
    estimate_messages_tokens,
)

budgeter = ContextBudgeter(model="gpt-4o-mini")
budget = budgeter.allocate()

# Check if messages fit in budget
messages = [...]  # Your conversation
tokens = estimate_messages_tokens(messages)

if tokens > budget.usable * 0.8:
    print("Warning: Approaching context limit!")

​
Best Practices

Check message totals against the budget to trigger compaction proactively.
Pass the model name so tiktoken or provider-specific counting applies when available.
Act before hard limits — retrieval trimming and summarisation need headroom.
Run validated mode on fixture conversations to catch drift in token counting.

Context Ledger

Track tokens by segment

Context Budgeter

Allocate token budgets