> ## Documentation Index
> Fetch the complete documentation index at: https://praison.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# Token Estimation Validation
> Validate token estimates and track estimation accuracy
```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
graph LR
subgraph "Token Estimation Validation"
Request[📋 User Request] --> Process[⚙️ Token Estimation Validation]
Process --> Result[✅ Result]
end
classDef input fill:#6366F1,stroke:#7C90A0,color:#fff
classDef process fill:#189AB4,stroke:#7C90A0,color:#fff
classDef output fill:#10B981,stroke:#7C90A0,color:#fff
class Request input
class Process process
class Result output
```
```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
sequenceDiagram
participant User
participant Agent
participant Feature as Token Estimation Validation
User->>Agent: Request
Agent->>Feature: Process request
Feature-->>Agent: Result
Agent-->>User: Response
```
```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from praisonaiagents import Agent
agent = Agent(name="estimator", instructions="Estimate and validate context token usage.")
agent.start("How many tokens will this 5000-word document use?")
```
Token estimation validation compares heuristic estimates against accurate counts, logging mismatches for debugging.
The user monitors context size; validated estimation compares heuristics to accurate counts and logs mismatches.
```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
graph LR
subgraph "Token Estimation Validation"
In[📝 Text] --> Heur[⚡ Heuristic Estimate]
Heur --> Val[🔍 Validate vs Accurate]
Val --> Agent[🤖 Agent]
Agent --> Out[✅ Logged Metrics]
end
classDef input fill:#6366F1,stroke:#7C90A0,color:#fff
classDef process fill:#F59E0B,stroke:#7C90A0,color:#fff
classDef agent fill:#8B0000,stroke:#7C90A0,color:#fff
classDef output fill:#10B981,stroke:#7C90A0,color:#fff
class In input
class Heur,Val process
class Agent agent
class Out output
```
## How It Works
```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
sequenceDiagram
participant User
participant Agent
participant ContextManager
participant Tiktoken
User->>Agent: Send message
Agent->>ContextManager: estimate_tokens(validate=True)
ContextManager->>Tiktoken: Accurate count
Tiktoken-->>ContextManager: Compare vs heuristic
ContextManager-->>User: Estimate + mismatch log
```
## Quick Start
```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from praisonaiagents import ContextManager, ManagerConfig, EstimationMode
config = ManagerConfig(
estimation_mode=EstimationMode.VALIDATED,
log_estimation_mismatch=True,
mismatch_threshold_pct=15.0,
)
manager = ContextManager(model="gpt-4o-mini", config=config)
tokens, metrics = manager.estimate_tokens(text, validate=True)
```
```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
praisonai chat
> /context config
```
## Estimation Modes
| Mode | Description | Performance |
| ----------- | ----------------------------- | ----------- |
| `HEURISTIC` | Fast character-based estimate | Fastest |
| `ACCURATE` | Use tiktoken if available | Slower |
| `VALIDATED` | Compare both, log mismatches | Slowest |
## Configuration
```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
config = ManagerConfig(
estimation_mode=EstimationMode.VALIDATED,
log_estimation_mismatch=True, # Log when mismatch > threshold
mismatch_threshold_pct=15.0, # 15% threshold
)
```
### Environment Variables
```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
export PRAISONAI_CONTEXT_ESTIMATION_MODE=validated
export PRAISONAI_CONTEXT_LOG_MISMATCH=true
```
## EstimationMetrics
```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
@dataclass
class EstimationMetrics:
heuristic_estimate: int # Fast estimate
accurate_estimate: int # Tiktoken count
error_pct: float # Percentage error
estimator_used: EstimationMode
```
## Mismatch Logging
When `log_estimation_mismatch=True` and error exceeds threshold:
```
WARNING: Token estimation mismatch: heuristic=1250, accurate=1100, error=13.6%
```
## Estimation Caching
Estimates are cached by content hash:
```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
# First call - computes estimate
tokens1, _ = manager.estimate_tokens(text)
# Second call - uses cache
tokens2, _ = manager.estimate_tokens(text)
# Cache key is MD5 hash of text
```
## Heuristic Algorithm
The heuristic uses character-based estimation:
```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
# ASCII characters: ~0.25 tokens per char
# Non-ASCII: ~1.3 tokens per char
# Plus overhead for message structure
```
## Accurate Estimation
When tiktoken is available:
```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
# Uses model-specific tokenizer
# Falls back to heuristic if unavailable
```
## CLI Usage
```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
# View estimation mode in config
praisonai chat
> /context config
# Shows:
# Estimation:
# estimation_mode: validated
# log_mismatch: True
```
## Best Practices
Heuristic estimation is fast and sufficient for most runs — reserve validated mode for debugging.
Fifteen to twenty percent is a typical threshold before logging estimation drift.
Spikes often indicate unusual Unicode, code blocks, or tool payloads — fix content, not just the estimator.
Model-specific tokenisers improve accuracy for billing-sensitive workloads.
## Related
Fast offline token counting
Track optimisation events and history