> ## Documentation Index
> Fetch the complete documentation index at: https://praison.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Token Estimation

> Fast offline token counting for context management

```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
graph LR
    subgraph "Token Estimation"
        Request[📋 User Request] --> Process[⚙️ Token Estimation]
        Process --> Result[✅ Result]
    end

    classDef input fill:#6366F1,stroke:#7C90A0,color:#fff
    classDef process fill:#189AB4,stroke:#7C90A0,color:#fff
    classDef output fill:#10B981,stroke:#7C90A0,color:#fff

    class Request input
    class Process process
    class Result output
```

```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
sequenceDiagram
    participant User
    participant Agent
    participant Feature as Token Estimation

    User->>Agent: Request
    Agent->>Feature: Process request
    Feature-->>Agent: Result
    Agent-->>User: Response
```

```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from praisonaiagents import Agent

agent = Agent(name="token-agent", instructions="Estimate token counts accurately.")
agent.start("Estimate how many tokens this message will consume.")
```

# Token Estimation

PraisonAI provides fast, offline token estimation that works without API calls. This enables real-time context budget tracking and optimization decisions.

The user estimates tokens for a prompt; heuristics predict cost before the agent calls the model.

```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
graph LR
    subgraph "Token Estimation"
        In[📝 Text] --> Est[⚡ Heuristic Count]
        Est --> Agent[🤖 Agent]
        Agent --> Out[✅ Token Estimate]
    end

    classDef input fill:#6366F1,stroke:#7C90A0,color:#fff
    classDef process fill:#F59E0B,stroke:#7C90A0,color:#fff
    classDef agent fill:#8B0000,stroke:#7C90A0,color:#fff
    classDef output fill:#10B981,stroke:#7C90A0,color:#fff

    class In input
    class Est process
    class Agent agent
    class Out output
```

## How It Works

```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
sequenceDiagram
    participant User
    participant Agent
    participant Estimator

    User->>Agent: Provide prompt
    Agent->>Estimator: estimate_tokens_heuristic(text)
    Estimator-->>Agent: Offline token count
    Agent-->>User: Budget decision before API call
```

## Quick Start

<Steps>
  <Step title="Estimate text tokens">
    ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
    from praisonaiagents import estimate_tokens_heuristic

    tokens = estimate_tokens_heuristic("Hello, how are you today?")
    print(f"Estimated: {tokens} tokens")
    ```
  </Step>

  <Step title="Estimate messages and tool schemas">
    ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
    from praisonaiagents import estimate_messages_tokens, estimate_tool_schema_tokens

    messages = [
        {"role": "system", "content": "You are helpful."},
        {"role": "user", "content": "What is Python?"},
    ]
    print(f"Messages: {estimate_messages_tokens(messages)} tokens")
    ```
  </Step>
</Steps>

## Estimation Algorithm

The heuristic estimator uses character-based rules optimized for typical LLM tokenization:

| Character Type      | Tokens per Character       |
| ------------------- | -------------------------- |
| ASCII text          | \~0.25 (4 chars = 1 token) |
| Non-ASCII (Unicode) | \~1.3 tokens per char      |
| Whitespace          | Counted normally           |

### Message Overhead

Each message includes overhead for role markers and formatting:

* **Base overhead**: 4 tokens per message
* **Role tokens**: \~2 tokens
* **Content**: Estimated via heuristic

## API Reference

### `estimate_tokens_heuristic(text: str) -> int`

Estimate tokens for a string using character-based heuristics.

```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
tokens = estimate_tokens_heuristic("Hello world!")
# Returns: ~3 tokens
```

### `estimate_messages_tokens(messages: List[Dict]) -> int`

Estimate total tokens for a list of chat messages.

```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
messages = [
    {"role": "user", "content": "Hello"},
    {"role": "assistant", "content": "Hi there!"},
]
tokens = estimate_messages_tokens(messages)
```

### `estimate_tool_schema_tokens(tools: List[Dict]) -> int`

Estimate tokens for tool/function schemas.

```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
tools = [{"name": "search", "description": "Search the web"}]
tokens = estimate_tool_schema_tokens(tools)
```

### `TokenEstimatorImpl`

Class-based estimator with caching:

```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from praisonaiagents import TokenEstimatorImpl

estimator = TokenEstimatorImpl()
tokens = estimator.estimate("Some text")
```

### `get_estimator() -> TokenEstimatorImpl`

Get a singleton estimator instance:

```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from praisonaiagents import get_estimator

estimator = get_estimator()
tokens = estimator.estimate("Text to estimate")
```

## Accuracy Considerations

The heuristic estimator is designed for speed over perfect accuracy:

| Scenario        | Accuracy |
| --------------- | -------- |
| English text    | \~90-95% |
| Code            | \~85-90% |
| Mixed content   | \~85-90% |
| Non-ASCII heavy | \~80-85% |

For budget decisions, the estimator adds a small safety margin to prevent underestimation.

## Performance

* **Speed**: \< 1ms for 100K characters
* **Memory**: O(1) - no caching required
* **No API calls**: Works completely offline

## Integration with Budgeter

```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from praisonaiagents import (
    ContextBudgeter,
    estimate_messages_tokens,
)

budgeter = ContextBudgeter(model="gpt-4o-mini")
budget = budgeter.allocate()

# Check if messages fit in budget
messages = [...]  # Your conversation
tokens = estimate_messages_tokens(messages)

if tokens > budget.usable * 0.8:
    print("Warning: Approaching context limit!")
```

## Best Practices

<AccordionGroup>
  <Accordion title="Estimate before each LLM call">
    Check message totals against the budget to trigger compaction proactively.
  </Accordion>

  <Accordion title="Use model-aware estimators">
    Pass the model name so tiktoken or provider-specific counting applies when available.
  </Accordion>

  <Accordion title="Warn at 80% utilisation">
    Act before hard limits — retrieval trimming and summarisation need headroom.
  </Accordion>

  <Accordion title="Validate estimates in CI">
    Run validated mode on fixture conversations to catch drift in token counting.
  </Accordion>
</AccordionGroup>

## Related

<CardGroup cols={2}>
  <Card title="Context Ledger" icon="book" href="/docs/features/context-ledger">
    Track tokens by segment
  </Card>

  <Card title="Context Budgeter" icon="coins" href="/docs/features/context-budgeter">
    Allocate token budgets
  </Card>
</CardGroup>