> ## Documentation Index
> Fetch the complete documentation index at: https://praison.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Token Utilities Module

> Token counting and auto-chunking utilities for LLM evaluation

# Token Utilities

Lightweight token counting and context-aware chunking utilities. Works without external dependencies, with optional accuracy boost from `litellm`.

```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
flowchart LR
    subgraph Input
        A[Text Content]
    end
    
    subgraph TokenUtils["Token Utilities"]
        B[estimate_tokens]
        C[count_tokens]
        D[get_context_length]
        E[needs_chunking]
    end
    
    subgraph Output
        F{Chunk?}
        G[Process Whole]
        H[Split & Process]
    end
    
    A --> C
    C --> E
    D --> E
    E --> F
    F -->|No| G
    F -->|Yes| H
    
    style A fill:#8B0000,color:#fff
    style G fill:#8B0000,color:#fff
    style H fill:#8B0000,color:#fff
    style B fill:#189AB4,color:#fff
    style C fill:#189AB4,color:#fff
    style D fill:#189AB4,color:#fff
    style E fill:#189AB4,color:#fff
```

## Quick Start

<CodeGroup>
  ```python Basic Usage theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
  from praisonaiagents.eval import estimate_tokens, needs_chunking

  # Estimate tokens (no dependencies)
  tokens = estimate_tokens("Hello world")
  print(tokens)  # 2

  # Check if chunking needed
  if needs_chunking(large_text, model="gpt-4o-mini"):
      # Split into chunks
      pass
  ```

  ```python With Details theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
  from praisonaiagents.eval import needs_chunking

  # Get detailed chunking info
  info = needs_chunking(
      text=my_content,
      model="gpt-4o-mini",
      safety_margin=0.8,
      return_info=True
  )

  print(f"Needs chunking: {info['needs_chunking']}")
  print(f"Token count: {info['estimated_tokens']}")
  print(f"Context window: {info['context_length']}")
  print(f"Utilization: {info['utilization']:.1%}")
  ```
</CodeGroup>

## Functions

### estimate\_tokens

Estimate token count without external dependencies.

<Tip>
  Uses OpenAI's heuristics: \~4 characters per token, \~0.75 tokens per word.
</Tip>

```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from praisonaiagents.eval import estimate_tokens

# Default (conservative - uses max of char/word methods)
tokens = estimate_tokens("Hello world")  # 2

# Character-based only
tokens = estimate_tokens("Hello world", method="chars")  # 2

# Word-based only  
tokens = estimate_tokens("Hello world", method="words")  # 2

# Average of both methods
tokens = estimate_tokens("Hello world", method="average")  # 2
```

<ParamField path="text" type="str" required>
  Text to estimate tokens for
</ParamField>

<ParamField path="method" type="str" default="max">
  Estimation method: `chars`, `words`, `max`, `min`, `average`
</ParamField>

### count\_tokens

Count tokens accurately using `litellm` if available, otherwise estimate.

```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from praisonaiagents.eval import count_tokens

# Uses litellm for accuracy (if installed)
tokens = count_tokens("Hello world", model="gpt-4o-mini")  # 2

# Force estimation (skip litellm)
tokens = count_tokens("Hello world", use_litellm=False)  # 2
```

<ParamField path="text" type="str" required>
  Text to count tokens for
</ParamField>

<ParamField path="model" type="str" default="gpt-4o-mini">
  Model name for tokenizer selection
</ParamField>

<ParamField path="use_litellm" type="bool" default="True">
  Whether to try litellm for accurate counting
</ParamField>

<Note>
  For very large text (>100K chars), automatically falls back to estimation for performance.
</Note>

### get\_context\_length

Get the context window size for a model.

```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from praisonaiagents.eval import get_context_length

# OpenAI — new 1M-context families
print(get_context_length("gpt-5"))            # 1047576
print(get_context_length("gpt-4.1"))          # 1047576

# OpenAI — existing 128K family
print(get_context_length("gpt-4o-mini"))      # 128000
print(get_context_length("gpt-4o"))           # 128000

# Anthropic (dated snapshot still resolvable via eval extras)
print(get_context_length("claude-3-5-sonnet-20241022"))  # 200000

# Google
print(get_context_length("gemini-1.5-pro"))   # 2097152

# Versioned name → partial match resolves to the specific entry, not the shorter prefix
print(get_context_length("gpt-4-32k-0314"))   # 32768  (not 8192)
```

<Note>
  Lookups use partial matching by descending key length, so a specific versioned name (e.g. `gpt-4-32k-0314`) resolves to the more specific entry (`gpt-4-32k` → 32768) before falling back to a shorter prefix (`gpt-4` → 8192). The shared budgeter path consults litellm's `model_cost` registry first — the 128,000-token default only applies when both litellm and this static table miss. See [How the context window is resolved](/docs/features/context-budgeter#how-the-context-window-is-resolved).
</Note>

<ParamField path="model" type="str" required>
  Model name (e.g., "gpt-4o-mini", "claude-3-5-sonnet-20241022")
</ParamField>

<ParamField path="use_litellm" type="bool" default="True">
  Whether to try litellm for accurate info
</ParamField>

<Accordion title="Supported Models">
  The eval path resolves the union of the canonical `MODEL_LIMITS` and the eval-only `_EVAL_EXTRA_LENGTHS`.

  | Model                                                                     | Context Window |
  | ------------------------------------------------------------------------- | -------------- |
  | gpt-5, gpt-5-mini, gpt-5-nano                                             | 1,047,576      |
  | gpt-4.1, gpt-4.1-mini, gpt-4.1-nano                                       | 1,047,576      |
  | gpt-4o, gpt-4o-mini                                                       | 128,000        |
  | gpt-4-turbo, gpt-4-turbo-preview                                          | 128,000        |
  | gpt-4                                                                     | 8,192          |
  | gpt-4-32k                                                                 | 32,768         |
  | gpt-3.5-turbo, gpt-3.5-turbo-16k                                          | 16,385         |
  | o3, o3-mini, o4-mini                                                      | 200,000        |
  | o1                                                                        | 200,000        |
  | o1-mini, o1-preview                                                       | 128,000        |
  | claude-3-5-sonnet, claude-3-5-haiku                                       | 200,000        |
  | claude-3-opus, claude-3-sonnet, claude-3-haiku                            | 200,000        |
  | claude-3-5-sonnet-20241022, claude-3-5-sonnet-latest                      | 200,000        |
  | claude-3-5-haiku-20241022                                                 | 200,000        |
  | claude-3-opus-20240229, claude-3-sonnet-20240229, claude-3-haiku-20240307 | 200,000        |
  | claude-2.1                                                                | 200,000        |
  | claude-2                                                                  | 100,000        |
  | gemini-1.5-pro                                                            | 2,097,152      |
  | gemini-1.5-flash, gemini-1.5-flash-8b                                     | 1,048,576      |
  | gemini-2.0-flash, gemini-2.0-flash-exp                                    | 1,048,576      |
  | gemini-pro                                                                | 32,760         |
  | mistral-large-latest                                                      | 128,000        |
  | mistral-medium-latest, mistral-small-latest, codestral-latest             | 32,000         |
  | deepseek-chat, deepseek-coder                                             | 64,000         |
  | llama-3.3-70b-versatile, llama-3.1-70b-versatile, llama-3.1-8b-instant    | 128,000        |
  | mixtral-8x7b-32768                                                        | 32,768         |
</Accordion>

### needs\_chunking

Determine if text needs to be chunked for the given model.

```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from praisonaiagents.eval import needs_chunking

# Simple check
if needs_chunking(large_text, "gpt-4o-mini"):
    print("Text too large, needs chunking")

# With custom safety margin (70% of context)
needs_chunk = needs_chunking(
    text=large_text,
    model="gpt-4o-mini",
    safety_margin=0.7
)

# Get detailed info
info = needs_chunking(large_text, "gpt-4o-mini", return_info=True)
# Returns: {
#   'needs_chunking': True,
#   'estimated_tokens': 150000,
#   'context_length': 128000,
#   'available_tokens': 102400,
#   'safety_margin': 0.8,
#   'utilization': 1.17
# }
```

<ParamField path="text" type="str" required>
  Text to evaluate
</ParamField>

<ParamField path="model" type="str" default="gpt-4o-mini">
  Model name to check context window for
</ParamField>

<ParamField path="safety_margin" type="float" default="0.8">
  Fraction of context window to use (leaves room for prompts)
</ParamField>

<ParamField path="return_info" type="bool" default="False">
  If True, return detailed info dict instead of bool
</ParamField>

### get\_recommended\_chunk\_size

Get recommended chunk size in characters for a model.

```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from praisonaiagents.eval import get_recommended_chunk_size

# Default (5 chunks)
chunk_size = get_recommended_chunk_size("gpt-4o-mini")  # ~10000

# More chunks for finer granularity
chunk_size = get_recommended_chunk_size("gpt-4o-mini", target_chunks=10)
```

<ParamField path="model" type="str" default="gpt-4o-mini">
  Model name
</ParamField>

<ParamField path="target_chunks" type="int" default="5">
  Target number of chunks
</ParamField>

<ParamField path="safety_margin" type="float" default="0.8">
  Fraction of context to use
</ParamField>

## CLI Integration

Use auto-chunking with the recipe judge command:

```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
# Auto-detect if chunking is needed
praisonai recipe judge run-abc123 --auto-chunk

# Force chunked evaluation
praisonai recipe judge run-abc123 --chunked --chunk-size 8000
```

<Steps>
  <Step title="Run Recipe">
    Execute your recipe to generate a trace
  </Step>

  <Step title="Judge with Auto-Chunk">
    Use `--auto-chunk` to automatically handle large outputs
  </Step>

  <Step title="Review Results">
    Get accurate evaluation even for large content
  </Step>
</Steps>

## How It Works

```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
flowchart TD
    A[Input Text] --> B{litellm available?}
    B -->|Yes| C[litellm.token_counter]
    B -->|No| D[estimate_tokens]
    C --> E[Token Count]
    D --> E
    
    F[Model Name] --> G{litellm available?}
    G -->|Yes| H[litellm.model_cost]
    G -->|No| I[DEFAULT_CONTEXT_LENGTHS]
    H --> J[Context Length]
    I --> J
    
    E --> K{tokens > context × margin?}
    J --> K
    K -->|Yes| L[Chunk Content]
    K -->|No| M[Process Whole]
    
    style A fill:#8B0000,color:#fff
    style L fill:#8B0000,color:#fff
    style M fill:#8B0000,color:#fff
    style C fill:#189AB4,color:#fff
    style D fill:#189AB4,color:#fff
    style H fill:#189AB4,color:#fff
    style I fill:#189AB4,color:#fff
```

<Accordion title="Token Estimation Heuristics">
  Based on OpenAI's guidance for English text:

  * **1 token ≈ 4 characters**
  * **1 token ≈ 0.75 words**
  * **100 tokens ≈ 75 words**

  The `max` method (default) uses the larger of character-based and word-based estimates for conservative results.
</Accordion>

## Related

* [Eval Module](/docs/sdk/praisonaiagents/eval/eval) - Evaluation framework
* [Recipe Judge CLI](/docs/cli/eval) - CLI evaluation commands