Token Utilities

Lightweight token counting and context-aware chunking utilities. Works without external dependencies, with optional accuracy boost from litellm.

Quick Start

from praisonaiagents.eval import estimate_tokens, needs_chunking

# Estimate tokens (no dependencies)
tokens = estimate_tokens("Hello world")
print(tokens)  # 2

# Check if chunking needed
if needs_chunking(large_text, model="gpt-4o-mini"):
    # Split into chunks
    pass

from praisonaiagents.eval import needs_chunking

# Get detailed chunking info
info = needs_chunking(
    text=my_content,
    model="gpt-4o-mini",
    safety_margin=0.8,
    return_info=True
)

print(f"Needs chunking: {info['needs_chunking']}")
print(f"Token count: {info['estimated_tokens']}")
print(f"Context window: {info['context_length']}")
print(f"Utilization: {info['utilization']:.1%}")

Functions

estimate_tokens

Estimate token count without external dependencies.

Uses OpenAI’s heuristics: ~4 characters per token, ~0.75 tokens per word.

from praisonaiagents.eval import estimate_tokens

# Default (conservative - uses max of char/word methods)
tokens = estimate_tokens("Hello world")  # 2

# Character-based only
tokens = estimate_tokens("Hello world", method="chars")  # 2

# Word-based only  
tokens = estimate_tokens("Hello world", method="words")  # 2

# Average of both methods
tokens = estimate_tokens("Hello world", method="average")  # 2

str

required

Text to estimate tokens for

str

default:"max"

Estimation method: chars, words, max, min, average

count_tokens

Count tokens accurately using litellm if available, otherwise estimate.

from praisonaiagents.eval import count_tokens

# Uses litellm for accuracy (if installed)
tokens = count_tokens("Hello world", model="gpt-4o-mini")  # 2

# Force estimation (skip litellm)
tokens = count_tokens("Hello world", use_litellm=False)  # 2

str

required

Text to count tokens for

str

default:"gpt-4o-mini"

Model name for tokenizer selection

bool

default:"True"

Whether to try litellm for accurate counting

For very large text (>100K chars), automatically falls back to estimation for performance.

get_context_length

Get the context window size for a model.

from praisonaiagents.eval import get_context_length

# OpenAI — new 1M-context families
print(get_context_length("gpt-5"))            # 1047576
print(get_context_length("gpt-4.1"))          # 1047576

# OpenAI — existing 128K family
print(get_context_length("gpt-4o-mini"))      # 128000
print(get_context_length("gpt-4o"))           # 128000

# Anthropic (dated snapshot still resolvable via eval extras)
print(get_context_length("claude-3-5-sonnet-20241022"))  # 200000

# Google
print(get_context_length("gemini-1.5-pro"))   # 2097152

# Versioned name → partial match resolves to the specific entry, not the shorter prefix
print(get_context_length("gpt-4-32k-0314"))   # 32768  (not 8192)

Lookups use partial matching by descending key length, so a specific versioned name (e.g. gpt-4-32k-0314) resolves to the more specific entry (gpt-4-32k → 32768) before falling back to a shorter prefix (gpt-4 → 8192). The shared budgeter path consults litellm’s model_cost registry first — the 128,000-token default only applies when both litellm and this static table miss. See How the context window is resolved.

str

required

Model name (e.g., “gpt-4o-mini”, “claude-3-5-sonnet-20241022”)

bool

default:"True"

Whether to try litellm for accurate info

Supported Models

The eval path resolves the union of the canonical MODEL_LIMITS and the eval-only _EVAL_EXTRA_LENGTHS.

Model	Context Window
gpt-5, gpt-5-mini, gpt-5-nano	1,047,576
gpt-4.1, gpt-4.1-mini, gpt-4.1-nano	1,047,576
gpt-4o, gpt-4o-mini	128,000
gpt-4-turbo, gpt-4-turbo-preview	128,000
gpt-4	8,192
gpt-4-32k	32,768
gpt-3.5-turbo, gpt-3.5-turbo-16k	16,385
o3, o3-mini, o4-mini	200,000
o1	200,000
o1-mini, o1-preview	128,000
claude-3-5-sonnet, claude-3-5-haiku	200,000
claude-3-opus, claude-3-sonnet, claude-3-haiku	200,000
claude-3-5-sonnet-20241022, claude-3-5-sonnet-latest	200,000
claude-3-5-haiku-20241022	200,000
claude-3-opus-20240229, claude-3-sonnet-20240229, claude-3-haiku-20240307	200,000
claude-2.1	200,000
claude-2	100,000
gemini-1.5-pro	2,097,152
gemini-1.5-flash, gemini-1.5-flash-8b	1,048,576
gemini-2.0-flash, gemini-2.0-flash-exp	1,048,576
gemini-pro	32,760
mistral-large-latest	128,000
mistral-medium-latest, mistral-small-latest, codestral-latest	32,000
deepseek-chat, deepseek-coder	64,000
llama-3.3-70b-versatile, llama-3.1-70b-versatile, llama-3.1-8b-instant	128,000
mixtral-8x7b-32768	32,768

needs_chunking

Determine if text needs to be chunked for the given model.

from praisonaiagents.eval import needs_chunking

# Simple check
if needs_chunking(large_text, "gpt-4o-mini"):
    print("Text too large, needs chunking")

# With custom safety margin (70% of context)
needs_chunk = needs_chunking(
    text=large_text,
    model="gpt-4o-mini",
    safety_margin=0.7
)

# Get detailed info
info = needs_chunking(large_text, "gpt-4o-mini", return_info=True)
# Returns: {
#   'needs_chunking': True,
#   'estimated_tokens': 150000,
#   'context_length': 128000,
#   'available_tokens': 102400,
#   'safety_margin': 0.8,
#   'utilization': 1.17
# }

str

required

Text to evaluate

str

default:"gpt-4o-mini"

Model name to check context window for

float

default:"0.8"

Fraction of context window to use (leaves room for prompts)

bool

default:"False"

If True, return detailed info dict instead of bool

get_recommended_chunk_size

Get recommended chunk size in characters for a model.

from praisonaiagents.eval import get_recommended_chunk_size

# Default (5 chunks)
chunk_size = get_recommended_chunk_size("gpt-4o-mini")  # ~10000

# More chunks for finer granularity
chunk_size = get_recommended_chunk_size("gpt-4o-mini", target_chunks=10)

str

default:"gpt-4o-mini"

Model name

int

default:"5"

Target number of chunks

float

default:"0.8"

Fraction of context to use

CLI Integration

Use auto-chunking with the recipe judge command:

# Auto-detect if chunking is needed
praisonai recipe judge run-abc123 --auto-chunk

# Force chunked evaluation
praisonai recipe judge run-abc123 --chunked --chunk-size 8000

Run Recipe

Execute your recipe to generate a trace

Judge with Auto-Chunk

Use --auto-chunk to automatically handle large outputs

Review Results

Get accurate evaluation even for large content

How It Works

Token Estimation Heuristics

Based on OpenAI’s guidance for English text:

1 token ≈ 4 characters
1 token ≈ 0.75 words
100 tokens ≈ 75 words

The max method (default) uses the larger of character-based and word-based estimates for conservative results.

Eval Module - Evaluation framework
Recipe Judge CLI - CLI evaluation commands

Guide

Reference

Token Utilities Module

Token Utilities

Quick Start

Functions

estimate_tokens

count_tokens

get_context_length

needs_chunking

get_recommended_chunk_size

CLI Integration

How It Works

​Token Utilities

​Quick Start

​Functions

​estimate_tokens

​count_tokens

​get_context_length

​needs_chunking

​get_recommended_chunk_size

​CLI Integration

​How It Works

​Related

Token Utilities

Quick Start

Functions

estimate_tokens

count_tokens

get_context_length

needs_chunking

get_recommended_chunk_size

CLI Integration

How It Works

Related