Token Utilities Module

Token counting and auto-chunking utilities for LLM evaluation

Token Utilities

Lightweight token counting and context-aware chunking utilities. Works without external dependencies, with optional accuracy boost from litellm.

Diagram

Quick Start

Basic Usage

from praisonaiagents.eval import estimate_tokens, needs_chunking

# Estimate tokens (no dependencies)
tokens = estimate_tokens("Hello world")
print(tokens)  # 2

# Check if chunking needed
if needs_chunking(large_text, model="gpt-4o-mini"):
    # Split into chunks
    pass

With Details

from praisonaiagents.eval import needs_chunking

# Get detailed chunking info
info = needs_chunking(
    text=my_content,
    model="gpt-4o-mini",
    safety_margin=0.8,
    return_info=True
)

print(f"Needs chunking: {info['needs_chunking']}")
print(f"Token count: {info['estimated_tokens']}")
print(f"Context window: {info['context_length']}")
print(f"Utilization: {info['utilization']:.1%}")

Functions

estimate_tokens

Estimate token count without external dependencies.

python

from praisonaiagents.eval import estimate_tokens

# Default (conservative - uses max of char/word methods)
tokens = estimate_tokens("Hello world")  # 2

# Character-based only
tokens = estimate_tokens("Hello world", method="chars")  # 2

# Word-based only  
tokens = estimate_tokens("Hello world", method="words")  # 2

# Average of both methods
tokens = estimate_tokens("Hello world", method="average")  # 2

text str

Text to estimate tokens for

method str

Estimation method: chars, words, max, min, average

count_tokens

Count tokens accurately using litellm if available, otherwise estimate.

python

from praisonaiagents.eval import count_tokens

# Uses litellm for accuracy (if installed)
tokens = count_tokens("Hello world", model="gpt-4o-mini")  # 2

# Force estimation (skip litellm)
tokens = count_tokens("Hello world", use_litellm=False)  # 2

text str

Text to count tokens for

model str

Model name for tokenizer selection

use_litellm bool

Whether to try litellm for accurate counting

get_context_length

Get the context window size for a model.

python

from praisonaiagents.eval import get_context_length

# OpenAI — new 1M-context families
print(get_context_length("gpt-5"))            # 1047576
print(get_context_length("gpt-4.1"))          # 1047576

# OpenAI — existing 128K family
print(get_context_length("gpt-4o-mini"))      # 128000
print(get_context_length("gpt-4o"))           # 128000

# Anthropic (dated snapshot still resolvable via eval extras)
print(get_context_length("claude-3-5-sonnet-20241022"))  # 200000

# Google
print(get_context_length("gemini-1.5-pro"))   # 2097152

# Versioned name → partial match resolves to the specific entry, not the shorter prefix
print(get_context_length("gpt-4-32k-0314"))   # 32768  (not 8192)

model str

Model name (e.g., “gpt-4o-mini”, “claude-3-5-sonnet-20241022”)

use_litellm bool

Whether to try litellm for accurate info

Supported Models

The eval path resolves the union of the canonical MODEL_LIMITS and the eval-only _EVAL_EXTRA_LENGTHS.

Model	Context Window
gpt-5, gpt-5-mini, gpt-5-nano	1,047,576
gpt-4.1, gpt-4.1-mini, gpt-4.1-nano	1,047,576
gpt-4o, gpt-4o-mini	128,000
gpt-4-turbo, gpt-4-turbo-preview	128,000
gpt-4	8,192
gpt-4-32k	32,768
gpt-3.5-turbo, gpt-3.5-turbo-16k	16,385
o3, o3-mini, o4-mini	200,000
o1	200,000
o1-mini, o1-preview	128,000
claude-3-5-sonnet, claude-3-5-haiku	200,000
claude-3-opus, claude-3-sonnet, claude-3-haiku	200,000
claude-3-5-sonnet-20241022, claude-3-5-sonnet-latest	200,000
claude-3-5-haiku-20241022	200,000
claude-3-opus-20240229, claude-3-sonnet-20240229, claude-3-haiku-20240307	200,000
claude-2.1	200,000
claude-2	100,000
gemini-1.5-pro	2,097,152
gemini-1.5-flash, gemini-1.5-flash-8b	1,048,576
gemini-2.0-flash, gemini-2.0-flash-exp	1,048,576
gemini-pro	32,760
mistral-large-latest	128,000
mistral-medium-latest, mistral-small-latest, codestral-latest	32,000
deepseek-chat, deepseek-coder	64,000
llama-3.3-70b-versatile, llama-3.1-70b-versatile, llama-3.1-8b-instant	128,000
mixtral-8x7b-32768	32,768

needs_chunking

Determine if text needs to be chunked for the given model.

python

from praisonaiagents.eval import needs_chunking

# Simple check
if needs_chunking(large_text, "gpt-4o-mini"):
    print("Text too large, needs chunking")

# With custom safety margin (70% of context)
needs_chunk = needs_chunking(
    text=large_text,
    model="gpt-4o-mini",
    safety_margin=0.7
)

# Get detailed info
info = needs_chunking(large_text, "gpt-4o-mini", return_info=True)
# Returns: {
#   'needs_chunking': True,
#   'estimated_tokens': 150000,
#   'context_length': 128000,
#   'available_tokens': 102400,
#   'safety_margin': 0.8,
#   'utilization': 1.17
# }

text str

Text to evaluate

model str

Model name to check context window for

safety_margin float

Fraction of context window to use (leaves room for prompts)

return_info bool

If True, return detailed info dict instead of bool

get_recommended_chunk_size

Get recommended chunk size in characters for a model.

python

from praisonaiagents.eval import get_recommended_chunk_size

# Default (5 chunks)
chunk_size = get_recommended_chunk_size("gpt-4o-mini")  # ~10000

# More chunks for finer granularity
chunk_size = get_recommended_chunk_size("gpt-4o-mini", target_chunks=10)

model str

Model name

target_chunks int

Target number of chunks

safety_margin float

Fraction of context to use

CLI Integration

Use auto-chunking with the recipe judge command:

bash

# Auto-detect if chunking is needed
praisonai recipe judge run-abc123 --auto-chunk

# Force chunked evaluation
praisonai recipe judge run-abc123 --chunked --chunk-size 8000

Run Recipe
Execute your recipe to generate a trace
Judge with Auto-Chunk
Use --auto-chunk to automatically handle large outputs
Review Results
Get accurate evaluation even for large content

How It Works

Diagram

Token Estimation Heuristics

Based on OpenAI’s guidance for English text:

1 token ≈ 4 characters
1 token ≈ 0.75 words
100 tokens ≈ 75 words

The max method (default) uses the larger of character-based and word-based estimates for conservative results.

Eval Module – Evaluation framework
Recipe Judge CLI – CLI evaluation commands

Token Utilities Module

Token Utilities

Quick Start

Functions

estimate_tokens

count_tokens

get_context_length

needs_chunking

get_recommended_chunk_size

CLI Integration

How It Works

Related