Chunking Strategies Overview

PraisonAI integrates chonkie, a high-performance chunking library, to provide flexible document processing strategies.

Quick Start

from praisonaiagents import Agent

# Agent with semantic chunking
agent = Agent(
    instructions="Answer questions from documents.",
    knowledge={
        "sources": ["research.pdf"],
        "chunker": {
            "type": "semantic",
            "chunk_size": 512
        }
    }
)

response = agent.start("What are the key findings?")

from praisonaiagents.knowledge.chunking import Chunking

# Create a chunker
chunker = Chunking(
    chunker_type="recursive",
    chunk_size=512,
    chunk_overlap=128
)

# Chunk some text
chunks = chunker.chunk("Your long document text here...")
for chunk in chunks:
    print(chunk.text)

Available Strategies

Strategy	Alias	Best For	Speed
Token	`token`	Fixed-size chunks	⚡ Fast
Sentence	`sentence`	Natural boundaries	⚡ Fast
Recursive	`recursive`	Structured documents	⚡ Fast
Semantic	`semantic`	Topic segmentation	🔄 Medium
SDPM	`sdpm`	Research papers	🔄 Medium
Late	`late`	Better embeddings	🔄 Medium

Choosing a Strategy

Agent Configuration

Simplest (Default Strategy)

from praisonaiagents import Agent

# Uses token chunking by default
agent = Agent(
    instructions="Answer from documents.",
    knowledge=["docs/"]  # Default chunking
)

With Chunking Config

from praisonaiagents import Agent

agent = Agent(
    instructions="Answer from documents.",
    knowledge={
        "sources": ["research.pdf", "data/"],
        "chunker": {
            "type": "semantic",       # Strategy type
            "chunk_size": 512,        # Tokens per chunk
            "chunk_overlap": 128,     # Overlap between chunks
            "embedding_model": "all-MiniLM-L6-v2"  # For semantic/sdpm/late
        }
    }
)

All Chunker Options

Option	Type	Default	Description
`type`	str	`"token"`	Chunker type: token, sentence, recursive, semantic, sdpm, late
`chunk_size`	int	512	Target tokens per chunk
`chunk_overlap`	int	128	Overlap between chunks
`tokenizer_or_token_counter`	str	`"gpt2"`	Tokenizer for counting
`embedding_model`	str	auto	Embedding model (semantic/sdpm/late only)

Strategy Details

Token Chunking

Fixed-size token chunks. Fast and predictable.

Sentence Chunking

Split at sentence boundaries. Natural flow.

Recursive Chunking

Hierarchical splitting. Great for markdown.

Semantic Chunking

Similarity-based splits. Topic coherence.

Installation

Chunking requires the knowledge extra:

pip install "praisonaiagents[knowledge]"

This installs the chonkie library automatically.

If you skip the [knowledge] extra and select a semantic-family chunker (semantic, sdpm, or late — semantic is the default), the first Knowledge.add(...) or agent.start(...) call raises:

ImportError: chonkie package not found. Please install it using: pip install 'praisonaiagents[knowledge]'

Fix by installing the extra, or by choosing a chunker that does not need chonkie’s embeddings (sentence, paragraph, fixed, token). Since PraisonAI PR #5105.

Getting Started

Install

Core Concepts

Guides

Features

Models

Databases

Observability

Memory

Telemetry & Analytics

Knowledge

RAG

Persistence

Tools

Other Features

Developers

Configuration

Best Practices

Getting Started (No Code)

Chunking Strategies Overview

Quick Start

Available Strategies

Choosing a Strategy

Agent Configuration

Simplest (Default Strategy)

With Chunking Config

All Chunker Options

Strategy Details

Token Chunking

Sentence Chunking

Recursive Chunking

Semantic Chunking

Installation

​Quick Start

​Available Strategies

​Choosing a Strategy

​Agent Configuration

​Simplest (Default Strategy)

​With Chunking Config

​All Chunker Options

​Strategy Details

Token Chunking

Sentence Chunking

Recursive Chunking

Semantic Chunking

​Installation

Quick Start

Available Strategies

Choosing a Strategy

Agent Configuration

Simplest (Default Strategy)

With Chunking Config

All Chunker Options

Strategy Details

Installation