Chunking Strategies

PraisonAI integrates chonkie for high-performance document chunking.

from praisonaiagents import Agent

agent = Agent(
    name="researcher",
    instructions="Use chunked knowledge for long documents",
    knowledge=True,
)

agent.start("Find the section about deployment limits")

The user queries long sources; chunking controls what reaches retrieval. PraisonAI integrates chonkie for high-performance document chunking.

Quick Start

Default chunking on an agent

from praisonaiagents import Agent

# Default chunking (token-based)
agent = Agent(
    instructions="Answer questions from documents.",
    knowledge=["research.pdf", "docs/"]
)

response = agent.start("What are the key findings?")

from praisonaiagents import Agent

agent = Agent(
    instructions="Answer questions from documents.",
    knowledge={
        "sources": ["research.pdf"],
        "chunker": {
            "type": "semantic",       # token, sentence, recursive, semantic, sdpm, late
            "chunk_size": 512,
            "chunk_overlap": 128
        }
    }
)

response = agent.start("What methodology was used?")

from praisonaiagents import Chunking

# Create chunker
chunker = Chunking(
    chunker_type="recursive",
    chunk_size=512,
    chunk_overlap=128
)

# Chunk text
chunks = chunker.chunk("Your document content here...")
for chunk in chunks:
    print(f"Tokens: {chunk.token_count}")
    print(chunk.text[:100])

Available Strategies

Strategy	Best For	Speed
`token`	Fixed-size chunks	⚡ Fastest
`sentence`	Natural boundaries	⚡ Fast
`recursive`	Structured docs (markdown)	⚡ Fast
`semantic`	Topic segmentation	🔄 Medium
`sdpm`	Research papers	🔄 Medium
`late`	Best embeddings	🔄 Medium

Token Chunking

Fixed-size token chunks. Fast and predictable.

Sentence Chunking

Split at sentence boundaries. Natural flow.

Recursive Chunking

Hierarchical splitting. Great for markdown.

Semantic Chunking

Similarity-based splits. Topic coherence.

Chunker Configuration

All Parameters

Parameter	Type	Default	Applies To
`type`	str	`"token"`	All
`chunk_size`	int	512	All
`chunk_overlap`	int	128	token, sentence
`tokenizer_or_token_counter`	str	`"gpt2"`	token, sentence, recursive
`embedding_model`	str	auto	semantic, sdpm, late

Strategy Examples

Token
Sentence
Recursive
Semantic

agent = Agent(
    instructions="Process documents.",
    knowledge={
        "sources": ["docs/"],
        "chunker": {
            "type": "token",
            "chunk_size": 256,
            "chunk_overlap": 50
        }
    }
)

agent = Agent(
    instructions="Process articles.",
    knowledge={
        "sources": ["articles/"],
        "chunker": {
            "type": "sentence",
            "chunk_size": 512
        }
    }
)

agent = Agent(
    instructions="Process markdown docs.",
    knowledge={
        "sources": ["README.md", "docs/"],
        "chunker": {
            "type": "recursive",
            "chunk_size": 512
        }
    }
)

agent = Agent(
    instructions="Process research papers.",
    knowledge={
        "sources": ["papers/"],
        "chunker": {
            "type": "semantic",
            "chunk_size": 512,
            "embedding_model": "all-MiniLM-L6-v2"
        }
    }
)

Which Strategy Should I Use?

How It Works

The user adds documents; the chunking engine splits them by the chosen strategy, embeds each chunk, and stores it for retrieval.

Installation

pip install "praisonaiagents[knowledge]"

This installs the chonkie library automatically.

Best Practices

Match chunk size to retrieval use case

Smaller chunks improve precision for Q&A; larger chunks preserve narrative for summarisation tasks.

Install the knowledge extra

Run pip install "praisonaiagents[knowledge]" so chonkie and indexing backends load only when needed.

Test overlap on sample docs

Tune overlap on a representative document before indexing an entire corpus.

Pair with smart retrieval

Combine chunking with hybrid search and reranking for better recall on long knowledge bases.

Knowledge Base

Configure knowledge sources and retrieval

RAG Agents

Build retrieval-augmented agents

Bot Platform Capabilities

How platform capabilities drive this feature

Getting Started

Install

Core Concepts

Guides

Features

Models

Databases

Observability

Memory

Telemetry & Analytics

Knowledge

RAG

Persistence

Tools

Other Features

Developers

Configuration

Best Practices

Getting Started (No Code)

Chunking Strategies

Quick Start

Available Strategies

Token Chunking

Sentence Chunking

Recursive Chunking

Semantic Chunking

Chunker Configuration

All Parameters

Strategy Examples

Which Strategy Should I Use?

How It Works

Installation

Best Practices

Knowledge Base

RAG Agents

Bot Platform Capabilities

​Quick Start

​Available Strategies

Token Chunking

Sentence Chunking

Recursive Chunking

Semantic Chunking

​Chunker Configuration

​All Parameters

​Strategy Examples

​Which Strategy Should I Use?

​How It Works

​Installation

​Best Practices

​Related

Knowledge Base

RAG Agents

Bot Platform Capabilities

Quick Start

Available Strategies

Chunker Configuration

All Parameters

Strategy Examples

Which Strategy Should I Use?

How It Works

Installation

Best Practices

Related