Praison AI

Chunking Strategies Overview

Understanding document chunking strategies for optimal RAG performance

PraisonAI integrates chonkie, a high-performance chunking library, to provide flexible document processing strategies.

Quick Start

Agent with Chunking
from praisonaiagents import Agent

# Agent with semantic chunking
agent = Agent(
    instructions="Answer questions from documents.",
    knowledge={
        "sources": ["research.pdf"],
        "chunker": {
            "type": "semantic",
            "chunk_size": 512
        }
    }
)

response = agent.start("What are the key findings?")
Direct Chunking API
from praisonaiagents.knowledge.chunking import Chunking

# Create a chunker
chunker = Chunking(
    chunker_type="recursive",
    chunk_size=512,
    chunk_overlap=128
)

# Chunk some text
chunks = chunker.chunk("Your long document text here...")
for chunk in chunks:
    print(chunk.text)

Available Strategies

StrategyAliasBest ForSpeed
TokentokenFixed-size chunks⚡ Fast
SentencesentenceNatural boundaries⚡ Fast
RecursiverecursiveStructured documents⚡ Fast
SemanticsemanticTopic segmentation🔄 Medium
SDPMsdpmResearch papers🔄 Medium
LatelateBetter embeddings🔄 Medium

Choosing a Strategy

Diagram

Yes

No

Yes

No

Yes

No

Document Type?

Structured?

Use recursive

Need Topic Segmentation?

Use semantic

Speed Important?

Use token

Use sentence

Agent Configuration

Simplest (Default Strategy)

python
from praisonaiagents import Agent

# Uses token chunking by default
agent = Agent(
    instructions="Answer from documents.",
    knowledge=["docs/"]  # Default chunking
)

With Chunking Config

python
from praisonaiagents import Agent

agent = Agent(
    instructions="Answer from documents.",
    knowledge={
        "sources": ["research.pdf", "data/"],
        "chunker": {
            "type": "semantic",       # Strategy type
            "chunk_size": 512,        # Tokens per chunk
            "chunk_overlap": 128,     # Overlap between chunks
            "embedding_model": "all-MiniLM-L6-v2"  # For semantic/sdpm/late
        }
    }
)

All Chunker Options

OptionTypeDefaultDescription
typestr"token"Chunker type: token, sentence, recursive, semantic, sdpm, late
chunk_sizeint512Target tokens per chunk
chunk_overlapint128Overlap between chunks
tokenizer_or_token_counterstr"gpt2"Tokenizer for counting
embedding_modelstrautoEmbedding model (semantic/sdpm/late only)

Strategy Details

Token Chunking

Fixed-size token chunks. Fast and predictable.

 Sentence Chunking

Split at sentence boundaries. Natural flow.

 Recursive Chunking

Hierarchical splitting. Great for markdown.

 Semantic Chunking

Similarity-based splits. Topic coherence.

Installation

Chunking requires the knowledge extra:

bash
pip install "praisonaiagents[knowledge]"

This installs the chonkie library automatically.