Chunking Strategies

Document chunking strategies for optimal RAG performance using the chonkie library

PraisonAI integrates chonkie for high-performance document chunking.

from praisonaiagents import Agent

agent = Agent(
    name="researcher",
    instructions="Use chunked knowledge for long documents",
    knowledge=True,
)

agent.start("Find the section about deployment limits")

The user queries long sources; chunking controls what reaches retrieval.

PraisonAI integrates chonkie for high-performance document chunking.

Quick Start

1. Default chunking on an agent

   from praisonaiagents import Agent
   
   # Default chunking (token-based)
   agent = Agent(
       instructions="Answer questions from documents.",
       knowledge=["research.pdf", "docs/"]
   )
   
   response = agent.start("What are the key findings?")

   from praisonaiagents import Agent
   
   agent = Agent(
       instructions="Answer questions from documents.",
       knowledge={
           "sources": ["research.pdf"],
           "chunker": {
               "type": "semantic",       # token, sentence, recursive, semantic, sdpm, late
               "chunk_size": 512,
               "chunk_overlap": 128
           }
       }
   )
   
   response = agent.start("What methodology was used?")

   from praisonaiagents import Chunking
   
   # Create chunker
   chunker = Chunking(
       chunker_type="recursive",
       chunk_size=512,
       chunk_overlap=128
   )
   
   # Chunk text
   chunks = chunker.chunk("Your document content here...")
   for chunk in chunks:
       print(f"Tokens: {chunk.token_count}")
       print(chunk.text[:100])

Available Strategies

Chunker Configuration

All Parameters

Strategy Examples

agent = Agent(
    instructions="Process documents.",
    knowledge={
        "sources": ["docs/"],
        "chunker": {
            "type": "token",
            "chunk_size": 256,
            "chunk_overlap": 50
        }
    }
)

agent = Agent(
    instructions="Process articles.",
    knowledge={
        "sources": ["articles/"],
        "chunker": {
            "type": "sentence",
            "chunk_size": 512
        }
    }
)

agent = Agent(
    instructions="Process markdown docs.",
    knowledge={
        "sources": ["README.md", "docs/"],
        "chunker": {
            "type": "recursive",
            "chunk_size": 512
        }
    }
)

agent = Agent(
    instructions="Process research papers.",
    knowledge={
        "sources": ["papers/"],
        "chunker": {
            "type": "semantic",
            "chunk_size": 512,
            "embedding_model": "all-MiniLM-L6-v2"
        }
    }
)

Which Strategy Should I Use?

How It Works

The user adds documents; the chunking engine splits them by the chosen strategy, embeds each chunk, and stores it for retrieval.

Installation

pip install "praisonaiagents[knowledge]"

This installs the chonkie library automatically.

Best Practices

Troubleshooting

Related