> ## Documentation Index
> Fetch the complete documentation index at: https://praison.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Chunking Strategies

> Document chunking strategies for optimal RAG performance using the chonkie library

PraisonAI integrates [chonkie](https://github.com/chonkie-inc/chonkie) for high-performance document chunking.

```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from praisonaiagents import Agent

agent = Agent(
    name="researcher",
    instructions="Use chunked knowledge for long documents",
    knowledge=True,
)

agent.start("Find the section about deployment limits")
```

The user queries long sources; chunking controls what reaches retrieval.

```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
graph LR
    subgraph "Chunking"
        Document[📋 Document] --> Chunker[✂️ Chunking Engine]
        Chunker --> VectorDB[💾 Vector DB]
        VectorDB --> Retrieval[✅ Retrieval]
    end

    classDef input fill:#6366F1,stroke:#7C90A0,color:#fff
    classDef tool fill:#189AB4,stroke:#7C90A0,color:#fff
    classDef store fill:#8B0000,stroke:#7C90A0,color:#fff
    classDef success fill:#10B981,stroke:#7C90A0,color:#fff
    class Document input
    class Chunker tool
    class VectorDB store
    class Retrieval success
```

PraisonAI integrates [chonkie](https://github.com/chonkie-inc/chonkie) for high-performance document chunking.

## Quick Start

<Steps>
  <Step title="Default chunking on an agent">
    <CodeGroup>
      ```python Agent with Chunking (Simplest) theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
      from praisonaiagents import Agent

      # Default chunking (token-based)
      agent = Agent(
          instructions="Answer questions from documents.",
          knowledge=["research.pdf", "docs/"]
      )

      response = agent.start("What are the key findings?")
      ```

      ```python Agent with Custom Chunking theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
      from praisonaiagents import Agent

      agent = Agent(
          instructions="Answer questions from documents.",
          knowledge={
              "sources": ["research.pdf"],
              "chunker": {
                  "type": "semantic",       # token, sentence, recursive, semantic, sdpm, late
                  "chunk_size": 512,
                  "chunk_overlap": 128
              }
          }
      )

      response = agent.start("What methodology was used?")
      ```

      ```python Direct Chunking API theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
      from praisonaiagents import Chunking

      # Create chunker
      chunker = Chunking(
          chunker_type="recursive",
          chunk_size=512,
          chunk_overlap=128
      )

      # Chunk text
      chunks = chunker.chunk("Your document content here...")
      for chunk in chunks:
          print(f"Tokens: {chunk.token_count}")
          print(chunk.text[:100])
      ```
    </CodeGroup>
  </Step>
</Steps>

## Available Strategies

| Strategy | Best For | Speed |
| - | - | - |
| `token` | Fixed-size chunks | ⚡ Fastest |
| `sentence` | Natural boundaries | ⚡ Fast |
| `recursive` | Structured docs (markdown) | ⚡ Fast |
| `semantic` | Topic segmentation | 🔄 Medium |
| `sdpm` | Research papers | 🔄 Medium |
| `late` | Best embeddings | 🔄 Medium |

<CardGroup cols={2}>
  <Card title="Token Chunking" icon="hashtag" href="/docs/features/rag/strategies/token">
    Fixed-size token chunks. Fast and predictable.
  </Card>

  <Card title="Sentence Chunking" icon="paragraph" href="/docs/features/rag/strategies/sentence">
    Split at sentence boundaries. Natural flow.
  </Card>

  <Card title="Recursive Chunking" icon="sitemap" href="/docs/features/rag/strategies/recursive">
    Hierarchical splitting. Great for markdown.
  </Card>

  <Card title="Semantic Chunking" icon="brain" href="/docs/features/rag/strategies/semantic">
    Similarity-based splits. Topic coherence.
  </Card>
</CardGroup>

## Chunker Configuration

<Note>
  Chunker settings (`chunk_size`, `chunk_overlap`, `type`) apply uniformly to every supported file type — PDFs, DOCX, media via MarkItDown, source files, and plain-text files (`.md`, `.txt`, `.csv`, `.json`, `.xml`, `.html`, `.htm`). On pre-[#4831](https://github.com/MervinPraison/PraisonAI/pull/4831) releases these settings were silently ignored for the plain-text extensions.
</Note>

### All Parameters

| Parameter | Type | Default | Applies To |
| - | - | - | - |
| `type` | str | `"token"` | All |
| `chunk_size` | int | 512 | All |
| `chunk_overlap` | int | 128 | token, sentence |
| `tokenizer_or_token_counter` | str | `"gpt2"` | token, sentence, recursive |
| `embedding_model` | str | auto | semantic, sdpm, late |

### Strategy Examples

<Tabs>
  <Tab title="Token">
    ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
    agent = Agent(
        instructions="Process documents.",
        knowledge={
            "sources": ["docs/"],
            "chunker": {
                "type": "token",
                "chunk_size": 256,
                "chunk_overlap": 50
            }
        }
    )
    ```
  </Tab>

  <Tab title="Sentence">
    ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
    agent = Agent(
        instructions="Process articles.",
        knowledge={
            "sources": ["articles/"],
            "chunker": {
                "type": "sentence",
                "chunk_size": 512
            }
        }
    )
    ```
  </Tab>

  <Tab title="Recursive">
    ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
    agent = Agent(
        instructions="Process markdown docs.",
        knowledge={
            "sources": ["README.md", "docs/"],
            "chunker": {
                "type": "recursive",
                "chunk_size": 512
            }
        }
    )
    ```
  </Tab>

  <Tab title="Semantic">
    ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
    agent = Agent(
        instructions="Process research papers.",
        knowledge={
            "sources": ["papers/"],
            "chunker": {
                "type": "semantic",
                "chunk_size": 512,
                "embedding_model": "all-MiniLM-L6-v2"
            }
        }
    )
    ```

    <Note>
      `semantic`, `sdpm`, and `late` chunkers lazily import `chonkie.embeddings.AutoEmbeddings` on the first lookup. Since PraisonAI PR [#5105](https://github.com/MervinPraison/PraisonAI/pull/5105) a missing `chonkie` install raises `ImportError: chonkie package not found. Please install it using: pip install 'praisonaiagents[knowledge]'` — a broken transitive dependency (e.g. `sentence-transformers`) keeps its original error unchanged.
    </Note>
  </Tab>
</Tabs>

## Which Strategy Should I Use?

```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
graph TB
    Start([What's your priority?]) --> B{Speed or quality?}
    B -->|Speed| C[token<br/>fast, fixed-size]
    B -->|Quality| D{Document type?}
    D -->|Structured/Markdown| E[recursive]
    D -->|Natural prose| F{Topic boundaries?}
    F -->|Yes| G[semantic]
    F -->|No| H[sentence]

    classDef start fill:#6366F1,stroke:#7C90A0,color:#fff
    classDef question fill:#F59E0B,stroke:#7C90A0,color:#fff
    classDef answer fill:#10B981,stroke:#7C90A0,color:#fff

    class Start start
    class B,D,F question
    class C,E,G,H answer
```

***

## How It Works

The user adds documents; the chunking engine splits them by the chosen strategy, embeds each chunk, and stores it for retrieval.

```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
sequenceDiagram
    participant User
    participant Agent
    participant Chunker
    participant VectorDB

    User->>Agent: knowledge=["docs/"]
    Agent->>Chunker: Split documents
    Chunker->>VectorDB: Store embedded chunks
    User->>Agent: start("Find deployment limits")
    Agent->>VectorDB: Retrieve relevant chunks
    VectorDB-->>User: Answer from chunks
```

## Installation

```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
pip install "praisonaiagents[knowledge]"
```

This installs the chonkie library automatically.

## Best Practices

<AccordionGroup>
  <Accordion title="Match chunk size to retrieval use case">
    Smaller chunks improve precision for Q\&A; larger chunks preserve narrative for summarisation tasks.
  </Accordion>

  <Accordion title="Install the knowledge extra">
    Run `pip install "praisonaiagents[knowledge]"` so chonkie and indexing backends load only when needed.
  </Accordion>

  <Accordion title="Test overlap on sample docs">
    Tune overlap on a representative document before indexing an entire corpus.
  </Accordion>

  <Accordion title="Pair with smart retrieval">
    Combine chunking with hybrid search and reranking for better recall on long knowledge bases.
  </Accordion>
</AccordionGroup>

## Troubleshooting

| Symptom | Cause | Fix |
| - | - | - |
| `ImportError: chonkie package not found. Please install it using: pip install 'praisonaiagents[knowledge]'` on the first `Knowledge.add(...)` / `agent.start(...)` call | `semantic` (the default), `sdpm`, or `late` chunker is selected but the `[knowledge]` extra is not installed. Since PraisonAI PR [#5105](https://github.com/MervinPraison/PraisonAI/pull/5105) both the chunker registry and the embedding backend surface the same actionable hint. | `pip install 'praisonaiagents[knowledge]'`. Or fall back to `chunking_strategy="sentence"` / `"paragraph"` / `"fixed"` / `"token"`, which do not need chonkie's embedding model. |
| `ImportError` that names `chonkie.embeddings` but **omits** the `pip install` hint | `chonkie` is installed but its embeddings submodule cannot import (broken `sentence-transformers`, `torch`, or `transformers`). PR [#5105](https://github.com/MervinPraison/PraisonAI/pull/5105) intentionally preserves the original error so real transitive failures are not masked. | Read the underlying error, then reinstall or upgrade that package (`pip install --upgrade sentence-transformers` is the most common fix). |

## Related

<CardGroup cols={2}>
  <Card title="Knowledge Base" icon="book" href="/docs/features/knowledge">
    Configure knowledge sources and retrieval
  </Card>

  <Card title="RAG Agents" icon="robot" href="/docs/features/rag">
    Build retrieval-augmented agents
  </Card>

  <Card title="Bot Platform Capabilities" icon="sliders" href="/docs/features/bot-platform-capabilities">
    How platform capabilities drive this feature
  </Card>
</CardGroup>


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.