> ## Documentation Index
> Fetch the complete documentation index at: https://praison.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Chunking Strategies

> Document chunking strategies for optimal RAG performance using the chonkie library

PraisonAI integrates [chonkie](https://github.com/chonkie-inc/chonkie) for high-performance document chunking.

```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from praisonaiagents import Agent

agent = Agent(
    name="researcher",
    instructions="Use chunked knowledge for long documents",
    knowledge=True,
)

agent.start("Find the section about deployment limits")
```

The user queries long sources; chunking controls what reaches retrieval.

```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
graph LR
    subgraph "Chunking"
        Document[📋 Document] --> Chunker[✂️ Chunking Engine]
        Chunker --> VectorDB[💾 Vector DB]
        VectorDB --> Retrieval[✅ Retrieval]
    end

    classDef input fill:#6366F1,stroke:#7C90A0,color:#fff
    classDef tool fill:#189AB4,stroke:#7C90A0,color:#fff
    classDef store fill:#8B0000,stroke:#7C90A0,color:#fff
    classDef success fill:#10B981,stroke:#7C90A0,color:#fff
    class Document input
    class Chunker tool
    class VectorDB store
    class Retrieval success
```

PraisonAI integrates [chonkie](https://github.com/chonkie-inc/chonkie) for high-performance document chunking.

## Quick Start

<Steps>
  <Step title="Default chunking on an agent">
    <CodeGroup>
      ```python Agent with Chunking (Simplest) theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
      from praisonaiagents import Agent

      # Default chunking (token-based)
      agent = Agent(
          instructions="Answer questions from documents.",
          knowledge=["research.pdf", "docs/"]
      )

      response = agent.start("What are the key findings?")
      ```

      ```python Agent with Custom Chunking theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
      from praisonaiagents import Agent

      agent = Agent(
          instructions="Answer questions from documents.",
          knowledge={
              "sources": ["research.pdf"],
              "chunker": {
                  "type": "semantic",       # token, sentence, recursive, semantic, sdpm, late
                  "chunk_size": 512,
                  "chunk_overlap": 128
              }
          }
      )

      response = agent.start("What methodology was used?")
      ```

      ```python Direct Chunking API theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
      from praisonaiagents import Chunking

      # Create chunker
      chunker = Chunking(
          chunker_type="recursive",
          chunk_size=512,
          chunk_overlap=128
      )

      # Chunk text
      chunks = chunker.chunk("Your document content here...")
      for chunk in chunks:
          print(f"Tokens: {chunk.token_count}")
          print(chunk.text[:100])
      ```
    </CodeGroup>
  </Step>
</Steps>

## Available Strategies

| Strategy    | Best For                   | Speed     |
| ----------- | -------------------------- | --------- |
| `token`     | Fixed-size chunks          | ⚡ Fastest |
| `sentence`  | Natural boundaries         | ⚡ Fast    |
| `recursive` | Structured docs (markdown) | ⚡ Fast    |
| `semantic`  | Topic segmentation         | 🔄 Medium |
| `sdpm`      | Research papers            | 🔄 Medium |
| `late`      | Best embeddings            | 🔄 Medium |

<CardGroup cols={2}>
  <Card title="Token Chunking" icon="hashtag" href="/docs/features/rag/strategies/token">
    Fixed-size token chunks. Fast and predictable.
  </Card>

  <Card title="Sentence Chunking" icon="paragraph" href="/docs/features/rag/strategies/sentence">
    Split at sentence boundaries. Natural flow.
  </Card>

  <Card title="Recursive Chunking" icon="sitemap" href="/docs/features/rag/strategies/recursive">
    Hierarchical splitting. Great for markdown.
  </Card>

  <Card title="Semantic Chunking" icon="brain" href="/docs/features/rag/strategies/semantic">
    Similarity-based splits. Topic coherence.
  </Card>
</CardGroup>

## Chunker Configuration

### All Parameters

| Parameter                    | Type | Default   | Applies To                 |
| ---------------------------- | ---- | --------- | -------------------------- |
| `type`                       | str  | `"token"` | All                        |
| `chunk_size`                 | int  | 512       | All                        |
| `chunk_overlap`              | int  | 128       | token, sentence            |
| `tokenizer_or_token_counter` | str  | `"gpt2"`  | token, sentence, recursive |
| `embedding_model`            | str  | auto      | semantic, sdpm, late       |

### Strategy Examples

<Tabs>
  <Tab title="Token">
    ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
    agent = Agent(
        instructions="Process documents.",
        knowledge={
            "sources": ["docs/"],
            "chunker": {
                "type": "token",
                "chunk_size": 256,
                "chunk_overlap": 50
            }
        }
    )
    ```
  </Tab>

  <Tab title="Sentence">
    ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
    agent = Agent(
        instructions="Process articles.",
        knowledge={
            "sources": ["articles/"],
            "chunker": {
                "type": "sentence",
                "chunk_size": 512
            }
        }
    )
    ```
  </Tab>

  <Tab title="Recursive">
    ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
    agent = Agent(
        instructions="Process markdown docs.",
        knowledge={
            "sources": ["README.md", "docs/"],
            "chunker": {
                "type": "recursive",
                "chunk_size": 512
            }
        }
    )
    ```
  </Tab>

  <Tab title="Semantic">
    ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
    agent = Agent(
        instructions="Process research papers.",
        knowledge={
            "sources": ["papers/"],
            "chunker": {
                "type": "semantic",
                "chunk_size": 512,
                "embedding_model": "all-MiniLM-L6-v2"
            }
        }
    )
    ```
  </Tab>
</Tabs>

## Which Strategy Should I Use?

```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
graph TB
    Start([What's your priority?]) --> B{Speed or quality?}
    B -->|Speed| C[token<br/>fast, fixed-size]
    B -->|Quality| D{Document type?}
    D -->|Structured/Markdown| E[recursive]
    D -->|Natural prose| F{Topic boundaries?}
    F -->|Yes| G[semantic]
    F -->|No| H[sentence]

    classDef start fill:#6366F1,stroke:#7C90A0,color:#fff
    classDef question fill:#F59E0B,stroke:#7C90A0,color:#fff
    classDef answer fill:#10B981,stroke:#7C90A0,color:#fff

    class Start start
    class B,D,F question
    class C,E,G,H answer
```

***

## How It Works

The user adds documents; the chunking engine splits them by the chosen strategy, embeds each chunk, and stores it for retrieval.

```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
sequenceDiagram
    participant User
    participant Agent
    participant Chunker
    participant VectorDB

    User->>Agent: knowledge=["docs/"]
    Agent->>Chunker: Split documents
    Chunker->>VectorDB: Store embedded chunks
    User->>Agent: start("Find deployment limits")
    Agent->>VectorDB: Retrieve relevant chunks
    VectorDB-->>User: Answer from chunks
```

## Installation

```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
pip install "praisonaiagents[knowledge]"
```

This installs the chonkie library automatically.

## Best Practices

<AccordionGroup>
  <Accordion title="Match chunk size to retrieval use case">
    Smaller chunks improve precision for Q\&A; larger chunks preserve narrative for summarisation tasks.
  </Accordion>

  <Accordion title="Install the knowledge extra">
    Run `pip install "praisonaiagents[knowledge]"` so chonkie and indexing backends load only when needed.
  </Accordion>

  <Accordion title="Test overlap on sample docs">
    Tune overlap on a representative document before indexing an entire corpus.
  </Accordion>

  <Accordion title="Pair with smart retrieval">
    Combine chunking with hybrid search and reranking for better recall on long knowledge bases.
  </Accordion>
</AccordionGroup>

## Related

<CardGroup cols={2}>
  <Card title="Knowledge Base" icon="book" href="/docs/features/knowledge">
    Configure knowledge sources and retrieval
  </Card>

  <Card title="RAG Agents" icon="robot" href="/docs/features/rag">
    Build retrieval-augmented agents
  </Card>

  <Card title="Bot Platform Capabilities" icon="sliders" href="/docs/features/bot-platform-capabilities">
    How platform capabilities drive this feature
  </Card>
</CardGroup>