Praison AI

Chunking

Text chunking utilities for RAG and document processing

Split long documents into chunks for RAG pipelines and knowledge bases.

Diagram

Document

Chunking

Chunks

Quick Start

  1. Simple Usage
    bash
    npm install praisonai
    typescript
    import { Chunking } from 'praisonai';

const chunker = new Chunking({ chunkSize: 500, overlap: 50 });
const chunks = chunker.chunk('Your long text here...');
  2. With Configuration
    typescript
    const chunker = new Chunking({ chunkSize: 500, overlap: 50, strategy: 'sentence' });
const chunks = chunker.chunkBySentence('First sentence. Second sentence.');

Basic Usage

typescript
import { Chunking } from 'praisonai';

const chunker = new Chunking({ chunkSize: 500, overlap: 50 });
const chunks = chunker.chunk('Your long text here...');

chunks.forEach(chunk => {
  console.log(`Chunk ${chunk.index}: ${chunk.content.substring(0, 50)}...`);
});

Chunking Strategies

By Size

typescript
import { Chunking } from 'praisonai';

const chunker = new Chunking({
  chunkSize: 500,
  overlap: 50,
  strategy: 'size'
});

const text = 'A'.repeat(1500);
const chunks = chunker.chunkBySize(text);
console.log(`Created ${chunks.length} chunks`);

By Sentence

typescript
import { Chunking } from 'praisonai';

const chunker = new Chunking({ strategy: 'sentence' });

const text = 'First sentence. Second sentence. Third sentence.';
const chunks = chunker.chunkBySentence(text);

chunks.forEach(chunk => {
  console.log(chunk.content);
});

By Paragraph

typescript
import { Chunking } from 'praisonai';

const chunker = new Chunking({ strategy: 'paragraph' });

const text = `Paragraph one.

Paragraph two.

Paragraph three.`;

const chunks = chunker.chunkByParagraph(text);
console.log(`Found ${chunks.length} paragraphs`);

Semantic Chunking

typescript
import { Chunking } from 'praisonai';

const chunker = new Chunking({ strategy: 'semantic' });

const text = `# Header 1
Content under header 1.

# Header 2
Content under header 2.`;

const chunks = chunker.chunkBySemantic(text);

Chunk Structure

typescript
interface Chunk {
  content: string;      // Chunk text content
  index: number;        // Chunk index
  startOffset: number;  // Start position in original
  endOffset: number;    // End position in original
  metadata?: Record<string, any>;
}

Merge Small Chunks

typescript
import { Chunking } from 'praisonai';

const chunker = new Chunking({ strategy: 'sentence' });
const chunks = chunker.chunkBySentence(text);

// Merge chunks smaller than 100 characters
const merged = chunker.mergeSmallChunks(chunks, 100);

Factory Function

typescript
import { createChunking } from 'praisonai';

const chunker = createChunking({
  chunkSize: 1000,
  overlap: 100,
  strategy: 'size'
});

Integration with Knowledge Base

typescript
import { Knowledge, Chunking } from 'praisonai';

const kb = new Knowledge();
const chunker = new Chunking({ strategy: 'size', chunkSize: 512, overlap: 50 });

const document = 'Your long document text...';

for (const chunk of chunker.chunk(document)) {
  await kb.store(chunk.content, {
    userId: 'u1',
    metadata: { startOffset: chunk.startOffset },
  });
}

Related

Chunking CLI

CLI knowledge commands

 Knowledge Base

Store chunked documents

 RAG Agent

Retrieval workflows