Knowledge Base

Advanced knowledge management system for AI agents

Knowledge Base System

The knowledge system provides sophisticated document processing and semantic search capabilities, enabling agents to access and utilise information from various sources.

Diagram

Key Features

Card

Process PDFs, documents, spreadsheets, images, and raw text

Card

Multiple strategies for optimal text segmentation

Card

Vector-based search with optional reranking

Card

User, agent, and run-specific knowledge scoping

Card

Optional relationship extraction and storage

Card

Automatic quality assessment for stored knowledge

Quick Start

Agent with Knowledge

from praisonaiagents import Agent

agent = Agent(
    name="Research Assistant",
    instructions="Answer questions using the knowledge base.",
    knowledge={
        "sources": ["research_paper.pdf", "data.txt"],
        "vector_store": {
            "provider": "chroma",
            "config": {
                "collection_name": "research_docs",
                "path": ".praison"
            }
        }
    }
)

response = agent.start("What are the key findings?")

Direct Knowledge Use

from praisonaiagents import Knowledge

# Initialise knowledge base
kb = Knowledge({
    "vector_store": {
        "provider": "chroma",
        "config": {"collection_name": "my_knowledge"}
    }
})

# Add documents (local files and directories)
kb.add("document.pdf", user_id="user123")

# Store text directly
kb.store("Important fact about AI", user_id="user123")

# Search
results = kb.search("AI applications", user_id="user123")

Advanced Search

# Search with reranking
results = kb.search(
    query="machine learning applications",
    user_id="user123",
    rerank=True,
    top_k=10
)

# Filter by metadata
results = kb.search(
    query="technical specs",
    user_id="user123",
    filters={"category": "engineering"}
)

Configuration Options

Basic Configuration

python

knowledge_config = {
    "vector_store": {
        "provider": "chroma",
        "config": {
            "collection_name": "knowledge_base",
            "path": ".praison",
            "distance_metric": "cosine"
        }
    },
    "embedder": {
        "provider": "openai",
        "config": {
            "model": "text-embedding-3-small"
        }
    }
}

Advanced Configuration with Graph Store

python

knowledge_config = {
    "vector_store": {
        "provider": "chroma",
        "config": {
            "collection_name": "knowledge_base",
            "path": ".praison"
        }
    },
    "graph_store": {
        "provider": "neo4j",
        "config": {
            "url": "bolt://localhost:7687",
            "username": "neo4j",
            "password": "password"
        }
    },
    "llm": {
        "provider": "openai",
        "config": {
            "model": "gpt-4o-mini",
            "temperature": 0
        }
    },
    "reranker": {
        "enabled": True,
        "default_rerank": False
    }
}

Chunking Strategies

The chosen strategy applies to every file type, including plain-text files.

Token-based Chunking

kb = Knowledge({
    "chunker": {
        "name": "token",
        "chunk_size": 500,
        "chunk_overlap": 50
    }
})

Best for: Consistent chunk sizes, token-limited models

Sentence-based Chunking

kb = Knowledge({
    "chunker": {
        "name": "sentence",
        "chunk_size": 10,  # sentences per chunk
        "min_chunk_size": 3
    }
})

Best for: Natural text boundaries, Q&A systems

Semantic Chunking

kb = Knowledge({
    "chunker": {
        "name": "semantic",
        "threshold": 0.7,
        "min_chunk_size": 100
    }
})

Best for: Topic-based segmentation, research papers

SDPM Chunking

kb = Knowledge({
    "chunker": {
        "name": "sdpm",
        "max_chunk_size": 1000
    }
})

Best for: Document structure preservation

Document Processing

Supported File Types

Card

PDF (.pdf)
Word (.doc, .docx)
Text (.txt)
Markdown (.md)
RTF (.rtf)

Card

Excel (.xls, .xlsx)
CSV (.csv)
JSON (.json)
XML (.xml)

Card

Images (OCR)
Local HTML files (.html, .htm)
Raw text strings
YouTube videos
Remote URLs (http:// / https://) — via MarkItDown, Knowledge.add(url)

Processing Options

python

# Add with metadata
kb.add(
    "research.pdf",
    user_id="user123",
    metadata={
        "category": "AI Research",
        "year": 2024,
        "author": "Dr. Smith"
    }
)

# Batch processing
documents = ["doc1.pdf", "doc2.txt", "doc3.md"]
for doc in documents:
    kb.add(doc, user_id="user123")

python

from praisonaiagents import Knowledge

kb = Knowledge(config={
    "vector_store": {"provider": "chroma", "config": {"collection_name": "web_docs"}}
})

kb.add("https://example.com/article", user_id="user123")
kb.add("https://arxiv.org/pdf/2301.00000.pdf", user_id="user123")   # PDFs via URL work too

# Metadata is merged with the auto-added source_type/source keys:
kb.add(
    "https://blog.example.com/post",
    user_id="user123",
    metadata={"category": "blog", "year": 2026},
)

results = kb.search("what does the article say?", user_id="user123")

Fetching URL knowledge safely

URL knowledge sources go through the same SSRF guard the OpenAPI toolset uses. A loopback (127.0.0.1, ::1), private (RFC1918), link-local (169.254.0.0/16), multicast or unspecified URL raises ValueError("Refusing to fetch unsafe URL: ...") before MarkItDown is called.

The guard checks the URL you pass to Knowledge.add(). It does not control how MarkItDown handles redirects.

Search Features

Basic Search

python

# Simple search
results = kb.search("artificial intelligence", limit=5)

# User-scoped search
results = kb.search(
    query="machine learning",
    user_id="user123",
    limit=10
)

Advanced Search Options

With Reranking

# Enable Mem0 reranking for better relevance
results = kb.search(
    query="neural networks",
    user_id="user123",
    rerank=True,
    top_k=20  # Retrieve more before reranking
)

Hybrid Search

# Combine keyword and semantic search
results = kb.search(
    query="transformer architecture",
    user_id="user123",
    keyword_search=True,  # Better recall
    filter_memories=True  # Better precision
)

Metadata Filtering

# Filter by metadata
results = kb.search(
    query="implementation details",
    user_id="user123",
    filters={
        "category": "technical",
        "year": {"$gte": 2023}
    }
)

Memory Integration

When used with agents, knowledge automatically integrates with memory:

python

agent = Agent(
    name="Research Assistant",
    knowledge={
        "sources": ["papers/"],
        "vector_store": {
            "provider": "chroma",
            "config": {"collection_name": "research_docs"}
        }
    },
    memory=True  # Enable memory integration
)

# Knowledge is automatically searched during conversations
response = agent.chat("What does the research say about transformers?")

Graph Store Features

Graph stores enable relationship extraction and complex queries beyond simple semantic search.

Configuration

python

knowledge_config = {
    "graph_store": {
        "provider": "neo4j",  # or "memgraph"
        "config": {
            "url": "bolt://localhost:7687",
            "username": "neo4j",
            "password": "password"
        }
    },
    "extract_relationships": True
}

Relationship Queries

python

# Find related concepts
results = kb.search_graph(
    "What concepts are related to transformers?",
    user_id="user123"
)

# Explore connections
results = kb.search_graph(
    "How is attention mechanism connected to BERT?",
    user_id="user123"
)

Best Practices

Chunking Strategy

Smaller chunks (100-200 tokens): Better precision
Larger chunks (500-1000 tokens): Better context
Match chunk size to query complexity

Organisation

Separate collections by domain
Use metadata for filtering
Regular cleanup of outdated content

Performance

Enable caching for repeated queries
Use appropriate embedding models
Batch document processing

Quality

Verify document processing
Monitor search relevance
Regular reindexing if needed

Example: Research Assistant

Complete Example

from praisonaiagents import Agent
from praisonaiagents import Knowledge

# Configure knowledge base
knowledge_config = {
    "vector_store": {
        "provider": "chroma",
        "config": {
            "collection_name": "research_papers",
            "path": ".praison"
        }
    },
    "chunker": {
        "name": "semantic",
        "threshold": 0.7
    },
    "embedder": {
        "provider": "openai",
        "config": {
            "model": "text-embedding-3-small"
        }
    },
    "reranker": {
        "enabled": True
    }
}

# Create research assistant
research_agent = Agent(
    name="Research Assistant",
    instructions="""You are an expert research assistant.
    Use the knowledge base to provide accurate, well-sourced answers.
    Always cite the specific documents you reference.""",
    knowledge={
        "sources": ["research_papers"],
        **knowledge_config
    }
)

# Use the assistant
response = research_agent.chat(
    "What are the main approaches to AI alignment?"
)

# Direct knowledge queries
kb = research_agent.knowledge_instance
papers = kb.search("alignment techniques", limit=5)
for paper in papers:
    print(f"- {paper['text'][:100]}...")
    print(f"  Source: {paper.get('metadata', {}).get('source')}")

Troubleshooting

Symptom	Likely cause	Fix
`RuntimeError: Chroma persist failed at '<path>' (PanicException: ...)` on `Agent.start(...)` or `Knowledge.add(...)`	The on-disk sqlite Chroma store at `<path>` is corrupt, locked by another process, or was written by an incompatible Chroma version.	Delete `<path>` and re-index, or point `knowledge={"vector_store": {"config": {"path": "/a/fresh/dir"}}}` at a fresh directory.
`RuntimeError: Knowledge indexing backend crashed on <file>` while indexing a directory	The vector-store backend panicked on that file (typically a Chroma rust-binding panic on a corrupt store).	Delete the persist directory or move `<file>` aside and re-run.
Two unrelated projects appear to share the same Knowledge base	You are on a pre-#4422 release where the default `path` was the cwd-relative string `"knowledge_db"`.	Upgrade to the release built from PraisonAI PR #4422; defaults are now absolute (`<project>/.praisonai/knowledge/chroma`).
Windows: `Agent(knowledge=["docs/"])` SIGSEGVs or hangs	Pre-#4422 issue #4376: pyo3 `PanicException` (a `BaseException`) unwound the process instead of surfacing as a Python error.	Upgrade; panics are now caught and re-raised as `RuntimeError` with an actionable message.

Next Steps

Card

Learn about memory integration

Card

Build RAG applications