Incremental Indexing

Skip unchanged files automatically — only modified content is re-indexed on each run, saving time on large document corpora.

from praisonaiagents import Agent

agent = Agent(
    name="DocExpert",
    instructions="Answer questions using the knowledge base.",
    knowledge=["./docs"],
)

agent.start("What changed in the docs since last week?")

The user queries the knowledge base; only changed files are re-indexed on each run.

Quick Start

Agent with incremental knowledge

from praisonaiagents import Agent

agent = Agent(
    name="DocExpert",
    instructions="Answer questions using the knowledge base.",
    knowledge=["./docs"],
    memory={"user_id": "my_user"},
)

response = agent.start("What are the main topics covered?")

On the second run, unchanged files are skipped automatically.

Direct index with result stats

from praisonaiagents import Agent, Knowledge

knowledge = Knowledge()

result = knowledge.index(
    "./docs",
    memory={"user_id": "my_user"},
    incremental=True,
)

print(f"Indexed: {result.files_indexed}, Skipped: {result.files_skipped}")
print(f"Duration: {result.duration_seconds:.2f}s")

How It Works

Component	Purpose
`FileTracker`	Tracks file hash and mtime across sessions
`IndexResult`	Reports indexed, skipped, and error counts
`CorpusStats`	Corpus-level statistics and strategy recommendation
`.praisonignore`	Gitignore-style patterns to exclude files

Configuration Options

`Knowledge.index()` parameters

Parameter	Type	Default	Description
`path`	`str`	—	Directory or file path to index
`incremental`	`bool`	`True`	Skip unchanged files
`force`	`bool`	`False`	Re-index all files regardless of changes
`include_glob`	`list[str]`	`None`	Glob patterns to include, e.g. `[".md", ".py"]`
`exclude_glob`	`list[str]`	`None`	Glob patterns to exclude, e.g. `[".log", "test_"]`
`user_id`	`str`	`None`	Scope index to a user
`agent_id`	`str`	`None`	Scope index to an agent
`run_id`	`str`	`None`	Scope index to a run

`IndexResult` fields

Field	Type	Default	Description
`success`	`bool`	`True`	Whether indexing succeeded
`files_indexed`	`int`	`0`	Files newly indexed
`files_skipped`	`int`	`0`	Files skipped (unchanged)
`chunks_created`	`int`	`0`	Chunks created from indexed files
`errors`	`list[str]`	`[]`	Error messages for failed files
`duration_seconds`	`float`	`0.0`	Total indexing time
`corpus_stats`	`CorpusStats \| None`	`None`	Corpus statistics

`CorpusStats` fields

Field	Type	Default	Description
`file_count`	`int`	`0`	Number of files in corpus
`chunk_count`	`int`	`0`	Number of chunks created
`total_tokens`	`int`	`0`	Estimated total tokens
`indexed_at`	`str \| None`	`None`	ISO timestamp of last index
`path`	`str \| None`	`None`	Path to corpus root
`strategy_recommendation`	`str`	computed	Retrieval strategy based on corpus size

`FileTracker` API

FileTracker records which vector-store chunks belong to each file so a re-index can remove stale content first.

Method	Signature	Description
`mark_indexed`	`mark_indexed(filepath, info, memory_ids=None)`	Record a file as indexed. `memory_ids` are the vector-store IDs produced for that file, so the next re-index knows which chunks to delete.
`get_memory_ids`	`get_memory_ids(filepath) -> list[str]`	Return the memory IDs previously stored for a file (empty list if none).

Why this matters: editing a file and re-indexing no longer leaves the old chunks behind, so search results reflect only the current content. IDs that fail to delete are retried on the next re-index — they are never orphaned.

What gets deleted — inspect the chunks that will be removed when a file changes:

from praisonaiagents import FileTracker

tracker = FileTracker(state_file=".praison/.index_state.json")
tracker.load()

print(tracker.get_memory_ids("./docs/readme.md"))
# ['mem_abc123', 'mem_def456', ...]  — chunks deleted on the next re-index if the file changes

Common Patterns

Check and report results

from praisonaiagents import Knowledge, IndexResult

knowledge = Knowledge()
result = knowledge.index("./docs", memory={"user_id": "alice"})

if result.errors:
    for err in result.errors:
        print(f"Error: {err}")

print(f"Total: {result.files_indexed + result.files_skipped} files")
print(f"New/updated: {result.files_indexed}")
print(f"Unchanged: {result.files_skipped}")

Force full re-index

from praisonaiagents import Knowledge

knowledge = Knowledge()

result = knowledge.index(
    "./docs",
    memory={"user_id": "alice"},
    force=True,
)
print(f"Full re-index complete: {result.files_indexed} files")

Selective indexing with glob patterns

from praisonaiagents import Knowledge

knowledge = Knowledge()

result = knowledge.index(
    "./project",
    memory={"user_id": "alice"},
    include_glob=["*.md", "*.txt", "*.py"],
    exclude_glob=["*.log", "test_*", "__pycache__/*"],
)

Corpus statistics

from praisonaiagents import Knowledge, CorpusStats

stats = CorpusStats.from_directory("./docs")
print(f"Files: {stats.file_count}")
print(f"Estimated tokens: {stats.total_tokens}")
print(f"Recommended strategy: {stats.strategy_recommendation}")

Low-level file tracking

from praisonaiagents import FileTracker

tracker = FileTracker(state_file=".praison/.index_state.json")
tracker.load()

if tracker.has_changed("./docs/readme.md"):
    print("File needs re-indexing")

Best Practices

Use .praisonignore to exclude noisy files

Create a .praisonignore in your corpus directory — it follows gitignore syntax and is auto-detected:

*.log
*.tmp
test/
tests/
__pycache__/
.env
secrets.txt

.gitignore is also read as a fallback.

Scope indexes per user for multi-tenant apps

Pass user_id or agent_id to isolate indexes per tenant. Without a scope identifier, the mem0 backend raises ScopeRequiredError.

result = knowledge.index("./shared-docs", memory={"user_id": user_id})

Inspect strategy_recommendation for large corpora

CorpusStats.strategy_recommendation returns the optimal retrieval strategy based on file count:

Files	Strategy
< 10	`direct`
< 100	`basic`
< 1000	`hybrid`
< 10000	`reranked`
< 100000	`compressed`
100000+	`hierarchical`

Schedule periodic full re-indexes

Weekly force=True re-indexes are still a good hygiene practice for hash-collision paranoia and to recover any IDs that repeatedly fail to delete, but they are no longer required to keep the vector store from growing unbounded on edited files — a changed file’s stale chunks are now deleted before its new chunks are added.

Knowledge Backends

Choose and configure the knowledge storage backend

Knowledge

Core knowledge retrieval and agent integration

Getting Started

Install

Core Concepts

Guides

Features

Models

Databases

Observability

Memory

Telemetry & Analytics

Knowledge

RAG

Persistence

Tools

Other Features

Developers

Configuration

Best Practices

Getting Started (No Code)

Incremental Indexing

Quick Start

How It Works

Configuration Options

`Knowledge.index()` parameters

`IndexResult` fields

`CorpusStats` fields

`FileTracker` API

Common Patterns

Check and report results

Force full re-index

Selective indexing with glob patterns

Corpus statistics

Low-level file tracking

Best Practices

Knowledge Backends

Knowledge

​Quick Start

​How It Works

​Configuration Options

​Knowledge.index() parameters

​IndexResult fields

​CorpusStats fields

​FileTracker API

​Common Patterns

​Check and report results

​Force full re-index

​Selective indexing with glob patterns

​Corpus statistics

​Low-level file tracking

​Best Practices

​Related

Knowledge Backends

Knowledge

Quick Start

How It Works

Configuration Options

`Knowledge.index()` parameters

`IndexResult` fields

`CorpusStats` fields

`FileTracker` API

Common Patterns

Check and report results

Force full re-index

Selective indexing with glob patterns

Corpus statistics

Low-level file tracking

Best Practices

Related