> ## Documentation Index
> Fetch the complete documentation index at: https://praison.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Incremental Indexing

> Index only changed files — skip unchanged content to keep knowledge bases fast and up to date

Skip unchanged files automatically — only modified content is re-indexed on each run, saving time on large document corpora.

```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from praisonaiagents import Agent

agent = Agent(
    name="DocExpert",
    instructions="Answer questions using the knowledge base.",
    knowledge=["./docs"],
)

agent.start("What changed in the docs since last week?")
```

The user queries the knowledge base; only changed files are re-indexed on each run.

```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
graph LR
    A[📁 Files] --> B[🔍 FileTracker]
    B --> C{Changed?}
    C -->|Yes| D[📥 Index]
    C -->|No| E[⏭ Skip]
    D --> F[📊 IndexResult]
    E --> F

    classDef input fill:#6366F1,stroke:#7C90A0,color:#fff
    classDef process fill:#F59E0B,stroke:#7C90A0,color:#fff
    classDef result fill:#10B981,stroke:#7C90A0,color:#fff

    class A input
    class B,C,D,E process
    class F result

```

## Quick Start

<Steps>
  <Step title="Agent with incremental knowledge">
    ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
    from praisonaiagents import Agent

    agent = Agent(
        name="DocExpert",
        instructions="Answer questions using the knowledge base.",
        knowledge=["./docs"],
        memory={"user_id": "my_user"},
    )

    response = agent.start("What are the main topics covered?")
    ```

    On the second run, unchanged files are skipped automatically.
  </Step>

  <Step title="Direct index with result stats">
    ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
    from praisonaiagents import Agent, Knowledge

    knowledge = Knowledge()

    result = knowledge.index(
        "./docs",
        memory={"user_id": "my_user"},
        incremental=True,
    )

    print(f"Indexed: {result.files_indexed}, Skipped: {result.files_skipped}")
    print(f"Duration: {result.duration_seconds:.2f}s")
    ```
  </Step>
</Steps>

***

## How It Works

```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
sequenceDiagram
    participant User
    participant Agent
    participant FileTracker
    participant Knowledge

    User->>Agent: start("What changed?")
    Agent->>Knowledge: index("./docs", incremental=True)
    Knowledge->>FileTracker: load state from .praison/.index_state.json
    loop For each file
        Knowledge->>FileTracker: has_changed(filepath)?
        alt File changed or new
            FileTracker-->>Knowledge: True
            Knowledge->>Knowledge: delete stale chunks (from previous memory_ids)
            Knowledge->>Knowledge: index file
            Knowledge->>FileTracker: mark_indexed(filepath, memory_ids=[...])
        else File unchanged
            FileTracker-->>Knowledge: False
            Knowledge->>Knowledge: skip
        end
    end
    Knowledge->>FileTracker: save state
    Knowledge-->>Agent: IndexResult
    Agent-->>User: answer
```

| Component        | Purpose                                             |
| ---------------- | --------------------------------------------------- |
| `FileTracker`    | Tracks file hash and mtime across sessions          |
| `IndexResult`    | Reports indexed, skipped, and error counts          |
| `CorpusStats`    | Corpus-level statistics and strategy recommendation |
| `.praisonignore` | Gitignore-style patterns to exclude files           |

***

## Configuration Options

### `Knowledge.index()` parameters

| Parameter      | Type        | Default | Description                                          |
| -------------- | ----------- | ------- | ---------------------------------------------------- |
| `path`         | `str`       | —       | Directory or file path to index                      |
| `incremental`  | `bool`      | `True`  | Skip unchanged files                                 |
| `force`        | `bool`      | `False` | Re-index all files regardless of changes             |
| `include_glob` | `list[str]` | `None`  | Glob patterns to include, e.g. `["*.md", "*.py"]`    |
| `exclude_glob` | `list[str]` | `None`  | Glob patterns to exclude, e.g. `["*.log", "test_*"]` |
| `user_id`      | `str`       | `None`  | Scope index to a user                                |
| `agent_id`     | `str`       | `None`  | Scope index to an agent                              |
| `run_id`       | `str`       | `None`  | Scope index to a run                                 |

### `IndexResult` fields

| Field              | Type                  | Default | Description                       |
| ------------------ | --------------------- | ------- | --------------------------------- |
| `success`          | `bool`                | `True`  | Whether indexing succeeded        |
| `files_indexed`    | `int`                 | `0`     | Files newly indexed               |
| `files_skipped`    | `int`                 | `0`     | Files skipped (unchanged)         |
| `chunks_created`   | `int`                 | `0`     | Chunks created from indexed files |
| `errors`           | `list[str]`           | `[]`    | Error messages for failed files   |
| `duration_seconds` | `float`               | `0.0`   | Total indexing time               |
| `corpus_stats`     | `CorpusStats \| None` | `None`  | Corpus statistics                 |

### `CorpusStats` fields

| Field                     | Type          | Default  | Description                             |
| ------------------------- | ------------- | -------- | --------------------------------------- |
| `file_count`              | `int`         | `0`      | Number of files in corpus               |
| `chunk_count`             | `int`         | `0`      | Number of chunks created                |
| `total_tokens`            | `int`         | `0`      | Estimated total tokens                  |
| `indexed_at`              | `str \| None` | `None`   | ISO timestamp of last index             |
| `path`                    | `str \| None` | `None`   | Path to corpus root                     |
| `strategy_recommendation` | `str`         | computed | Retrieval strategy based on corpus size |

### `FileTracker` API

`FileTracker` records which vector-store chunks belong to each file so a re-index can remove stale content first.

| Method           | Signature                                       | Description                                                                                                                                |
| ---------------- | ----------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
| `mark_indexed`   | `mark_indexed(filepath, info, memory_ids=None)` | Record a file as indexed. `memory_ids` are the vector-store IDs produced for that file, so the next re-index knows which chunks to delete. |
| `get_memory_ids` | `get_memory_ids(filepath) -> list[str]`         | Return the memory IDs previously stored for a file (empty list if none).                                                                   |

**Why this matters:** editing a file and re-indexing no longer leaves the old chunks behind, so search results reflect only the current content. IDs that fail to delete are retried on the next re-index — they are never orphaned.

<Note>
  **What gets deleted** — inspect the chunks that will be removed when a file changes:

  ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
  from praisonaiagents import FileTracker

  tracker = FileTracker(state_file=".praison/.index_state.json")
  tracker.load()

  print(tracker.get_memory_ids("./docs/readme.md"))
  # ['mem_abc123', 'mem_def456', ...]  — chunks deleted on the next re-index if the file changes
  ```
</Note>

***

## Common Patterns

### Check and report results

```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from praisonaiagents import Knowledge, IndexResult

knowledge = Knowledge()
result = knowledge.index("./docs", memory={"user_id": "alice"})

if result.errors:
    for err in result.errors:
        print(f"Error: {err}")

print(f"Total: {result.files_indexed + result.files_skipped} files")
print(f"New/updated: {result.files_indexed}")
print(f"Unchanged: {result.files_skipped}")
```

### Force full re-index

```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from praisonaiagents import Knowledge

knowledge = Knowledge()

result = knowledge.index(
    "./docs",
    memory={"user_id": "alice"},
    force=True,
)
print(f"Full re-index complete: {result.files_indexed} files")
```

### Selective indexing with glob patterns

```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from praisonaiagents import Knowledge

knowledge = Knowledge()

result = knowledge.index(
    "./project",
    memory={"user_id": "alice"},
    include_glob=["*.md", "*.txt", "*.py"],
    exclude_glob=["*.log", "test_*", "__pycache__/*"],
)
```

### Corpus statistics

```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from praisonaiagents import Knowledge, CorpusStats

stats = CorpusStats.from_directory("./docs")
print(f"Files: {stats.file_count}")
print(f"Estimated tokens: {stats.total_tokens}")
print(f"Recommended strategy: {stats.strategy_recommendation}")
```

### Low-level file tracking

```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from praisonaiagents import FileTracker

tracker = FileTracker(state_file=".praison/.index_state.json")
tracker.load()

if tracker.has_changed("./docs/readme.md"):
    print("File needs re-indexing")
```

***

## Best Practices

<AccordionGroup>
  <Accordion title="Use .praisonignore to exclude noisy files">
    Create a `.praisonignore` in your corpus directory — it follows gitignore syntax and is auto-detected:

    ```gitignore theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
    *.log
    *.tmp
    test/
    tests/
    __pycache__/
    .env
    secrets.txt
    ```

    `.gitignore` is also read as a fallback.
  </Accordion>

  <Accordion title="Scope indexes per user for multi-tenant apps">
    Pass `user_id` or `agent_id` to isolate indexes per tenant. Without a scope identifier, the mem0 backend raises `ScopeRequiredError`.

    ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
    result = knowledge.index("./shared-docs", memory={"user_id": user_id})
    ```
  </Accordion>

  <Accordion title="Inspect strategy_recommendation for large corpora">
    `CorpusStats.strategy_recommendation` returns the optimal retrieval strategy based on file count:

    | Files     | Strategy       |
    | --------- | -------------- |
    | \< 10     | `direct`       |
    | \< 100    | `basic`        |
    | \< 1000   | `hybrid`       |
    | \< 10000  | `reranked`     |
    | \< 100000 | `compressed`   |
    | 100000+   | `hierarchical` |
  </Accordion>

  <Accordion title="Schedule periodic full re-indexes">
    Weekly `force=True` re-indexes are still a good hygiene practice for hash-collision paranoia and to recover any IDs that repeatedly fail to delete, but they are no longer required to keep the vector store from growing unbounded on edited files — a changed file's stale chunks are now deleted before its new chunks are added.
  </Accordion>
</AccordionGroup>

***

## Related

<CardGroup cols={2}>
  <Card title="Knowledge Backends" icon="database" href="/docs/features/knowledge-backends">
    Choose and configure the knowledge storage backend
  </Card>

  <Card title="Knowledge" icon="book" href="/docs/features/knowledge">
    Core knowledge retrieval and agent integration
  </Card>
</CardGroup>