> ## Documentation Index
> Fetch the complete documentation index at: https://praison.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Docs Code Validation

> Automatically validate Python code blocks in documentation

# Docs Code Validation

The `praisonai docs` command extracts and validates Python code blocks from your Mintlify documentation, ensuring all examples are runnable and correct.

## Quick Start

```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
# List all Python code blocks in docs
praisonai docs list

# Run all runnable Python blocks (dry-run first)
praisonai docs run --dry-run

# Execute with reports
praisonai docs run --report-dir ./reports
```

## Commands

### `praisonai docs run`

Extracts Python code blocks from documentation, classifies them as runnable or partial, executes runnable blocks, and generates reports.

```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
praisonai docs run [OPTIONS]
```

**Options:**

| Option             | Default                      | Description                                                        |
| ------------------ | ---------------------------- | ------------------------------------------------------------------ |
| `--docs-path, -p`  | Auto-detect                  | Path to documentation directory                                    |
| `--group, -g`      | None                         | Run only specific groups (top-level dirs), repeatable              |
| `--include, -i`    | `*`                          | Include patterns (glob)                                            |
| `--exclude, -e`    | None                         | Exclude patterns (glob)                                            |
| `--languages, -l`  | `python`                     | Languages to execute (comma-separated)                             |
| `--timeout, -t`    | `60`                         | Per-snippet timeout in seconds                                     |
| `--max-snippets`   | None                         | Maximum snippets to process                                        |
| `--fail-fast, -x`  | False                        | Stop on first failure                                              |
| `--dry-run`        | `False`                      | Discover and classify blocks without executing them (safe preview) |
| `--mode, -m`       | `lenient`                    | `lenient` or `strict`                                              |
| `--report-dir, -r` | `./reports/docs/[timestamp]` | Report output directory                                            |
| `--no-json`        | False                        | Skip JSON report                                                   |
| `--no-md`          | False                        | Skip Markdown report                                               |
| `--no-csv`         | False                        | Skip CSV report                                                    |
| `--require-env`    | None                         | Required env vars (skip if missing)                                |
| `--python`         | Current interpreter          | Python executable to use                                           |
| `--quiet, -q`      | False                        | Minimal output                                                     |

**Examples:**

```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
# Run with custom docs path
praisonai docs run --docs-path /path/to/docs

# Run only specific groups (top-level directories)
praisonai docs run --group models --group tools

# Run only quickstart docs
praisonai docs run --include "quickstart*"

# Run with 2-minute timeout per snippet
praisonai docs run --timeout 120

# Strict mode - fail if any blocks can't be run
praisonai docs run --mode strict

# Limit to first 10 snippets for quick check
praisonai docs run --max-snippets 10

# Use specific Python interpreter
praisonai docs run --python /path/to/python
```

### `--dry-run` preview mode

Use `--dry-run` to see exactly what `docs run` **would** execute — no subprocesses, no API calls, no cost.

```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
praisonai docs run --dry-run
```

```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
graph LR
    subgraph "docs run --dry-run"
        A[📄 Doc File] --> B[🔍 Extract Blocks]
        B --> C[🧠 Classify]
        C --> D{Runnable?}
        D -->|Yes| E[📝 Record 'not_run']
        D -->|No| F[⏭️ Skip]
        E --> G[📊 report.json]
        F --> G
    end

    classDef input fill:#6366F1,stroke:#7C90A0,color:#fff
    classDef process fill:#F59E0B,stroke:#7C90A0,color:#fff
    classDef decision fill:#189AB4,stroke:#7C90A0,color:#fff
    classDef output fill:#10B981,stroke:#7C90A0,color:#fff

    class A input
    class B,C process
    class D decision
    class E,F,G output
```

Every discovered runnable block is written to the report with:

* `status: not_run`
* `skip_reason: "Dry run"`
* The `runnable_decision` field the classifier assigned (so you can audit which blocks would have run)

<Note>
  Dry-run does **not** validate that code executes correctly — it only proves that the runner sees the block and classifies it as runnable. Use it as a fast preview before a real run, or in PRs to confirm no new blocks were unintentionally added or removed.
</Note>

#### When to use `--dry-run`

Choose between `--dry-run`, `--mode strict`, and the default lenient run:

```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
graph TB
    Start[🤔 What do you want?]
    Start --> Q1{Just check that<br/>blocks are discovered?}
    Q1 -->|Yes| DryRun[✅ Use --dry-run]
    Q1 -->|No| Q2{Want to fail CI if<br/>any blocks are unrunnable?}
    Q2 -->|Yes| Strict[✅ Use --mode strict]
    Q2 -->|No| Default[✅ Default lenient run]

    classDef start fill:#6366F1,stroke:#7C90A0,color:#fff
    classDef question fill:#F59E0B,stroke:#7C90A0,color:#fff
    classDef answer fill:#10B981,stroke:#7C90A0,color:#fff

    class Start start
    class Q1,Q2 question
    class DryRun,Strict,Default answer
```

**Contributor opening a docs PR** — add an `.mdx` file with a Python block, run `praisonai docs run --dry-run --docs-path ./docs`, confirm the new block is classified `not_run` with `skip_reason: "Dry run"`, then push so CI runs the full suite.

**Maintainer auditing a large docs tree** — run `praisonai docs run --dry-run --report-dir ./audit`, then filter `report.csv` for `status=not_run` to see the full inventory of runnable blocks — no cost, no LLM waiting.

**CI PR-preview job** — on PR events, call `praisonai docs run --dry-run` and upload `report.md` as an artifact so reviewers see what a real run would attempt without spending API tokens.

**Example: preview before a real run**

```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
# 1. See what would run
praisonai docs run --dry-run --report-dir ./preview

# 2. Inspect
cat ./preview/report.json | jq '.results[] | select(.skip_reason=="Dry run") | {source_path, block_index, runnable_decision}'

# 3. Real run once satisfied
praisonai docs run --report-dir ./actual
```

### `praisonai docs list`

Lists all discovered code blocks without executing them.

```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
praisonai docs list [OPTIONS]
```

**Options:**

| Option            | Default     | Description                     |
| ----------------- | ----------- | ------------------------------- |
| `--docs-path, -p` | Auto-detect | Path to documentation directory |
| `--group, -g`     | None        | Filter by group (top-level dir) |
| `--include, -i`   | None        | Include patterns (glob)         |
| `--exclude, -e`   | None        | Exclude patterns (glob)         |
| `--languages, -l` | `python`    | Languages to show               |
| `--code, -c`      | False       | Show code preview               |
| `--groups`        | False       | Show available groups only      |

**Examples:**

```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
# List available groups
praisonai docs list --groups

# List blocks in specific group
praisonai docs list --group models --code
```

## Code Block Classification

The system classifies each Python code block as:

| Status       | Description                                                   |
| ------------ | ------------------------------------------------------------- |
| **Runnable** | Has imports AND terminal action (`.start()`, `print()`, etc.) |
| **Partial**  | Missing imports or incomplete - not executed                  |
| **Skipped**  | Marked with skip directive or contains `input()`              |

### Terminal Actions Detected

* `agents.start()` / `agent.start()`
* `print()`
* `asyncio.run()`
* `if __name__ == "__main__":`

## Directives

Control execution behavior with HTML comments above code blocks:

````markdown theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
<!-- praisonai: runnable=true timeout=120 require_env=OPENAI_API_KEY -->
```python
from praisonaiagents import Agent
agent = Agent(instructions="test")
agent.start()
````

```

### Available Directives

| Directive | Values | Description |
|-----------|--------|-------------|
| `runnable` | `true/false` | Force block to be runnable or skipped |
| `skip` | `true/false` | Skip this block entirely |
| `timeout` | seconds | Custom timeout for this block |
| `require_env` | `KEY1,KEY2` | Required environment variables |

## Reports

Reports are generated in the specified directory:

```

reports/docs/\[timestamp]/
├── report.json      # Machine-readable results
├── report.md        # Human-readable summary
├── report.csv       # CSV for quick scanning/filtering
└── logs/
├── quickstart\_\_0.stdout.log
└── quickstart\_\_0.stderr.log

````

### CSV Report

The CSV report contains one row per item for easy filtering:

| Column | Description |
|--------|-------------|
| `suite` | "docs" |
| `group` | Top-level directory (e.g., "models", "tools") |
| `item_id` | Unique identifier |
| `source_path` | Path to doc file |
| `block_index` | Code block index in file |
| `status` | passed/failed/skipped/timeout/not_run |
| `duration_seconds` | Execution time |
| `error_message` | Error details if failed |
| `code_hash` | Content hash for change tracking |

### JSON Report Schema

```json
{
  "metadata": {
    "timestamp": "2026-01-09T11:30:00Z",
    "platform": "Darwin-23.0.0-arm64",
    "python_version": "3.11.0",
    "praisonai_version": "2.0.0",
    "docs_path": "/path/to/docs",
    "totals": {
      "passed": 10,
      "failed": 2,
      "skipped": 5,
      "timeout": 0,
      "not_run": 15
    }
  },
  "results": [
    {
      "item_id": "docs/quickstart.mdx::0",
      "source_path": "docs/quickstart.mdx",
      "block_index": 0,
      "language": "python",
      "line_start": 30,
      "line_end": 39,
      "runnable_decision": "heuristic_standalone",
      "status": "not_run",
      "skip_reason": "Dry run",
      "code_hash": "sha256:…"
    }
  ]
}
````

`skip_reason: "Dry run"` identifies items that were only previewed. Other `not_run` items come from `runnable=false` directives or lenient-mode classification.

## CI/CD Integration

Add to your GitHub Actions workflow:

```yaml theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
name: Docs Validation

on: [push, pull_request]

jobs:
  validate-docs:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      
      - name: Install dependencies
        run: pip install praisonai
      
      - name: Validate docs code blocks
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: |
          praisonai docs run \
            --docs-path ./docs \
            --report-dir ./reports \
            --no-stream
      
      - name: Upload reports
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: docs-validation-report
          path: ./reports/
```

**Pull-request preview (no API keys required):**

```yaml theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
- name: Preview docs code blocks (no execution)
  run: |
    praisonai docs run \
      --docs-path ./docs \
      --dry-run \
      --report-dir ./preview

- name: Comment discovery summary
  # Post the report.md as a PR comment so reviewers see what a full run would attempt
```

### `praisonai docs run-all`

Runs all documentation code blocks group-by-group with parallel execution by default.

```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
praisonai docs run-all [OPTIONS]
```

**Options:**

| Option                    | Default                                | Description                                       |
| ------------------------- | -------------------------------------- | ------------------------------------------------- |
| `--docs-path, -p`         | Auto-detect                            | Path to documentation directory                   |
| `--timeout, -t`           | `60`                                   | Per-snippet timeout in seconds                    |
| `--report-dir, -r`        | `~/Downloads/reports/docs/[timestamp]` | Report output directory                           |
| `--parallel/--sequential` | `parallel`                             | Run groups in parallel or sequential              |
| `--workers, -w`           | `4`                                    | Max parallel workers                              |
| `--quiet, -q`             | False                                  | Minimal output                                    |
| `--ci`                    | False                                  | CI-friendly output (no colors, proper exit codes) |

**Examples:**

```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
# Run all groups in parallel (default)
praisonai docs run-all

# Run sequentially for debugging
praisonai docs run-all --sequential

# Run with more workers for faster execution
praisonai docs run-all --workers 8

# CI-friendly output
praisonai docs run-all --ci
```

### `praisonai docs stats`

Shows statistics for documentation code blocks by group.

```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
praisonai docs stats [OPTIONS]
```

**Options:**

| Option            | Default     | Description                     |
| ----------------- | ----------- | ------------------------------- |
| `--docs-path, -p` | Auto-detect | Path to documentation directory |
| `--group, -g`     | None        | Filter by group (top-level dir) |
| `--languages, -l` | `python`    | Languages to show               |

## Exit Codes

| Code | Meaning                                |
| ---- | -------------------------------------- |
| `0`  | All runnable blocks passed             |
| `1`  | One or more blocks failed or timed out |
| `2`  | Invalid path or configuration error    |

In strict mode (`--mode strict`), exit code 1 is also returned if any blocks are marked as `not_run`.

***

## CLI Command Validation

The `praisonai docs cli` subcommand group validates CLI commands found in bash code blocks within documentation. It discovers `praisonai` commands and validates them by running with `--help` to ensure they exist and work correctly.

### Quick Start

```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
# Show CLI command statistics
praisonai docs cli stats

# List all discovered CLI commands
praisonai docs cli list

# Validate all CLI commands from docs
praisonai docs cli run-all

# Validate with limited scope
praisonai docs cli run-all --max-items 10 --timeout 5
```

### `praisonai docs cli run-all`

Validates all CLI commands from documentation by running each with `--help`.

```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
praisonai docs cli run-all [OPTIONS]
```

**Options:**

| Option                    | Default                                    | Description                                       |
| ------------------------- | ------------------------------------------ | ------------------------------------------------- |
| `--docs-path, -p`         | `~/PraisonAIDocs/docs/cli`                 | Path to CLI documentation directory               |
| `--timeout, -t`           | `10`                                       | Per-command timeout in seconds                    |
| `--report-dir, -r`        | `~/Downloads/reports/docs-cli/<timestamp>` | Report output directory                           |
| `--parallel/--sequential` | `parallel`                                 | Run commands in parallel or sequential            |
| `--workers, -w`           | `4`                                        | Max parallel workers                              |
| `--max-items`             | None                                       | Maximum commands to run                           |
| `--group, -g`             | None                                       | Run only specific groups (repeatable)             |
| `--quiet, -q`             | False                                      | Minimal output                                    |
| `--ci`                    | False                                      | CI-friendly output (no colors, proper exit codes) |

**Examples:**

```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
# Run all CLI commands in parallel (default)
praisonai docs cli run-all

# Validate first 10 commands for quick check
praisonai docs cli run-all --max-items 10

# Run with more workers for faster execution
praisonai docs cli run-all --workers 8

# CI-friendly output with proper exit codes
praisonai docs cli run-all --ci --timeout 5

# Run sequentially for debugging
praisonai docs cli run-all --sequential
```

### `praisonai docs cli list`

Lists all discovered CLI commands from documentation without executing them.

```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
praisonai docs cli list [OPTIONS]
```

**Options:**

| Option            | Default     | Description                     |
| ----------------- | ----------- | ------------------------------- |
| `--docs-path, -p` | Auto-detect | Path to documentation directory |
| `--group, -g`     | None        | Filter by group (repeatable)    |
| `--runnable`      | False       | Show only runnable commands     |
| `--groups`        | False       | Show available groups only      |

**Examples:**

```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
# List all CLI commands
praisonai docs cli list

# List available groups
praisonai docs cli list --groups

# List only runnable commands
praisonai docs cli list --runnable
```

### `praisonai docs cli stats`

Shows statistics for CLI commands found in documentation.

```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
praisonai docs cli stats [OPTIONS]
```

**Options:**

| Option            | Default     | Description                     |
| ----------------- | ----------- | ------------------------------- |
| `--docs-path, -p` | Auto-detect | Path to documentation directory |

**Example Output:**

```
CLI Commands Statistics
========================================
Total commands:    1463
Runnable:          933
Skipped:           530

By Group:
----------------------------------------
  databases              31/37   runnable
  monitoring              8/9    runnable
  root                  894/1417 runnable
```

### `praisonai docs cli report`

View CLI validation report with failures and error grouping.

```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
praisonai docs cli report [REPORT_DIR] [OPTIONS]
```

**Options:**

| Option           | Default                        | Description                       |
| ---------------- | ------------------------------ | --------------------------------- |
| `--base-dir, -b` | `~/Downloads/reports/docs-cli` | Base directory for auto-detection |
| `--limit, -n`    | `20`                           | Max items to show (0 = unlimited) |
| `--format, -f`   | `table`                        | Output format: `table` or `json`  |

**Examples:**

```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
# View latest report
praisonai docs cli report

# View specific report directory
praisonai docs cli report ./my-report

# JSON output for parsing
praisonai docs cli report --format json
```

### Command Classification

CLI commands are classified as:

| Status       | Description                                                     |
| ------------ | --------------------------------------------------------------- |
| **Runnable** | Valid `praisonai` command without placeholders                  |
| **Skipped**  | Contains placeholders like `<file>`, `[OPTIONS]`, or `path/to/` |

### Placeholder Detection

Commands are automatically skipped if they contain:

* `[OPTIONS]` or `[...]`
* `<file>`, `<name>`, `<path>` style placeholders
* `path/to/`, `/my/project/` style example paths
* `your_`, `example_` prefixed values

### CI/CD Integration for CLI Validation

Add to your GitHub Actions workflow:

```yaml theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
name: CLI Docs Validation

on: [push, pull_request]

jobs:
  validate-cli-docs:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      
      - name: Install dependencies
        run: pip install praisonai
      
      - name: Validate CLI commands in docs
        run: |
          praisonai docs cli run-all \
            --docs-path ./docs/cli \
            --ci \
            --timeout 10
      
      - name: Upload reports
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: cli-validation-report
          path: ~/Downloads/reports/docs-cli/
```