> ## Documentation Index
> Fetch the complete documentation index at: https://praison.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Benchmarks

> Performance benchmarks for AI SDK vs Native backends

PraisonAI includes a benchmark suite to measure performance with and without AI SDK loaded.

```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
graph LR
    App([App]) --> Bench[Benchmarks]
    Bench --> Metrics([Metrics])

    classDef agent fill:#8B0000,stroke:#7C90A0,color:#fff
    classDef tool fill:#189AB4,stroke:#7C90A0,color:#fff

    class Bench agent
    class App,Metrics tool
    classDef agent fill:#8B0000,color:#fff
    classDef tool fill:#189AB4,color:#fff
```

## Quick Start

<Steps>
  <Step title="Simple Usage">
    ```typescript theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
    import { resolveBackend, isAISDKAvailable } from 'praisonai';

    const available = await isAISDKAvailable();
    console.log('AI SDK available:', available);

    const { source } = await resolveBackend('openai/gpt-4o-mini');
    console.log('Backend source:', source);
    ```
  </Step>

  <Step title="With Configuration">
    ```typescript theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
    process.env.PRAISONAI_BACKEND = 'auto';
    const { provider, source } = await resolveBackend('openai/gpt-4o-mini');
    console.log(source, provider);
    ```
  </Step>
</Steps>

## What's Measured

### Import Time

Measures cold start import time:

* **Core Import**: Just the Agent class (no AI SDK)
* **AI SDK Import**: Loading the `ai` package
* **Full Import**: Complete praisonai module

### Memory Usage

Measures heap memory after:

* Module import
* Agent creation
* First LLM call

### Latency

Measures time for:

* Backend resolution
* First API call (with real keys)
* Streaming throughput

### Embedding Throughput

Measures vectors per second for batch embeddings.

## Running Benchmarks

### Programmatic

```typescript theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
import { resolveBackend, isAISDKAvailable } from 'praisonai';

async function benchmark() {
  const iterations = 10;
  const times: number[] = [];
  
  for (let i = 0; i < iterations; i++) {
    const start = performance.now();
    await resolveBackend('openai/gpt-4o-mini');
    times.push(performance.now() - start);
  }
  
  const mean = times.reduce((a, b) => a + b) / times.length;
  console.log(`Backend resolution: ${mean.toFixed(2)}ms`);
}
```

### Benchmark Script

Create a benchmark script:

```typescript theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
// benchmarks/run.ts
import { Agent } from 'praisonai';
import { embed, embedMany } from 'praisonai';

async function runBenchmarks() {
  console.log('=== PraisonAI Benchmarks ===\n');
  
  // 1. Import time
  console.log('1. Import Time');
  const importStart = performance.now();
  const { resolveBackend } = await import('praisonai');
  console.log(`   Full import: ${(performance.now() - importStart).toFixed(2)}ms`);
  
  // 2. Backend resolution
  console.log('\n2. Backend Resolution');
  const resolveStart = performance.now();
  const { source } = await resolveBackend('openai/gpt-4o-mini');
  console.log(`   Resolution: ${(performance.now() - resolveStart).toFixed(2)}ms`);
  console.log(`   Source: ${source}`);
  
  // 3. Agent creation
  console.log('\n3. Agent Creation');
  const agentStart = performance.now();
  const agent = new Agent({ instructions: 'Test' });
  console.log(`   Creation: ${(performance.now() - agentStart).toFixed(2)}ms`);
  
  // 4. Embedding (if API key available)
  if (process.env.OPENAI_API_KEY) {
    console.log('\n4. Embedding Throughput');
    const texts = Array(10).fill('Test text for embedding');
    const embedStart = performance.now();
    await embedMany(texts);
    const embedTime = performance.now() - embedStart;
    console.log(`   10 embeddings: ${embedTime.toFixed(2)}ms`);
    console.log(`   Throughput: ${(10000 / embedTime).toFixed(2)} vectors/sec`);
  }
  
  console.log('\n=== Complete ===');
}

runBenchmarks();
```

## Benchmark Results

Typical results on modern hardware:

| Metric             | Without AI SDK | With AI SDK | Overhead |
| ------------------ | -------------- | ----------- | -------- |
| Core Import        | \~25ms         | \~25ms      | 0ms      |
| AI SDK Import      | N/A            | \~35ms      | +35ms    |
| Full Import        | \~30ms         | \~60ms      | +30ms    |
| Backend Resolution | \~1ms          | \~5ms       | +4ms     |
| Memory (Agent)     | \~2MB          | \~3MB       | +1MB     |

### Key Findings

1. **AI SDK adds \~35ms import overhead** - Only when actually loaded
2. **Lazy loading works** - AI SDK not loaded until needed
3. **Memory overhead minimal** - \~1MB additional
4. **No runtime penalty** - After initial load, performance is identical

## Zero Impact Guarantee

PraisonAI guarantees zero performance impact when AI SDK is not used:

```typescript theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
// This does NOT load AI SDK
import { Agent } from 'praisonai';
const agent = new Agent({ instructions: 'Test' });

// AI SDK only loaded when:
// 1. Backend resolver selects AI SDK
// 2. Embeddings are used
// 3. PRAISONAI_BACKEND=ai-sdk is set
```

### Verification

```typescript theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
// Check if AI SDK is loaded
const modulesBefore = Object.keys(require.cache).filter(k => k.includes('ai'));
console.log('AI SDK modules before:', modulesBefore.length); // 0

// Create agent (doesn't load AI SDK)
const agent = new Agent({ instructions: 'Test' });

const modulesAfter = Object.keys(require.cache).filter(k => k.includes('/ai/'));
console.log('AI SDK modules after:', modulesAfter.length); // Still 0
```

## Environment Control

### Force Native Backend

```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
# Skip AI SDK entirely
export PRAISONAI_BACKEND=native
```

### Force AI SDK

```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
# Always use AI SDK (error if not installed)
export PRAISONAI_BACKEND=ai-sdk
```

### Auto Selection (Default)

```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
# Use AI SDK if available, fallback to native
export PRAISONAI_BACKEND=auto
```

## CI/CD Integration

### GitHub Actions

```yaml theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
- name: Run Benchmarks
  run: |
    npm run benchmark
    
- name: Check Performance Regression
  run: |
    # Fail if import time > 100ms
    node -e "
      const start = Date.now();
      require('praisonai');
      const time = Date.now() - start;
      if (time > 100) process.exit(1);
    "
```

### Benchmark in Tests

```typescript theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
describe('Performance', () => {
  it('should import in under 100ms', async () => {
    const start = performance.now();
    await import('praisonai');
    expect(performance.now() - start).toBeLessThan(100);
  });
  
  it('should not load AI SDK unless needed', () => {
    const aiModules = Object.keys(require.cache)
      .filter(k => k.includes('/ai/'));
    expect(aiModules.length).toBe(0);
  });
});
```

## Related

<CardGroup cols={2}>
  <Card title="Benchmarks CLI" icon="terminal" href="/docs/js/benchmarks-cli">
    Run benchmarks from the terminal
  </Card>

  <Card title="AI SDK" icon="book" href="/docs/js/ai-sdk">
    Optional AI SDK backend
  </Card>

  <Card title="Agent" icon="robot" href="/docs/js/agent">
    Agent runtime
  </Card>
</CardGroup>