Benchmarks

Performance benchmarks comparing PraisonAI Agents with other frameworks

Performance Benchmarks

PraisonAI Agents includes comprehensive benchmarks to measure and compare performance against other popular AI agent frameworks.

Benchmark Types

Quick Start

cd src/praisonai-agents

# Run instantiation benchmark
python benchmarks/simple_benchmark.py

# Run tools benchmark
python benchmarks/tools_benchmark.py

# Run real execution benchmark (requires API key)
export OPENAI_API_KEY=your_key
python benchmarks/execution_benchmark.py

Simple Benchmark

Measures PraisonAI agent instantiation time (no API calls, no tools) and compares against installed frameworks. Every framework is built without tools.

CLI Options

python benchmarks/simple_benchmark.py [OPTIONS]

Thresholds file

The script reads benchmarks/thresholds.json for CI budgets:

{
  "praisonai_instantiation_us_max": 150,
  "iterations": 100,
  "warmup": 10
}

• praisonai_instantiation_us_max — hard cap (µs) for --fail-over-threshold. Set to ≈3× the typical ~48 μs we measure on the GitHub-hosted runner.
• iterations / warmup — override the script’s defaults.

Examples

# Default run (same output as before)
python benchmarks/simple_benchmark.py

# CI baseline: skip the hang-prone framework, emit machine-readable JSON
python benchmarks/simple_benchmark.py --skip-crewai --json

# CI gate: fail the job on a regression
python benchmarks/simple_benchmark.py --skip-crewai --json --fail-over-threshold \
  | tee agent-perf-baseline.json

Performance Benchmark

performance_benchmark.py produces the cross-framework table in BENCHMARK_RESULTS.md. Competitor frameworks are imported lazily, so it runs with only praisonaiagents installed and skips anything missing.

# PraisonAI only, quick run
python benchmarks/performance_benchmark.py --only praisonai --iterations 200

Execution Benchmark

The execution benchmark tests real agent execution with actual LLM API calls.

Each framework makes untimed warm-up calls first, so cold imports and the first connection are not counted. The timed runs are interleaved round by round, so network drift affects every framework alike. A run that raises or returns an empty answer counts as a failure, not a timing. The summary reports the median, the mean and the number of successful and failed runs.

CLI Options

python benchmarks/execution_benchmark.py [OPTIONS]

Examples

# Default (10 rounds after 1 warm-up call each, gpt-4o-mini, not saved)
python benchmarks/execution_benchmark.py

# More rounds for steadier medians
python benchmarks/execution_benchmark.py -i 30

# Custom model
python benchmarks/execution_benchmark.py -m gpt-4o

# Custom prompt
python benchmarks/execution_benchmark.py --prompt "What is the capital of France?"

# Save results
python benchmarks/execution_benchmark.py --save

# Combine options
python benchmarks/execution_benchmark.py -m gpt-4o -i 20 --save

Results

Agent Instantiation (October 2026)

Measured 2026-10-15 on Linux x86_64 (4 CPUs, Python 3.12) · one tool · 10 warmup + 1000 iterations · single script run with the same setup for every framework · source: src/praisonai-agents/benchmarks/BENCHMARK_RESULTS.md

Frameworks Compared

Output Files

Benchmarks save results to markdown files in the benchmarks/ directory:

Nothing is written unless you pass --save.

Methodology

Running Your Own Benchmarks

1. Clone the repository:

git clone https://github.com/MervinPraison/PraisonAI.git
cd PraisonAI/src/praisonai-agents

1. Install dependencies:

pip install "praisonaiagents[llm]" agno crewai pydantic-ai openai-agents langgraph langchain-openai

1. Set your API key:

export OPENAI_API_KEY=your_key

1. Run benchmarks:

python benchmarks/simple_benchmark.py
python benchmarks/tools_benchmark.py
python benchmarks/execution_benchmark.py