Performance Benchmarks

PraisonAI Agents includes comprehensive benchmarks to measure and compare performance against other popular AI agent frameworks.

Benchmark Types

Benchmark	Description	File
Simple	Agent instantiation time (no API calls). LangGraph and CrewAI are constructed with a tool in this script; use `performance_benchmark.py` to compare like with like.	`simple_benchmark.py`
Tools	Agent instantiation with tools	`tools_benchmark.py`
Execution	Real agent execution with LLM API calls	`execution_benchmark.py`

Quick Start

cd praisonai-agents

# Run instantiation benchmark
python benchmarks/simple_benchmark.py

# Run tools benchmark
python benchmarks/tools_benchmark.py

# Run real execution benchmark (requires API key)
export OPENAI_API_KEY=your_key
python benchmarks/execution_benchmark.py

Simple Benchmark

Measures PraisonAI agent instantiation time (no API calls, no tools) and compares against installed frameworks. LangGraph and CrewAI are constructed with a tool in this script; use performance_benchmark.py to compare like with like.

CLI Options

python benchmarks/simple_benchmark.py [OPTIONS]

Option	Default	Description
`--save`	`False`	Save `BENCHMARK_RESULTS.md` and update project `README.md`.
`--skip-crewai`	`False`	Skip the optional CrewAI benchmark (its import can hang 4+ min on some platforms).
`--json`	`False`	Emit `{"results": {...}, "iterations": N, "warmup": M}` to stdout. Human progress output is suppressed; save-status goes to stderr so stdout stays machine-readable.
`--fail-over-threshold`	`False`	Exit non-zero when PraisonAI instantiation exceeds `praisonai_instantiation_us_max` in `thresholds.json`. Prints `ERROR: PraisonAI instantiation X us > threshold Y us` on stderr.

Thresholds file

The script reads benchmarks/thresholds.json for CI budgets:

{
  "praisonai_instantiation_us_max": 450,
  "iterations": 100,
  "warmup": 10
}

praisonai_instantiation_us_max — hard cap (µs) for --fail-over-threshold.
iterations / warmup — override the script’s defaults.

Examples

# Default run (same output as before)
python benchmarks/simple_benchmark.py

# CI baseline: skip the hang-prone framework, emit machine-readable JSON
python benchmarks/simple_benchmark.py --skip-crewai --json

# CI gate: fail the job on a regression
python benchmarks/simple_benchmark.py --skip-crewai --json --fail-over-threshold \
  | tee agent-perf-baseline.json

Execution Benchmark

The execution benchmark tests real agent execution with actual LLM API calls.

CLI Options

python benchmarks/execution_benchmark.py [OPTIONS]

Option	Short	Default	Description
`--model`	`-m`	`gpt-4o-mini`	Model to use
`--iterations`	`-i`	`3`	Number of iterations
`--prompt`	`-p`	`"What is 2+2?..."`	Prompt to use
`--no-save`	-	`False`	Don’t save results to file

Examples

# Default (3 iterations, gpt-4o-mini)
python benchmarks/execution_benchmark.py

# Custom iterations
python benchmarks/execution_benchmark.py --iterations 5
python benchmarks/execution_benchmark.py -i 10

# Custom model
python benchmarks/execution_benchmark.py --model gpt-4o
python benchmarks/execution_benchmark.py -m gpt-4o

# Custom prompt
python benchmarks/execution_benchmark.py --prompt "What is the capital of France?"

# Don't save results
python benchmarks/execution_benchmark.py --no-save

# Combine options
python benchmarks/execution_benchmark.py -m gpt-4o -i 5 --no-save

Results

Agent Instantiation (June 2026)

Framework	Avg Time (μs)
OpenAI Agents SDK 0.7.0	5.05
Agno 2.4.2	5.27
PraisonAI 1.6.60	13.62 (min 12.58, median 12.96)
PraisonAI (LiteLLM route)	14.49

Measured 2026-06-17 · 10 warmup + 1000 iterations · source: src/praisonai-agents/benchmarks/BENCHMARK_RESULTS.md

Competitor rows have no min/median columns and look like no-tools runs, while the file states one tool per framework. Treat cross-framework gaps as indicative only.

Historical (superseded)

Earlier results (December 2025) and the tools and execution benchmarks were measured under different setups and are no longer published. Past results live in benchmarks/archive/.

Frameworks Compared

Framework	Execution Method
PraisonAI	`agent.start()`
PraisonAI (LiteLLM)	`agent.start()`
Agno	`agent.run()`
CrewAI	`crew.kickoff()`
OpenAI Agents SDK	`Runner.run()`
LangGraph	`create_react_agent()`
PydanticAI	`Agent()`

Output Files

Benchmarks save results to markdown files in the benchmarks/ directory:

File	Description
`BENCHMARK_RESULTS.md`	Current instantiation benchmark results
`archive/`	Past results

Methodology

Measures only Agent(...) construction: no LLM call, no import time. 10 warmup + 1000 iterations with time.perf_counter(), one tool. Single machine; numbers vary by hardware and version. Reproduce with python benchmarks/performance_benchmark.py. Last measured: 2026-06-17.

Running Your Own Benchmarks

Clone the repository:

git clone https://github.com/MervinPraison/PraisonAI.git
cd PraisonAI/src/praisonai-agents

Install dependencies:

pip install praisonaiagents agno crewai pydantic-ai openai-agents langgraph

Set your API key:

export OPENAI_API_KEY=your_key

Run benchmarks:

python benchmarks/simple_benchmark.py
python benchmarks/tools_benchmark.py
python benchmarks/execution_benchmark.py

Getting Started

Install

Core Concepts

Guides

Features

Models

Databases

Observability

Memory

Telemetry & Analytics

Knowledge

RAG

Persistence

Tools

Other Features

Developers

Configuration

Best Practices

Getting Started (No Code)

Benchmarks

Performance Benchmarks

Benchmark Types

Quick Start

Simple Benchmark

CLI Options

Thresholds file

Examples

Execution Benchmark

CLI Options

Examples

Results

Agent Instantiation (June 2026)

Frameworks Compared

Output Files

Methodology

Methodology

Running Your Own Benchmarks

​Performance Benchmarks

​Benchmark Types

​Quick Start

​Simple Benchmark

​CLI Options

​Thresholds file

​Examples

​Execution Benchmark

​CLI Options

​Examples

​Results

​Agent Instantiation (June 2026)

​Frameworks Compared

​Output Files

​Methodology

Methodology

​Running Your Own Benchmarks

Performance Benchmarks

Benchmark Types

Quick Start

Simple Benchmark

CLI Options

Thresholds file

Examples

Execution Benchmark

CLI Options

Examples

Results

Agent Instantiation (June 2026)

Frameworks Compared

Output Files

Methodology

Running Your Own Benchmarks