Testing Guide

How to run and organize tests in PraisonAI

PraisonAI Testing Guide

This guide explains how to run tests, understand test tiers, and work with provider-specific tests in PraisonAI.

Quick Start

# Run smoke tests (fastest, ~2 min)
praisonai test --tier smoke

# Run main tests (default, ~5 min)
praisonai test

# Run with parallelization
praisonai test --parallel auto

# Run OpenAI live tests (requires API key)
praisonai test --provider openai --live

Test Tiers

PraisonAI uses a tiered testing approach to balance speed and coverage:

Running Specific Tiers

# Smoke tier - fastest, for quick validation
praisonai test --tier smoke

# Main tier - default for development
praisonai test --tier main

# Extended tier - all providers
praisonai test --tier extended --live

# Nightly tier - comprehensive
praisonai test --tier nightly --live

Provider Tests

Tests that require specific LLM providers are automatically detected and gated:

Running Provider-Specific Tests

# Run only OpenAI tests
praisonai test --provider openai --live

# Run only Anthropic tests
praisonai test --provider anthropic --live

# Run all provider tests
praisonai test --provider all --live

Environment Variables

Placeholder API Keys in Mock Tests

The autouse setup_test_environment fixture in src/praisonai/tests/conftest.py injects a 'test-key' placeholder for OPENAI_API_KEY, ANTHROPIC_API_KEY, GOOGLE_API_KEY, XAI_API_KEY, GROQ_API_KEY, and COHERE_API_KEY so unit tests can construct LLM clients without real credentials.

Since May 2026 (PR #1663), the placeholder is only injected when the variable is unset or empty. If you already exported a real key, it is left untouched.

Implication: if you export a real key and run mock unit tests, any test that isn’t fully mocked will make real, billable API calls. To force the placeholder path, unset the keys first (or run in a clean shell). The fixture skips placeholder injection entirely for tests marked real, network, e2e, or provider_*, and for tests under tests/integration/, tests/live/, or tests/e2e/.

Provider API keys in unit/mock tests

The setup_test_environment autouse fixture preserves any provider API key already present in the shell environment.

Placeholder "test-key" values are injected only for keys that are unset, so SDK imports and guardrails never crash on missing keys. On teardown, originally-set keys keep their values; keys that were missing and got placeholders are removed (not left behind as "test-key").

Tests marked @pytest.mark.real, @pytest.mark.network, @pytest.mark.e2e, or any @pytest.mark.provider_* skip the placeholder injection entirely, so they always see the real environment. Tests located under tests/integration/, tests/live/, or tests/e2e/ are also treated as real and skip placeholder injection.

Provider keys covered by the fixture: OPENAI_API_KEY, ANTHROPIC_API_KEY, GOOGLE_API_KEY, XAI_API_KEY, GROQ_API_KEY, COHERE_API_KEY.

CLI Options

praisonai test [OPTIONS]

Options:
  --tier, -t TEXT        Test tier: smoke, main, extended, nightly
  --provider, -p TEXT    Provider: openai, anthropic, google, ollama, etc.
  --live, -l             Enable live API tests
  --allow-network        Allow network access in tests
  --parallel, -n TEXT    Parallel workers: auto or number
  --verbose, -v          Verbose output
  --coverage, -c         Generate coverage report
  --fail-fast, -x        Stop on first failure

Test Markers

Tests are automatically marked based on their location and content:

Type Markers

• unit – Pure unit tests, no network
• integration – Local integration tests
• e2e – End-to-end tests with external calls

Provider Markers

• provider_openai – Requires OpenAI API
• provider_anthropic – Requires Anthropic API
• provider_google – Requires Google/Gemini API
• provider_ollama – Requires Ollama running locally
• provider_grok_xai – Requires xAI/Grok API
• provider_groq – Requires Groq API
• provider_cohere – Requires Cohere API

Infrastructure Markers

• network – Makes outbound network calls
• local_service – Requires Docker or local service
• offline – Fully mocked; opts the test out of provider and network auto-marking. Use on a test or as pytestmark = pytest.mark.offline at the top of a fully-mocked module that mentions provider names in its source.
• slow – Takes more than 5 seconds
• flaky – Known intermittent failures
• allow_sleep – Opt out of fast_sleep fixture

Timing-sensitive tests (fast_sleep / allow_sleep)

By default, unit tests replace time.sleep with a near-instant stub so suites finish quickly. That breaks tests which assert real delays, backoff, or scheduler timing.

import pytest
import time

@pytest.mark.allow_sleep
def test_retry_backoff_waits():
    start = time.monotonic()
    time.sleep(0.05)  # real sleep — not patched when allow_sleep is set
    assert time.monotonic() - start >= 0.04

Run only tests that need real sleeps:

pytest tests/unit/test_scheduler.py -m allow_sleep

Or disable the fast_sleep fixture for one file by marking every test @pytest.mark.allow_sleep.

Direct pytest Usage

You can also use pytest directly:

# Smoke tests
cd src/praisonai
pytest tests/unit/ -m "not slow and not network" -q --timeout=30

# Main tests with parallelization
pytest tests/unit/ tests/integration/ \
  -m "not provider_anthropic and not provider_google" \
  -n 2 --timeout=60

# OpenAI live tests
PRAISONAI_LIVE_TESTS=1 pytest tests/ -m "provider_openai" --timeout=120

Network Guard

By default, unit tests block outbound network calls. This prevents:

• Accidental API calls during development
• Flaky tests due to network issues
• Unexpected costs from live API calls

To allow network access:

# Via CLI
praisonai test --live

# Via environment
PRAISONAI_ALLOW_NETWORK=1 pytest tests/

CI/CD Integration

The test suite is designed for CI with separate jobs:

1. smoke – Runs on every push, fast validation
2. main – Runs on PR/main, comprehensive unit + integration
3. openai-live – Runs when OPENAI_API_KEY secret is set
4. extended-* – Provider-specific jobs, nightly schedule

See .github/workflows/test-optimized.yml for the full CI configuration.

Writing Tests

Test Location

• tests/unit/ – Unit tests (no network)
• tests/integration/ – Integration tests
• tests/e2e/ – End-to-end tests
• tests/live/ – Live API tests

Marking Provider Tests

Tests are auto-detected, but you can explicitly mark:

import pytest

@pytest.mark.provider_anthropic
def test_anthropic_feature():
    """This test requires Anthropic API."""
    pass

@pytest.mark.network
def test_external_api():
    """This test makes network calls."""
    pass

@pytest.mark.offline
def test_fully_mocked():
    """Fully mocked — opt out of provider/network auto-marking."""
    pass

@pytest.mark.slow
def test_long_running():
    """This test takes a while."""
    pass

Provider markers are inferred per test function. The gating plugin parses each test’s own source segment plus its decorators; module-level context (imports, pytestmark, fixtures) is shared. A provider name inside one test body does not leak to siblings.

If your test file is fully mocked but references a provider name — for example a test_openai_*.py that patches litellm.completion — declare it offline so CI actually selects it:

import pytest
from unittest.mock import patch

pytestmark = pytest.mark.offline

@patch('litellm.completion')
def test_something(mock_completion):
    ...

offline disables both provider and network auto-marking for that test.

The network marker itself is still inferred whole-file from provider keywords, so a live-network test cannot silently un-gate just because a neighbour narrowed its provider surface. Only offline (or an explicit real/network marker) changes that.

Using Fixtures

Common fixtures are available in conftest.py:

• setup_test_environment – autouse fixture that fills missing provider API keys with placeholders; preserves real keys from your shell
• mock_llm_response – provides mock LLM responses for testing
• temp_directory – provides temporary directory for file operations

def test_with_mock_llm(mock_llm_response):
    """Use mock LLM response."""
    assert mock_llm_response['choices'][0]['message']['content']

def test_with_temp_dir(temp_directory):
    """Use temporary directory."""
    file_path = temp_directory / "test.txt"
    file_path.write_text("test")

Troubleshooting

Tests Skipped Unexpectedly

Check if the test is being auto-marked as a provider test:

pytest tests/path/to/test.py --collect-only -q

Network Blocked Error

If you see NetworkBlockedError, the test needs network access:

PRAISONAI_ALLOW_NETWORK=1 pytest tests/path/to/test.py

Timeout Errors

Increase the timeout for slow tests:

pytest tests/ --timeout=120

Or mark the test as slow:

@pytest.mark.slow
def test_slow_operation():
    pass

Pytest Failure Categories

For common test failures and how to distinguish real regressions from stale mocks, see the Pytest Failure Categories Guide.