Skip to main content
Run integration tests against live LLM providers — gated by RUN_REAL_KEY_TESTS so CI stays free and fast by default. 
import os
import pytest
from praisonaiagents import Agent

@pytest.mark.skipif(
    os.getenv("RUN_REAL_KEY_TESTS", "").lower() not in ("1", "true", "yes"),
    reason="Set RUN_REAL_KEY_TESTS=1 to enable",
)
def test_agent_simple_chat():
    agent = Agent(
        name="TestAgent",
        instructions="Reply briefly.",
        llm="gpt-4o-mini",
    )
    response = agent.start("Say hello in one word")
    assert len(response) > 0
The user sets RUN_REAL_KEY_TESTS=1; pytest runs a real agent chat against live provider keys.

​
How It Works

​
Quick Start

1

Simple Usage

Enable the gate and set your API key:
export RUN_REAL_KEY_TESTS=1
export OPENAI_API_KEY="$(printenv OPENAI_API_KEY)"

python -m pytest tests/integration/test_real_api.py -v
2

With Configuration

Run a single test class for one provider:
export RUN_REAL_KEY_TESTS=1
export ANTHROPIC_API_KEY="$(printenv ANTHROPIC_API_KEY)"

python -m pytest tests/integration/test_real_api.py::TestAnthropicAPI -v

​
How It Works

All tests in tests/integration/test_real_api.py use a module-level skip when RUN_REAL_KEY_TESTS is unset. Accepted values: 1, true, yes (case-insensitive).
VariableRequiredPurpose
RUN_REAL_KEY_TESTSYesEnables live API tests
OPENAI_API_KEYFor OpenAI testsProvider authentication
ANTHROPIC_API_KEYFor Claude testsOptional provider
GOOGLE_API_KEYFor Gemini testsOptional provider

​
Configuration Options

OptionTypeDefaultDescription
RUN_REAL_KEY_TESTSenvunsetMaster gate — tests skipped when absent
llmstr"gpt-4o-mini"Use fast models to minimise cost
outputpreset"silent"Suppress Rich output in tests

​
Best Practices

Use minimal prompts and prefer gpt-4o-mini — real tests consume tokens and cost money.
Read keys from environment variables; store CI secrets in GitHub Secrets.
Trigger real API tests manually in GitHub Actions, not on every push.
Reuse pytest.mark.skipif on RUN_REAL_KEY_TESTS for consistent gating across modules.

​
CI Example

name: Real API Tests
on:
  workflow_dispatch:
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - run: pip install praisonaiagents pytest
      - name: Run real API tests
        env:
          RUN_REAL_KEY_TESTS: "1"
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: python -m pytest tests/integration/test_real_api.py -v

Real API Testing CLI

CLI commands for live API tests

Performance Benchmarks

Measure agent performance