Praison AI

Real API Testing CLI

CLI commands for running integration tests with real API keys

Real API Testing CLI

Commands for running integration tests with real API keys.

Commands

Run All Real API Tests

bash
export OPENAI_API_KEY="your-key"
export RUN_REAL_KEY_TESTS=1
python -m pytest tests/integration/test_real_api.py -v

Output:

tests/integration/test_real_api.py::TestAgentRealAPI::test_agent_simple_chat PASSED
tests/integration/test_real_api.py::TestAgentRealAPI::test_agent_with_tool PASSED
tests/integration/test_real_api.py::TestAgentRealAPI::test_agent_chat_history PASSED
tests/integration/test_real_api.py::TestLiteAgentRealAPI::test_lite_agent_with_openai PASSED
tests/integration/test_real_api.py::TestLiteAgentRealAPI::test_lite_agent_no_litellm_loaded PASSED

Run Specific Provider Tests

OpenAI Tests

bash
export OPENAI_API_KEY="your-key"
export RUN_REAL_KEY_TESTS=1
python -m pytest tests/integration/test_real_api.py::TestAgentRealAPI -v

Anthropic Tests

bash
export ANTHROPIC_API_KEY="your-key"
export RUN_REAL_KEY_TESTS=1
python -m pytest tests/integration/test_real_api.py::TestAnthropicAPI -v

Google Gemini Tests

bash
export GOOGLE_API_KEY="your-key"
export RUN_REAL_KEY_TESTS=1
python -m pytest tests/integration/test_real_api.py::TestGoogleAPI -v

Run LiteAgent Tests

bash
export OPENAI_API_KEY="your-key"
export RUN_REAL_KEY_TESTS=1
python -m pytest tests/integration/test_real_api.py::TestLiteAgentRealAPI -v

Quick Verification

Verify Agent Works

bash
export OPENAI_API_KEY="your-key"
python -c "
from praisonaiagents import Agent

agent = Agent(
    name='Test',
    instructions='Reply with one word.',
    llm='gpt-4o-mini',
    output="silent"
)

response = agent.chat('Say hello')
print(f'Response: {response}')
print('Agent test: OK')
"

Verify LiteAgent Works

bash
export OPENAI_API_KEY="your-key"
python -c "
from praisonaiagents.lite import LiteAgent, create_openai_llm_fn

llm_fn = create_openai_llm_fn(model='gpt-4o-mini')
agent = LiteAgent(name='Test', llm_fn=llm_fn)

response = agent.chat('Say hi')
print(f'Response: {response}')
print('LiteAgent test: OK')
"

Verify Tool Execution

bash
export OPENAI_API_KEY="your-key"
python -c "
from praisonaiagents import Agent

def add(a: int, b: int) -> int:
    '''Add two numbers.'''
    return a + b

agent = Agent(
    name='Math',
    instructions='Use tools.',
    llm='gpt-4o-mini',
    tools=[add],
    output="silent"
)

response = agent.chat('What is 5+3?')
assert '8' in response, f'Expected 8 in response: {response}'
print('Tool test: OK')
"

Environment Variables

VariableDescriptionRequired
RUN_REAL_KEY_TESTSEnable real API testsYes
OPENAI_API_KEYOpenAI API keyFor OpenAI tests
ANTHROPIC_API_KEYAnthropic API keyFor Anthropic tests
GOOGLE_API_KEYGoogle API keyFor Gemini tests

CI/CD Integration

GitHub Actions

yaml
name: Real API Tests

on:
  workflow_dispatch:

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      - run: pip install -e ".[test]"
      - name: Run Tests
        env:
          RUN_REAL_KEY_TESTS: "1"
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: python -m pytest tests/integration/test_real_api.py -v

Shell Script

bash
#!/bin/bash
set -e

export RUN_REAL_KEY_TESTS=1

if [ -z "$OPENAI_API_KEY" ]; then
    echo "Error: OPENAI_API_KEY not set"
    exit 1
fi

python -m pytest tests/integration/test_real_api.py -v
echo "All real API tests passed!"

Troubleshooting

Tests Skipped

If tests are skipped, ensure environment variable is set:

bash
# Check if variable is set
echo $RUN_REAL_KEY_TESTS

# Set it
export RUN_REAL_KEY_TESTS=1

API Key Errors

bash
# Verify key is set
python -c "import os; print('Key set:', bool(os.environ.get('OPENAI_API_KEY')))"

Rate Limits

If hitting rate limits, add delays between tests:

bash
python -m pytest tests/integration/test_real_api.py -v --tb=short -x

Related