Pytest Failure Categories

Guide to distinguish real regressions from stale mocks in PraisonAI test failures

Pytest failures in PraisonAI fall into predictable categories that help distinguish real regressions from outdated test mocks.

Quick Start

1. Identify the Category

   Run the one-liner check to identify which category your test failure belongs to.

2. Determine Real vs Stale

   Use the verification commands to distinguish between real regressions and outdated mocks.

3. Apply the Fix

   Update mock targets or escalate to SDK maintainers based on the analysis.

Failure Categories

Registry get_instance

Symptom: AttributeError where registry mock returns None instead of registry instance

One-liner check:

python -c "from praisonai.llm.registry import LLMProviderRegistry; print(LLMProviderRegistry.get_instance())"

Likely root cause: Stale mock — registry API changed in SDK, mock needs updating

Common failure signature:

AttributeError: 'NoneType' object has no attribute 'resolve'

Fix approach:

• Update mock patch target to match current registry location
• Verify mock returns actual registry instance, not None
• Check for singleton pattern changes in LLMProviderRegistry.get_instance()

execute_code Export

Symptom: ImportError on from praisonaiagents.tools import execute_code

One-liner check:

python -c "from praisonaiagents.tools import execute_code; print('✓ Import successful')"

Likely root cause: Export removed/renamed in SDK

Common failure signature:

ImportError: cannot import name 'execute_code' from 'praisonaiagents.tools'

Fix approach:

• Check TOOL_MAPPINGS in praisonaiagents/tools/__init__.py
• Verify execute_code is mapped to .python_tools module
• Update import paths if module structure changed

Bot Lazy Loaders

Symptom: First call returns None instead of bot instance due to lazy loading bypass

One-liner check:

pytest tests/unit/test_bot_loaders.py -x

Likely root cause: Mock targets the eager path; lazy loader bypasses it

Common failure signature:

AssertionError: Expected bot instance, got None

Fix approach:

• Mock the lazy loading mechanism itself
• Ensure mocks cover both eager and lazy initialization paths
• Check for __getattr__ method changes in tools modules

aiui Patch Path

Symptom: Mock patch silently no-ops, test passes when it should fail

One-liner check:

python -c "import praisonai.ui; print(praisonai.ui.__file__)"

Likely root cause: Patch target moved; update mock.patch path

Common failure signature:

# Test passes but shouldn't - mock not being applied

Fix approach:

• Verify praisonaiui module location has not changed
• Update patch paths from praisonai.aiui to praisonaiui if needed
• Check for import alias changes (import praisonaiui as aiui)

conftest Key Behavior

Symptom: Tests pass locally but fail in CI, or vice versa

One-liner check:

env | grep -E '^(OPENAI|ANTHROPIC|GOOGLE|XAI|GROQ|COHERE)_API_KEY='

Likely root cause: Real key exported in one environment but not the other

Context: Since May 2026 (PR #1663), setup_test_environment only injects placeholder API keys when variables are unset or empty. Real keys from shell environment are preserved.

Fix approach:

• unset API keys before running unit tests to force placeholder injection
• Use clean shell environment for consistent test behavior
• Mark tests as real/network/e2e if they need real API keys

Fully-Mocked Test Invisible to CI

Symptom: A fully-mocked file mentions a provider name (e.g. openai/mistral, ollama/llama2), so the gating plugin infers network and the CI marker expression deselects it.

One-liner check:

pytest path/to/test_file.py -m "not network and not provider_ollama" --collect-only -q

Likely root cause: Whole-file network inference on a mocked file that references a provider keyword

Common failure signature:

N deselected / 0 selected

Fix approach:

• Add pytestmark = pytest.mark.offline at module scope to opt out of provider and network auto-marking
• Confirm the file is truly mocked (patches litellm.completion or similar) before marking it offline

Troubleshooting Workflow

Related