Local Models

Run PraisonAI against Ollama or any OpenAI-compatible local server — keyless

Point PraisonAI at Ollama, LM Studio, llama.cpp, or vLLM — no cloud key required.

from praisonaiagents import Agent

agent = Agent(
    name="Local Assistant",
    instructions="Answer briefly",
    llm="ollama/llama3.2",
)
agent.start("Why is the sky blue?")

Quick Start

1. Zero-config with Ollama

   Start Ollama, pull a model, and run — PraisonAI detects the endpoint when no cloud key is set.

   ollama serve
   ollama pull llama3.2
   praisonai run "hello"
   # > No cloud key found; using local model ollama/llama3.2.

2. Point at any OpenAI-compatible server

   Set OPENAI_BASE_URL to target LM Studio, llama.cpp, or vLLM.

   export OPENAI_BASE_URL=http://localhost:1234/v1
   praisonai run "hello"

How It Works

PraisonAI probes for a reachable local server only when no cloud provider key is set, then adopts the first model it finds.

sequenceDiagram
    participant CLI
    participant Detect as detect_local_model
    participant Server as Local Server

    CLI->>Detect: no cloud key — probe?
    Detect->>Server: GET /api/tags
    Server-->>Detect: models[0].name → ollama/<name>
    Detect->>Server: GET /v1/models (or /models if base ends in /v1)
    Server-->>Detect: any 2xx → present; data[0].id → openai/<id>
    Detect-->>CLI: LocalModel(model, base_url)

    classDef cli fill:#8B0000,stroke:#7C90A0,color:#fff
    classDef process fill:#189AB4,stroke:#7C90A0,color:#fff
    classDef output fill:#10B981,stroke:#7C90A0,color:#fff

    class CLI cli
    class Detect process
    class Server output

Since PR #4870, the /v1/models probe always runs and any 2xx counts as present. Even if GET / returns 404 (as llama-server, LM Studio and vLLM do), the /v1/models probe still runs and finds the server — earlier versions reported these servers as “did not answer (refused)” while they were actively serving.

Detection is timeout-bounded (1 s per probe, so a slow first Ollama response is still caught) and caches negatives for 30 s. A closed port is refused instantly, so the first-run hot path stays fast when nothing is listening.

Helper model for internal calls

PraisonAI makes a small number of internal LLM calls for things like session-title generation, context compaction, memory quality scoring, and workflow routing. By default they use gpt-4o-mini. If your local server doesn’t serve a model called gpt-4o-mini (Ollama, LM Studio, and vLLM don’t) those calls will fail with a “model not found” error, even though the agent itself works fine.

Set PRAISONAI_AUXILIARY_MODEL to a small local model:

export OPENAI_BASE_URL=http://localhost:11434/v1
export PRAISONAI_AUXILIARY_MODEL=ollama/qwen3:0.6b
praisonai run "hello"

The complete four-line copy-paste example:

import os
from praisonaiagents import Agent

os.environ["OPENAI_BASE_URL"] = "http://localhost:11434/v1"
os.environ["OPENAI_MODEL_NAME"] = "ollama/llama3.3:70b"
os.environ["PRAISONAI_AUXILIARY_MODEL"] = "ollama/qwen3:0.6b"

agent = Agent(name="Local", instructions="Answer briefly")
agent.start("Explain black holes to a 5-year-old")

Which knob to set:

Configuration Options

Precedence (first match wins): --model <name> → any cloud provider key → reachable local endpoint (OPENAI_BASE_URL → OLLAMA_HOST → 127.0.0.1:11434) → gpt-4o-mini fallback.

Common Patterns

ollama serve
ollama pull llama3.2
praisonai run "hello"

export OPENAI_BASE_URL=http://localhost:1234/v1
praisonai run "hello"

export OPENAI_BASE_URL=http://localhost:8080/v1
praisonai run "hello"

export OPENAI_BASE_URL=http://localhost:8000/v1
praisonai run "hello"

Pin a specific model explicitly with --model:

praisonai run "hello" --model ollama/llama3.2

Mixing local and hosted models

Setting a local base_url for local development is safe to combine with hosted model names — a base_url says where the server is, not what it serves.

from praisonaiagents import Agent

# Hosted model, behind a local base_url — safe; keeps DefaultAdapter.
gpt = Agent(
    name="hosted",
    instructions="Answer briefly",
    llm="gpt-4o",
    base_url="http://localhost:11434/v1",
    api_key="sk-...",   # real OpenAI key
)

# Same base_url, open-weights local model — gets OllamaAdapter with tool-call repair.
local = Agent(
    name="local",
    instructions="Answer briefly",
    llm="llama3.2",
    base_url="http://localhost:11434/v1",
)

Detection runs in a fixed order: an explicit ollama/ prefix wins, then closed-weights names keep their native adapter, then the base_url is checked, then the model name, then the default.

The ollama/ prefix is the user speaking explicitly — it wins over everything, even a closed-weights name:

from praisonaiagents import Agent

# Even a closed-weights-looking name is Ollama when the prefix says so.
agent = Agent(llm="ollama/gpt-oss")

Nested and vendor-qualified routes resolve to the family they name, so they keep their native adapter behind a local base_url too:

from praisonaiagents import Agent

agent = Agent(
    llm="bedrock/anthropic.claude-3-5-sonnet",
    base_url="http://localhost:11434/v1",   # safe: still routes to AnthropicAdapter
)

Best Practices

Related