Local Model Resolver

Point an Agent at whatever local model server is running with llm=\"local\" — no URL, no model name, no key

Set llm="local" and the agent finds a running local model server, picks the best model on it, and configures itself — no URL, no model name, no key.

from praisonaiagents import Agent

agent = Agent(instructions="You are helpful", llm="local")
agent.start("Why is the sky blue?")

Discovery fires only when llm="local". Passing gpt-4o, ollama/llama3.2, or anthropic/… runs zero probes.

Quick Start

1. Discover whatever is running

   Start any local server (Ollama, llama.cpp, LM Studio, vLLM), then run four lines.

   from praisonaiagents import Agent
   
   agent = Agent(instructions="You are helpful", llm="local")
   agent.start("Why is the sky blue?")

   No OPENAI_BASE_URL, no ollama/ prefix, no model name.

2. Pin an engine, let it pick the model

   Name the engine and the resolver chooses the best model on it.

   from praisonaiagents import Agent
   
   agent = Agent(instructions="You are helpful", llm="local:ollama")
   agent.start("Explain gravity in one sentence")

   Swap ollama for llama_cpp, lm_studio, vllm, mlx_lm, or transformers_serve.

3. Pin engine and model

   Name both for reproducibility.

   from praisonaiagents import Agent
   
   agent = Agent(instructions="You are helpful", llm="local:ollama/qwen3:0.6b")
   agent.start("Summarize the water cycle")

How It Works

The agent calls the resolver, which probes loopback ports, identifies the engine, selects a model, and returns a target the normal LLM path uses unchanged.

The resolver never infers identity from a port — a server matches only when its probe rules hold. This prevents silently talking to the wrong server.

Base URL selection per engine

The base_url handed to litellm depends on the engine, because Ollama and the OpenAI-compatible engines expect different roots.

The resolver exposes both roots on LocalTarget as base_url (Ollama-shaped) and openai_base_url (OpenAI-shaped). The agent picks the right one per engine.

Precedence ladder

The resolver checks sources in a fixed order. The first three are authoritative: if one names a server that does not answer, resolution raises instead of quietly probing another port.

Which option should I pick?

Pick the shortest spec that gives you the certainty you need.

Supported local engines

Identity is decided by probe rules, never by the port — several engines share :8080 and :8000.

Local embedder auto-selection

When Agent(llm="local", ...) needs embeddings (any of retrieval_config, embedder_config, or memory set) and no embedder is configured, the resolver picks one from the same local server.

Precedence:

1. PRAISONAI_LOCAL_EMBED_MODEL env var (validated against the server’s listing).
2. PREFERRED_EMBED_MODELS order: nomic-embed-text, mxbai-embed-large, bge-m3, snowflake-arctic-embed, all-minilm.
3. Any remaining embedding-capable model.

The server’s own capability list wins when available (Ollama exposes "embedding"); otherwise names containing embed or starting with bge- are treated as embedders.

The picker returns None when the server serves none — the agent then warns and never silently falls back to a remote provider.

Since PR #4870, a base URL with a path prefix is preserved end-to-end: OLLAMA_HOST=http://gateway:8000/ollama keeps the /ollama prefix on every request.

Since PR #4924, the measurements the resolver already takes now drive behaviour: the picked embedder carries its measured embedding_dims to the vector store, and the model’s context window comes from the server rather than the litellm default. Tool schemas are made server-safe per engine — see Local Tool Schemas for the Optional[...] collapse and the tools + structured-output refusal.

Advanced: build the block yourself

The same helpers are exported from praisonaiagents.local:

from praisonaiagents.local import (
    select_embedding_model, local_embedder_config, discover,
)

discovery = discover()   # finds a running local server
model = select_embedding_model(discovery.models, discovery.model_meta)
embedder_block = local_embedder_config(discovery.engine, discovery.base_url, model)

Environment Variables

Every source is optional. The three authoritative sources raise when they name an unreachable server.

How model selection works

When you do not name a model, the resolver ranks the server’s models and picks the best one to chat with.

• Tools-capable models rank first — an agent that can call tools is preferred.
• Embedding-only models rank last — an embedder cannot hold a conversation, so it is never picked by accident.
• Ties break by recency — among equally-capable models, the newest wins.

Worked example. On a machine serving qwen3:0.6b, all-minilm, nomic-embed-text, and mxbai-embed-large, the resolver picks qwen3:0.6b — even when the embedders are newer — because the three embed-only models are ranked last.

Errors and gotchas

The resolver fails loudly and specifically instead of guessing.

Local runtime at http://127.0.0.1:80 did not answer (refused). It was named
explicitly by OLLAMA_HOST='http://127.0.0.1', so no other port was probed.
Note that OLLAMA_HOST with an explicit http:// scheme and no port means port
80, not 11434.

Performance notes

Resolution is opt-in and cheap.

• Discovery fires only for llm="local" — cloud models trigger zero probes.
• Importing praisonaiagents does not import the resolver, so import time is unchanged.
• A full resolve issues a handful of loopback HTTP requests and caches the result for 30 s (PRAISONAI_LOCAL_TTL).
• The whole scan is budgeted (PRAISONAI_LOCAL_TIMEOUT, default 1.5 s; per-probe 0.4 s), so nothing stalls when no server is listening.

Explicit vs implicit local

llm="local" is the explicit request — you ask for a local model by name. The keyless local-first fallback is the implicit form — it only kicks in when no cloud key is set and you named no model at all.

Related