Skip to main content

LLM Module

The LLM module provides the core language model client and utilities for agent interactions, supporting multiple providers through LiteLLM.

Param Cluster Map

Note: base_url and api_key remain separate parameters (connection/auth constraint).

Precedence Ladder

Instance > Config > Array > Dict > String > Bool > Default

Ways to Use llm=

Legacy (Deprecated)

Installation

Quick Start

Classes

LLM

Main LLM client class supporting multiple providers.

Constructor

Methods

LLMContextLengthExceededException

Exception raised when context length is exceeded.

LLMResponseError

Raised by LLM.get_response() when the tool-calling loop fails mid-iteration and cannot produce a response. The original exception is chained via raise … from e, so e.__cause__ holds the underlying error.
Before this behaviour landed, the same failure was silently swallowed and returned as an empty string, so agent.chat() persisted an empty assistant message and retries burned through their budget on a phantom success. Now the failure surfaces as a distinguishable exception.
LLMResponseError and LLMContextLengthExceededException coexist. Context-length errors still raise their dedicated exception; LLMResponseError is strictly the “tool loop failed mid-iteration” case. If you already wrap agent.chat() in try/except Exception, nothing changes — it still catches this. Catch LLMResponseError only if you want to distinguish tool-loop failures.

OpenAIClient

OpenAI-compatible client for direct API access.

ModelRouter

Intelligent model routing based on task complexity.

TaskComplexity

ModelProfile

Profile for a model’s capabilities.

Response Types

ChatCompletion

ChatCompletionMessage

ToolCall

CompletionUsage

Utility Functions

supports_structured_outputs

Check if a model supports structured outputs.

supports_streaming_with_tools

Check if a model supports streaming with tool calls.

process_stream_chunks

Process streaming response chunks.

create_routing_agent

Create an agent with model routing.

Usage Examples

Basic Chat

Streaming Response

With Tools

Model Routing

Different Providers

Environment Variables