Realtime Agent

Real-time voice conversations with WebSocket-based audio streaming

Build voice assistants with the RealtimeAgent — bidirectional audio streaming over WebSockets for live, interactive conversations.

python
from praisonaiagents import RealtimeAgent

agent = RealtimeAgent(name="VoiceAssistant")

await agent.aconnect()
await agent.asend_text("Hello, how can I help you?")
Diagram

Realtime Agent

📋 Voice/Text

🤖 RealtimeAgent

🧠 WebSocket Stream

✅ Audio Response

The RealtimeAgent enables real-time voice conversations using WebSocket connections for bidirectional audio streaming. It’s designed for interactive voice applications, live transcription, and real-time AI assistants.

Quick Start

  1. Simple Usage

    Connect and start a conversation.

    python
    from praisonaiagents import RealtimeAgent
    
    agent = RealtimeAgent(
        name="Assistant",
        instructions="You are a helpful voice assistant",
    )
    
    agent.connect()
    agent.send_text("Hello!")
    
    def on_message(message):
        print(f"Received: {message}")
    
    agent.on_message(on_message)
  2. With Configuration

    Choose a voice and tune turn detection.

    python
    from praisonaiagents import RealtimeAgent, RealtimeConfig
    
    config = RealtimeConfig(
        voice="nova",
        turn_detection="server_vad",
    )
    
    agent = RealtimeAgent(
        name="VoiceBot",
        model="gpt-realtime",  # optional; this is the default
        realtime=config,
        instructions="Keep responses brief.",
    )
    
    agent.connect()
    agent.send_text("Tell me a fun fact.")

How It Works

Diagram
WebSocketRealtimeAgentUserWebSocketRealtimeAgentUserconnect() + send_text/audioOpen bidirectional streamStreamed audio + texton_message / on_audio callbacks

Installation

bash
pip install praisonaiagents[realtime]

Basic Usage

Simple Voice Assistant

python
from praisonaiagents import RealtimeAgent

# Create agent with default settings
agent = RealtimeAgent(
    name="Assistant",
    instructions="You are a helpful voice assistant"
)

# Connect to realtime API
agent.connect()

# Send text message
agent.send_text("Hello!")

# Handle incoming messages
def on_message(message):
    print(f"Received: {message}")

agent.on_message(on_message)

With Custom Configuration

python
from praisonaiagents import RealtimeAgent, RealtimeConfig

config = RealtimeConfig(
    voice="alloy",
    modalities=["text", "audio"],
    turn_detection="server_vad"
)

agent = RealtimeAgent(
    name="CustomVoice",
    model="gpt-realtime",
    realtime=config,
    verbose=True
)

Configuration

RealtimeConfig Options

ParameterTypeDefaultDescription
voicestr"alloy"Voice (alloy, echo, fable, onyx, nova, shimmer)
modalitieslist["text", "audio"]Output modality — GA accepts a single output modality; audio includes a transcript
turn_detectionstr"server_vad"Turn detection mode (server_vad, none)
input_audio_formatstr"pcm16"Input audio format (pcm16, g711_ulaw, g711_alaw)
output_audio_formatstr"pcm16"Output audio format (pcm16, g711_ulaw, g711_alaw)
~~temperature~~~~float~~~~0.8~~Deprecated — ignored on the GA API. Setting it emits a DeprecationWarning. The field is kept for back-compat; the value is never sent.
max_response_output_tokensintNoneMapped to max_output_tokens in the GA session
instructionsstrNoneSystem instructions for the session
python
from praisonaiagents import RealtimeConfig

config = RealtimeConfig(
    voice="nova",
    turn_detection="server_vad"
)

Methods

Connection Methods

python
# Synchronous
agent.connect()
agent.disconnect()

# Asynchronous
await agent.aconnect()
await agent.adisconnect()

Sending Data

python
# Send text message
agent.send_text("Hello, how are you?")

# Send audio data (bytes)
agent.send_audio(audio_bytes)

Receiving Data

python
# Register message handler
def handle_message(message):
    print(f"Text: {message}")

agent.on_message(handle_message)

# Register audio handler
def handle_audio(audio_data):
    # Process audio bytes
    pass

agent.on_audio(handle_audio)

Async Usage

python
import asyncio
from praisonaiagents import RealtimeAgent

async def main():
    agent = RealtimeAgent(name="AsyncVoice")

    try:
        await agent.aconnect()

        # Send message
        await agent.asend_text("Hello!")

        # Listen for responses
        async for message in agent.listen():
            print(f"Response: {message}")

    finally:
        await agent.adisconnect()

asyncio.run(main())

Voice Options

VoiceDescription
alloyNeutral, balanced voice
echoWarm, conversational
fableExpressive, storytelling
onyxDeep, authoritative
novaFriendly, upbeat
shimmerClear, professional

Example: Interactive Voice Bot

python
import asyncio
from praisonaiagents import RealtimeAgent, RealtimeConfig

async def voice_bot():
    config = RealtimeConfig(
        voice="nova",
        turn_detection="server_vad"
    )

    agent = RealtimeAgent(
        name="VoiceBot",
        model="gpt-realtime",
        realtime=config,
        instructions="You are a friendly assistant. Keep responses brief."
    )

    await agent.aconnect()
    print("Connected! Start speaking...")

    # Handle incoming audio
    def on_audio(audio):
        # Play audio through speakers
        play_audio(audio)

    agent.on_audio(on_audio)

    # Keep running
    try:
        while True:
            await asyncio.sleep(1)
    except KeyboardInterrupt:
        await agent.adisconnect()

asyncio.run(voice_bot())

Error Handling

python
from praisonaiagents import RealtimeAgent

agent = RealtimeAgent(name="SafeVoice")

try:
    agent.connect()
    agent.send_text("Hello")
except ConnectionError as e:
    print(f"Connection failed: {e}")
except TimeoutError as e:
    print(f"Request timed out: {e}")
finally:
    agent.disconnect()

Best Practices

Enable turn detection for natural flow

Keep turn_detection="server_vad" so the agent knows when the user has stopped speaking. Without server-side VAD, conversations feel stilted and responses arrive at the wrong moments.

Handle disconnects gracefully

WebSocket connections drop. Wrap connect() in reconnection logic and catch ConnectionError so a dropped link doesn’t end the session.

Buffer audio for smooth playback

Collect audio chunks from on_audio before playing them. Playing raw chunks as they arrive produces choppy, garbled speech.

Keep system prompts short

Long instructions slow the first response. A brief prompt keeps latency low, which matters far more in voice than in text.

Migration from beta Realtime

Existing code keeps working; check these three cases.

I pinned llm="gpt-4o-realtime-preview" or model="gpt-4o-realtime-preview"

Switch to model="gpt-realtime", or drop the argument to use the default. The preview model is still accepted when you pass it explicitly.

I set RealtimeConfig(temperature=…)

Remove it. The GA API has no session temperature, so the value is ignored and emits a DeprecationWarning.

I use websockets < 14

Still supported. The agent picks the right header argument (extra_headers on 12/13, additional_headers on 14+). The declared range is websockets>=12.0.


Related