Realtime Voice Agent

Open a real WebSocket to OpenAI's Realtime API for live voice and text

RealtimeAgent opens a real WebSocket to OpenAI’s Realtime API for bidirectional audio and text.

Quick Start

1. Connect and send text

   import { RealtimeAgent } from 'praisonai';
   
   const agent = new RealtimeAgent({
     name: 'VoiceAssistant',
     llm: 'gpt-4o-realtime-preview',
     apiKey: process.env.OPENAI_API_KEY,
   });
   
   agent.onAudio((chunk) => playAudio(chunk));   // Uint8Array of decoded audio
   agent.onMessage((text) => console.log(text)); // text deltas
   
   await agent.connect();          // Real WebSocket — throws on auth/network failure
   await agent.sendText('Hello!'); // Real API frames — throws if not connected

2. Stream microphone audio

   import { RealtimeAgent } from 'praisonai';
   
   const agent = new RealtimeAgent({
     instructions: 'You are a helpful voice assistant.',
     realtime: { voice: 'nova' },
     apiKey: process.env.OPENAI_API_KEY,
   });
   
   agent.onAudio((chunk) => playAudio(chunk));
   
   await agent.connect();
   agent.sendAudio(microphoneBuffer);  // Uint8Array | ArrayBuffer of PCM16
   agent.commitAudio();                // ask the server to process the buffer

How It Works

Both audio and text turns travel over the same live socket.

Events

Register handlers by event type, or use '*' to receive every server event. Convenience callbacks mirror the Python agent.

import { RealtimeAgent } from 'praisonai';

const agent = new RealtimeAgent({ apiKey: process.env.OPENAI_API_KEY });

// Typed event handlers
agent.on('response.audio.delta', (event) => playAudio(event.data));
agent.on('response.text.delta', (event) => process.stdout.write(event.data.delta));

// Receive everything, including unknown server types (passed through verbatim)
agent.on('*', (event) => console.log(event.type));

// Python-parity convenience callbacks
agent.onMessage((text) => console.log(text));   // decoded text deltas
agent.onAudio((chunk) => playAudio(chunk));      // decoded audio bytes
agent.onError((err) => console.error(err));

await agent.connect();

Known event types include session.created, session.updated, response.text.delta, response.audio.delta, response.done, and error. Any other type the server sends is emitted untouched.

Configuration Options

Pass realtime as a config object to tune voice and audio.

Agent-level options: llm (default gpt-4o-realtime-preview), apiKey (falls back to OPENAI_API_KEY), url (endpoint override), headers, webSocket (inject a constructor), and connectTimeoutMs (default 30000).

import { RealtimeAgent } from 'praisonai';

const agent = new RealtimeAgent({
  name: 'VoiceAssistant',
  llm: 'gpt-4o-realtime-preview',
  apiKey: process.env.OPENAI_API_KEY,
  connectTimeoutMs: 10000,
  realtime: {
    voice: 'nova',
    inputFormat: 'pcm16',
    vadEnabled: true,
  },
});

Runtime Requirements

The WebSocket implementation is resolved at call time: an injected { webSocket } constructor first, then globalThis.WebSocket, then the optional ws package.

If no implementation is available, connect() throws with the three ways to fix it (upgrade Node, install ws, or inject a constructor) — it never pretends to connect.

Best Practices

Related