OpenAI Realtime API Guide 2026: Build a Voice Agent That Interrupts and Listens Live
You want a voice assistant that responds in under a second without the turn-taking lag of STT-to-LLM-to-TTS pipelines. That is what the Realtime API was built for.
💡 What You Will Learn
You want a voice assistant that responds in under a second without the turn-taking lag of STT-to-LLM-to-TTS pipelines. That is what the Realtime API was built for.
📜 Table of Contents
OpenAI Realtime API: Speech-to-Speech Without the Pipeline Lag
The OpenAI Realtime API, launched in October 2024, lets you send audio directly to the model and receive audio back - no separate speech-to-text or text-to-speech calls in between. Latency drops to roughly the time it takes the model to think, and the API supports natural turn-taking, including the model hearing you while it speaks and stopping when interrupted.
Why Teams Choose It Over the Classic Pipeline
The classic stack (Whisper to transcribe, an LLM to think, TTS to speak) has two problems: each hop adds 300-800ms, and the pipeline has no sense of conversation timing. The Realtime API keeps one WebSocket connection open and streams audio both ways. For phone agents, kiosks, and in-car assistants, that changes the feel from walkie-talkie to conversation.
What You Get in the API
- Audio in, audio out over a single WebSocket, using the gpt-4o-realtime or gpt-4o-mini-realtime family of models.
- Function calling - the agent can look up order status or book a slot mid-conversation and fold the result into its answer.
- Turn detection built in - the model detects when the user stops speaking and when it is being interrupted.
- Input and output audio samples - you get the exact audio both ways for recording and compliance.
Pricing Reality Check
Audio is billed per minute, and audio tokens are more expensive than text tokens - roughly $0.06 per minute of audio input and $0.24 per minute of audio output on the standard realtime model tier (see OpenAI's pricing page for current rates). A 5-minute support call therefore costs on the order of a dollar and a half before any text-tool calls. That is why many production voice agents only use the Realtime API for the actual conversation and fall back to cheaper text calls for background lookups.
A Minimal Working Example
from openai import OpenAI
client = OpenAI()
with client.beta.realtime.connect(model="gpt-4o-realtime") as conn:
conn.send({"type": "session.update", "session": {"instructions": "You are a hotel concierge."}})
# stream mic audio in, receive audio deltas out
The pattern is the same for every app: open a socket, send session config, stream audio in, play audio deltas out.
FAQ
Do I need a separate TTS model? No - the model produces speech directly.
Can I use it for phone calls? Yes, it pairs with telephony providers like Twilio; the audio bridges into the same WebSocket.
Is the Realtime API more expensive than text? Yes, audio tokens cost more than text tokens per minute of use.
Does it support languages other than English? Yes, the underlying models handle dozens of languages natively.
❓ FAQ
Do I need a separate TTS model?
No - the model produces speech directly.
Can I use it for phone calls?
Yes, it pairs with telephony providers like Twilio; the audio bridges into the same WebSocket.
Is the Realtime API more expensive than text?
Yes, audio tokens cost more than text tokens per minute of use.
Does it support languages other than English?
Yes, the underlying models handle dozens of languages natively.
Written by our editorial team; tools listed here are tested or verified against public sources. Links point to official sites or GitHub repos for reference only โ no paid placements.
