OpenAI Realtime API指南2026：构建可实时打断和倾听的语音Agent

想要一个一秒内响应的语音助手，而不是STT转LLM再转TTS的串行延迟。

OpenAI Realtime API: Speech-to-Speech Without the Pipeline Lag

The OpenAI Realtime API, launched in October 2024, lets you send audio directly to the model and receive audio back - no separate speech-to-text or text-to-speech calls in between. Latency drops to roughly the time it takes the model to think, and the API supports natural turn-taking, including the model hearing you while it speaks and stopping when interrupted.

Why Teams Choose It Over the Classic Pipeline

The classic stack (Whisper to transcribe, an LLM to think, TTS to speak) has two problems: each hop adds 300-800ms, and the pipeline has no sense of conversation timing. The Realtime API keeps one WebSocket connection open and streams audio both ways. For phone agents, kiosks, and in-car assistants, that changes the feel from walkie-talkie to conversation.

What You Get in the API

• Audio in, audio out over a single WebSocket, using the gpt-4o-realtime or gpt-4o-mini-realtime family of models.
• Function calling - the agent can look up order status or book a slot mid-conversation and fold the result into its answer.
• Turn detection built in - the model detects when the user stops speaking and when it is being interrupted.
• Input and output audio samples - you get the exact audio both ways for recording and compliance.

Pricing Reality Check

Audio is billed per minute, and audio tokens are more expensive than text tokens - roughly $0.06 per minute of audio input and $0.24 per minute of audio output on the standard realtime model tier (see OpenAI's pricing page for current rates). A 5-minute support call therefore costs on the order of a dollar and a half before any text-tool calls. That is why many production voice agents only use the Realtime API for the actual conversation and fall back to cheaper text calls for background lookups.

A Minimal Working Example

from openai import OpenAI

client = OpenAI()
with client.beta.realtime.connect(model="gpt-4o-realtime") as conn:
    conn.send({"type": "session.update", "session": {"instructions": "You are a hotel concierge."}})
    # stream mic audio in, receive audio deltas out

The pattern is the same for every app: open a socket, send session config, stream audio in, play audio deltas out.

FAQ

Do I need a separate TTS model? No - the model produces speech directly.

Can I use it for phone calls? Yes, it pairs with telephony providers like Twilio; the audio bridges into the same WebSocket.

Is the Realtime API more expensive than text? Yes, audio tokens cost more than text tokens per minute of use.

Does it support languages other than English? Yes, the underlying models handle dozens of languages natively.