Building Voice AI: Talk Back in 2026

There are two ways to build a talking agent today, and picking the right one is the first decision.

The pipeline (cascaded). Speech-to-text (STT) transcribes the user, an LLM reasons over the text, and text-to-speech (TTS) speaks the reply. Three swappable stages. You get full control: any LLM, any voice, text tools, logging at every hop. Verified components: Deepgram or OpenAI Whisper for STT; Cartesia or ElevenLabs for TTS; any chat model in the middle.

Native speech-to-speech (S2S). A single model takes audio in and emits audio out, skipping the text round-trip. The two leading options are OpenAI's Realtime API (the gpt-realtime speech-to-speech model family, generally available since 2025) and Google's Gemini Live API (native-audio models). S2S wins on latency and natural prosody and handles interruptions gracefully, but you trade away some control over the reasoning step and text-tool integration.

Humans notice a conversational gap around a few hundred milliseconds and start to feel awkward past roughly a second. Your target is voice-to-voice latency under about 800ms — the time from the user finishing speaking to hearing the first syllable back. In a pipeline that budget is spent across four stages, so streaming everywhere is non-negotiable.

• Endpointing — detecting the user actually stopped (not just paused).
• STT — stream partial transcripts; don't wait for the final. Deepgram markets its Nova-3 streaming model around sub-300ms latency.
• LLM time-to-first-token — usually the biggest single chunk. Start TTS on the first sentence, not the whole answer.
• TTS time-to-first-audio — Cartesia's Sonic and ElevenLabs' Flash advertise time-to-first-audio in the tens of milliseconds.

The trick that makes pipelines feel instant: stream between every stage. As soon as the LLM emits its first sentence, send it to TTS; as soon as TTS produces its first audio chunk, play it. Never wait for a stage to fully finish. Native S2S models collapse this whole chain, which is where their latency edge comes from.

Real conversation isn't walkie-talkie. People interrupt, back-channel (“mm-hm”), and start speaking before you've finished. An agent that ignores this feels robotic and, worse, talks over the user.

• Voice activity detection (VAD) tells you when someone is speaking. A model like Silero VAD is the common building block in pipelines.
• Endpointing decides when a turn is done — silence-based is simple; semantic endpointing (is the sentence complete?) is better for natural pacing.
• Barge-in is the critical one: when the user starts talking while the agent is speaking, stop the TTS immediately and start listening. Nothing feels worse than an agent that plows on.

Native speech-to-speech models handle interruption in the model itself — the Gemini Live API and OpenAI Realtime both let the user cut in and the model yields. In a pipeline you wire this logic yourself, which is a big reason frameworks (next lesson) exist.

You almost never hand-wire STT, an LLM, and TTS from scratch — you use a framework that orchestrates the pipeline, VAD, barge-in, and transport for you. Two verified open-source options:

• Pipecat — a Python framework with dozens of pre-wired STT/LLM/TTS integrations (Deepgram, Cartesia, ElevenLabs, OpenAI, Gemini). Swapping a vendor is a one-line change.
• LiveKit Agents — a WebRTC-native framework; in 2026 it added native SIP and phone numbers, so telephony no longer needs a separate bridge.

Web: capture the mic in the browser, stream audio over a WebSocket to your server, run the pipeline, stream audio back. Phone: connect a carrier to your pipeline — classically via Twilio Voice Media Streams (bidirectional audio over a WebSocket), or through LiveKit's built-in SIP. The pipeline logic is identical; only the transport changes.

# Pipecat-style pipeline (illustrative)
pipeline = Pipeline([
    transport.input(),        # mic or phone audio in
    stt_service,              # Deepgram streaming STT
    llm_service,              # your chat model + tools
    tts_service,              # Cartesia / ElevenLabs streaming TTS
    transport.output(),       # speaker or phone audio out
])
# The framework handles VAD, barge-in, and streaming between stages.

Voice cost is billed by the minute of audio and the tokens in between. In a pipeline you pay three meters:

• STT — priced per minute of audio transcribed (often a fraction of a cent per minute for streaming).
• LLM — priced per token; a chatty agent with long system prompts is the usual cost sink. Prompt caching and concise replies cut this.
• TTS — priced per character or per minute of audio generated.

Native speech-to-speech APIs bill by audio tokens (input and output), which folds all three into one meter but can cost more per minute than a tuned pipeline. The big levers are the same either way: use a cheaper/faster model where quality allows, cache the stable prompt, keep responses short (users prefer it anyway), and cap turn length. Sketch your own numbers with our API cost estimator before you commit to a stack.

Voice isn't only a convenience — it's an access ramp. A well-built voice interface can be the difference between a product someone can use and one they can't.

• Low-vision and blind users get a hands-and-eyes-free path that pairs naturally with screen readers.
• Motor-impairment users avoid precise tapping and typing entirely.
• Hands-busy and eyes-busy contexts — driving, cooking, on a factory floor — become usable.
• Multilingual reach — modern STT and TTS support dozens of languages, lowering the literacy and language barriers a text UI imposes.

Design for it deliberately: always provide a synchronized text transcript (captions serve deaf and hard-of-hearing users and everyone in a quiet room), offer a text fallback for when voice fails, respect the user's pace, and build gentle error recovery for misheard input. Accessibility and good voice design are the same design.

The demo is easy; the phone call in a noisy car is hard. Budget for these before launch:

• Echo and noise — without echo cancellation the agent hears itself and interrupts its own reply. Use the transport's built-in AEC.
• STT errors — transcription is never perfect; design prompts and confirmations that tolerate a wrong word without derailing.
• Interruption edge cases — a cough shouldn't count as barge-in; tune your VAD thresholds.
• Consent and logging — recording voice has legal weight. Disclose it, and store transcripts responsibly.
• Graceful fallback — when a stage times out, say something human (“one moment”) rather than going silent.