AI Voice Agents 2026: Real-Time Voice AI Architecture

Real-time voice agents — systems that listen, think, and speak within conversational latency — became a production category in 2026. The architecture is a three-stage pipeline (speech-to-text, LLM reasoning, text-to-speech) with two hard constraints: end-to-end latency must stay under 500-800ms for natural conversation, and the agent must handle interruptions without corrupting state. This guide covers pipeline design, latency budgeting, interrupt handling, speaker separation, cost, and deployment — with concrete numbers and code.

The Pipeline and Its Latency Budget

User speaks ──► STT (ASR) ──► LLM reasoning ──► TTS ──► Agent speaks
                150-300ms        200-500ms       100-250ms
                 │                │               │
                 └──── 450-1050ms total (goal: <800ms p50) ────┘

The budget rule: the total must stay under 800ms p50; every stage over 300ms is a problem. This drives model choice harder than any other factor — a reasoning model with 900ms TTFT (DeepSeek R1) is simply unusable in interactive voice; the latency benchmarks page shows why mini-class models dominate voice pipelines.

Architecture: The Two Loops

Production voice agents run two concurrent loops — a streaming recognition loop and a generation loop — coordinated by a small state machine:

class VoiceAgent:
    def __init__(self):
        self.conversation = []       # transcript history
        self.state = "listening"     # listening | thinking | speaking
        self.barge_in = False

    async def audio_chunk(self, pcm):       # called ~20x/sec
        if self.state == "speaking" and is_speech(pcm):
            self.barge_in = True            # user interrupted us
        text = await self.stt.stream(pcm)   # partial transcripts
        if text.endswith((".", "?", "!")) or self.pause_detected():
            await self.finalize_utterance(text)

    async def finalize_utterance(self, text):
        self.conversation.append({"role": "user", "content": text})
        self.state = "thinking"
        # Pre-emptively cancel TTS if we were speaking
        self.tts.cancel() if self.barge_in
        response = await self.llm.chat(self.conversation)   # streamed
        self.state = "speaking"
        await self.tts.speak(response)      # streaming synthesis

Two design decisions matter here: barge-in (the agent must stop speaking the moment the user starts) and state integrity (an interrupted turn must not corrupt the transcript — the partial user utterance joins the conversation cleanly).

Interrupt Handling: The Hard Part

Humans interrupt constantly. The agent must:

1. Detect — voice activity detection (VAD) with a fast trigger (<150ms) distinguishes speech from noise
2. Stop — cancel TTS synthesis and playback mid-stream (server-side cancel + client-side audio buffer flush)
3. Reconcile — merge the interrupted agent sentence and the new user utterance into the transcript without duplication or loss

# Reconciliation: what the LLM should see after a barge-in
def reconcile(agent_text, user_text):
    # If user started before agent finished, both go in order:
    return [
        {"role": "assistant", "content": agent_text.strip() + " […]"},  # truncated
        {"role": "user", "content": user_text},
    ]
# Never feed the LLM a half-sentence as if it were complete.

Failing at reconciliation is how voice agents "lose the plot" — the model gets confused transcripts and starts hallucinating context. Test this path explicitly with a scripted interruption suite.

Speaker Separation and Multi-Party

Call-center and meeting agents need diarization — knowing who said what:

• Two-party (agent + caller) — separation is trivial since the agent's own audio is known; only the caller needs STT
• Multi-party (meetings) — use diarization-aware ASR (speaker labels in the transcript) or a separate diarization pass
• Realtime multi-speaker — beamforming on the device or channel separation upstream; model the speakers as separate transcript threads

For most voice-agent products (support, sales, reception), two-party separation suffices — keep the scope tight.

Model Selection for Voice

Escalation pattern: route simple turns to mini-class models, and escalate to a frontier model only when the turn needs it (detected by intent or confidence). This keeps p50 latency low and cost 60-80% below all-frontier pipelines — the same routing logic as text APIs, applied per turn.

Cost Per Conversation Minute

# Rough per-minute costs (STT + LLM + TTS), typical voice agent:
#   STT:  ~60s audio  →  $0.0006 (streaming ASR pricing)
#   LLM:  ~4 turns × 400 tokens → $0.0003 (mini-class, cached prefix)
#   TTS:  ~30s speech → $0.0012
#   Total: ~$0.002/min → $0.12/hour of conversation
#   At 10K call-minutes/month: ~$120 — dominated by TTS, not LLM

Surprise cost driver: TTS dominates because audio tokens are expensive and you generate ~3x more speech than you think (greetings, confirmations, filler). Cache greeting/confirmation audio segments client-side — a 10-second greeting played 1,000 times/day should be synthesized once.

Latency Optimization Checklist

• Stream everything: STT partial results, LLM tokens, TTS audio chunks — never batch at any stage
• Run STT and LLM in parallel where possible (predict the turn while ASR finalizes)
• Keep the LLM system prompt under 300 tokens for voice (voice turns are short; don't burn TTFT on a 2K-token preamble)
• Use prompt caching for the stable prefix — 30-60% input savings on multi-turn conversations
• Warm connections: keep WebSocket/HTTP connections alive between turns
• Pre-synthesize common responses (acknowledgments, hold messages)
• Instrument per-stage latency from day one — a voice agent without stage-level metrics is undebuggable (see agent observability)

Deployment Patterns

• WebRTC + gateway — browser/mobile clients stream audio over WebRTC to a media server; the agent pipeline runs server-side. Best for productized voice agents.
• SIP/VoIP bridge — connect to phone networks via a SIP gateway; add echo cancellation and DTMF handling. Best for call-center replacement.
• Edge + cloud hybrid — VAD and wake-word on-device; STT/LLM/TTS in cloud. Saves bandwidth and improves barge-in responsiveness.

All three patterns share the same core pipeline; the difference is transport. Keep the agent logic transport-agnostic (plain async functions over a message interface) so you can add channels without rewriting.

Evaluation: How Good Is Your Voice Agent?

Voice adds two evaluation dimensions beyond text quality:

• Task success — did the call achieve its goal (booking, resolution, info captured)? Score on recorded calls with a rubric.
• Conversational fluency — latency perception (how often users talk over the agent), interruption recovery (did the agent stay coherent after barge-ins), and naturalness ratings.

Automate with: scripted call replay (deterministic test calls through the full pipeline), latency percentile dashboards, and periodic human evaluation of recorded samples. The evaluation guide's LLM-judge patterns extend to voice transcripts.

Bottom Line

Real-time voice agents are an architecture problem more than a model problem: a tight latency budget, disciplined interrupt handling, and stage-level metrics determine success more than which LLM you pick. Start with mini-class models, stream every stage, pre-synthesize the boring parts, and measure per-stage latency from day one. For the component APIs, the voice assistant API guide covers STT/TTS selection, and DrAI's gateway (free tier at ai.dr-ai.top/signin, pricing) provides the OpenAI-compatible LLM stage with routing and caching built in.