What is an AI voice agent, and how is it different from a chatbot with a voice?
An AI voice agent is a real-time software system that listens to a phone or web call, converts speech to text, reasons over that text with an LLM, and speaks a response back — all inside a sub-second latency budget, with the ability to call external tools like a calendar or CRM mid-conversation. Telnyx's latency benchmarks put a typical stitched pipeline — speech-to-text, LLM inference, text-to-speech, plus network round trips between vendors — at 600ms to 1.7 seconds per turn, and that range is the whole engineering problem in one sentence.
The dividing line against a voice bot is behavioral, not acoustic: a voice bot follows a decision tree and reads scripted lines; a voice agent reasons over open-ended input, can be interrupted mid-sentence, and can take an action. This post covers what's under the hood — pipeline architecture, the latency math, telephony, interruption handling, tool-calling, and what to monitor after launch. If you're trying to decide whether a voice agent should staff your front desk, read ai-receptionist instead — that post covers the use case. If you're comparing packaged answering-service vendors, read ai-phone-answering-service — that's the buying guide. This one is the build-and-operate layer underneath both.
How does the speech-to-text → LLM → text-to-speech loop actually work?
A voice agent moves audio through three stages in sequence on every turn, or through one unified model that does all three at once. In the cascaded (three-model) architecture — still the default for most production builds because it lets you swap any single component — the caller's audio streams into an ASR (automatic speech recognition) engine that emits partial and final transcripts, the transcript feeds an LLM that generates a text response (often with a function call embedded), and that text streams into a TTS engine that synthesizes audio back to the caller, chunk by chunk, so playback starts before the full sentence has finished generating.
The alternative is a speech-to-speech voice agent that processes audio input to audio output in a single pass, skipping the intermediate text representation entirely. It cuts the handoff latency between stages, but it locks you into one vendor's voice, reasoning quality, and tool-calling implementation — you lose the ability to mix a best-in-class ASR from one vendor with a cheaper LLM and a specific brand voice from TTS. Most builds still choose cascaded for that modularity, and eat the latency cost by streaming aggressively at each stage rather than waiting for complete outputs.
Why does sub-second latency matter so much for a voice agent?
A voice agent lives or dies on a single number the caller can feel: dead air. Telnyx frames the perception thresholds bluntly — humans hand off conversational turns in roughly 200ms, anything above 800ms "feels noticeably delayed," and above 1,500ms callers report that the conversation "feels broken." For context, the ITU-T G.114 telephony standard allows no more than 150ms of one-way transmission delay for good interactive quality, and that budget is consumed before your AI stack does any work at all. Deepgram's latency guide makes the same point from the STT side: streaming architecture, not batch processing, is what makes live conversation viable.
The budget breaks down roughly like this — the left column is Telnyx's "typical stitched stack," the right is what Prodinit reports as achievable with every layer tuned and streaming:
| Stage | Typical stitched stack | Achievable when tuned |
|---|---|---|
| Voice activity detection | — | 10–30ms |
| Speech-to-text (streaming) | 100–300ms | 60–120ms |
| LLM inference / first token | 350–1,000ms | 100–250ms |
| Text-to-speech (first chunk) | 90–200ms | 40–100ms |
| Network / transport round trips | 50–200ms | 20–60ms |
| Total per turn | 600ms–1.7s | 230–560ms |
The LLM is almost always the biggest single line item, which is why aggressive builds lean on smaller, faster models for the conversational turn and reserve larger models for offline summarization or QA, not the live loop. STT is rarely the bottleneck: Deepgram's own documentation puts typical client-side total transcript latency at 200–500ms with transcription processing itself targeted at 300ms or less. The LLM and the network hops between vendors are where the budget goes.
One caveat on published benchmarks worth holding onto: Telnyx also reports full-turn medians for competing platforms (Retell 680ms, Vapi 720ms, Bland 850ms) alongside a 450ms figure for its own co-located architecture. Telnyx sells in this category, so treat vendor-run comparisons as directional evidence that co-location and streaming matter — not as a neutral leaderboard. Prodinit, for its part, targets "sub-250ms at p50" and "p95 below 800ms" as production goals, and notes that batch transcription alone adds 600–1,200ms before your LLM call even fires.
What does telephony actually look like under a voice agent?
A phone-facing voice agent doesn't talk to callers directly — it sits behind a telephony layer that terminates the call and hands audio to the agent as a media stream. For traditional PSTN calls, that means a SIP trunk (Twilio, Telnyx, or a carrier-direct SIP provider) that converts the phone call into a WebSocket or RTP audio stream your pipeline can consume in real time. For web-based voice (in-app or browser calling), WebRTC does the same job with lower latency and no telephony carrier in the loop at all.
This layer matters more than it looks. SIP introduces its own jitter and packet loss characteristics that differ from WebRTC, codec choice (G.711 vs. Opus) affects both audio quality and bandwidth, and regional colocation of your telephony provider, STT, LLM, and TTS services all matters — a call routed through three different continents before it reaches your LLM will blow the latency budget no matter how fast each individual service is. Persistent WebSocket connections rather than per-turn HTTP requests are the other structural requirement; reconnecting a socket every turn adds handshake latency you can't get back.
How do voice agents handle interruptions without sounding robotic?
Voice agents run continuous voice-activity detection (VAD) on the caller's incoming audio stream throughout their own response, and the moment it detects new speech, the agent's TTS output stops immediately — not at the end of the current sentence, mid-word if necessary — and the partial caller utterance gets fed back into the LLM as new context for the next turn. This is called barge-in, and getting it wrong is the fastest way to make an otherwise well-engineered voice agent feel broken: callers interrupt constantly in real conversation, and an agent that talks over them or takes a full second to notice reads as deaf, not slow.
The harder edge cases are backchannel noises ("mm-hmm," "yeah," "okay") that shouldn't trigger a full interruption, and distinguishing a mid-sentence pause from a completed thought — turn-detection models purpose-built for this (rather than simple silence-timeout VAD) are what separates production-grade voice agents from early demos that cut people off constantly or wait too long to respond.
How does tool-calling let a voice agent actually do something, not just talk?
A voice agent becomes operational, not just conversational, the moment tool-calling enters the loop: mid-call, the LLM can invoke a defined function — check calendar availability, create or update a CRM record, look up an order or account status, transfer to a human — and the result gets woven back into the spoken response without the caller perceiving a separate step. Architecturally this means the agent's output isn't just text to speak; it's a structured decision between "speak this" and "call this function, then speak the result," and the agent has to keep the conversation alive (often with a filler phrase or continued small talk) while a slower external API call completes in the background. Set against the thresholds above, the arithmetic is unforgiving: a two-second CRM write inside a budget where 800ms already reads as delayed will create exactly the dead air the whole architecture was built to avoid.
This is also where outbound agents diverge from inbound ones. An outbound voice agent dials from a list, handles voicemail detection and compliance-timing logic, and on a live pickup runs the same STT-LLM-TTS loop with tool-calling into the same calendar or CRM — the difference is initiation and pre-call logic, not the conversational engine itself.
What should you monitor after a voice agent goes live?
A voice agent's launch is the start of the operating problem, not the end of the build. The metrics that matter in production are different from the ones that matter in a demo:
- Latency percentiles, not averages — p95 and p99 turn latency catch the calls that actually frustrate people; an average that looks fine can hide a tail of three-second turns.
- Barge-in accuracy — false interruptions (agent stops talking on background noise) and missed interruptions (agent talks over the caller) both need to be tracked separately.
- Tool-call success and latency — every CRM write or calendar check is a dependency that can fail or slow down independent of your voice pipeline.
- Transcript accuracy on names and numbers — ASR error rates spike on proper nouns, phone numbers, and addresses, which is disproportionately costly on calls where getting a callback number wrong kills the lead.
- Escalation and drop-off rate — how often the agent hands off to a human or the caller hangs up mid-conversation, which is the closest thing to a satisfaction signal you get without a survey.
None of this is optional instrumentation — a voice agent with no monitoring is a voice agent you're flying blind on the moment call volume passes what one person can spot-check by ear.
Build vs. buy vs. platform: what does each actually cost you?
A voice agent's build-vs-buy decision isn't really about upfront cost — it's about who owns the latency tuning, the prompt, and the failure modes when something breaks at 2am.
| Approach | What you control | What it costs you | Best fit |
|---|---|---|---|
| Full build (raw STT/LLM/TTS APIs, own orchestration) | Everything — model choice, prompt, latency budget, data, voice | Engineering time to build and keep holding the latency budget as volume scales; you own every outage | Teams with real call volume and engineering capacity who need the agent tied into proprietary systems |
| Platform (Vapi, Bland, Retell-style builders) | Prompt, voice, some model swapping | Per-minute pricing, vendor lock-in on infrastructure, limited control over the underlying orchestration | Fastest path to a live agent; right for most small businesses testing the channel |
| Managed / BPO-style service | Almost nothing technical — you approve scripts | Recurring service fee, least differentiation, slowest to change | Businesses that want voice AI handled entirely off their plate |
The pattern to notice: control and time investment move together, in opposite directions from speed-to-launch. Most businesses should start on a platform, prove the use case works and the ROI is real, and only move to a full build once call volume and internal engineering capacity justify owning the infrastructure. Building day one, before you know your actual call patterns and failure modes, usually means rebuilding twice.
On the market question, one number gets quoted constantly and is worth stating precisely rather than loosely. In an August 2022 press release, Gartner predicted that "by 2026, conversational artificial intelligence (AI) deployments within contact centers will reduce agent labor costs by $80 billion," and that one in 10 agent interactions would be automated by 2026, up from an estimated 1.6% at the time. That is a forecast published four years ago, not a measured outcome — useful as a signal that the category was expected to industrialize, not as proof of what your own agent will save.
Where this fits into the rest of your GTM stack
A voice agent doesn't operate alone — it's one channel feeding the same calendar, CRM, and follow-up systems your team already runs, and it only pays off when those systems are wired together rather than bolted on. If you're weighing voice against other automation investments, the AI workflow automation guide and the AI for small business hub are the places to zoom back out before you commit engineering time to any one channel.
Small businesses evaluating this category tend to get the sequencing backwards: they pick a voice platform before they've defined what the agent should actually be allowed to do — book directly, or just qualify and hand off. That decision belongs upstream of the STT-LLM-TTS pipeline, not inside it.