Voice AI comes with a pile of acronyms: STT, TTS, VAD, WebRTC, SIP. Voice AI Explained is our series of 22 short videos on how voice AI works, one question at a time, in plain language. Together they run a little over two hours, and most are under six minutes. This guide puts them in learning order, with a short answer to each question: read the answer here, then watch the video for the full explanation.
1. The big picture
How does a voice agent work?
A voice agent does what a person does in a conversation: it listens, thinks and speaks. In the classic pipeline, speech-to-text turns the caller's words into text and notices when they have finished, a language model decides what to say, and text-to-speech speaks the answer. The steps overlap: the agent keeps listening, and the model streams its first words to the voice before it has finished thinking. The other design is one speech-to-speech model that does all three jobs: simpler to start with, but usually pricier and harder to tune. Either way, apps and browsers get the best audio over WebRTC rather than WebSockets.
▶ Watch: How Voice Agents Work: STT, LLM, TTS, WebRTC (11:44)
What is the voice AI stack?
The stack is the set of layers a voice product is built from. Models are the ingredients: speech-to-text, a language model and text-to-speech, or one speech-to-speech model, plus small parts like voice activity and turn detection. The plumbing moves the audio: WebRTC for apps, telephony for phone calls. Then come open-source frameworks you assemble yourself, managed platforms you configure, finished agents for one industry, evaluation tools that test them, and hardware. The borders keep moving and one company can play several roles, so check that two vendors play the same one before you compare them.
▶ Watch: The Voice AI Stack Explained (2026) (4:19)
Should you build a cascaded agent or use a speech-to-speech model?
A cascaded agent chains three models like a relay team: speech-to-text, a language model, then text-to-speech. You can swap each part, check the text before it is spoken and see where an error came from. A speech-to-speech model hears and answers in one step: it can pick up the tone, hesitation and humour that a transcript loses, and often replies faster, but you get one model's voices and less control. The rule of thumb: a cascade when you need a specific brand voice, strict rules or detailed records, as in banking or healthcare; speech-to-speech for the most natural back-and-forth with a standard voice. Test both on real calls.
▶ Watch: Cascaded vs Speech-to-Speech Agents (3:58)
2. Listening: speech-to-text
How does speech-to-text work?
A microphone only measures air pressure, often 16,000 times a second for speech-to-text. The model slices that audio into short overlapping windows and draws it as a spectrogram, and an encoder network turns each moment into numbers describing the sounds. One of three methods then writes the words: CTC, which guesses a letter or nothing at each moment; a decoder that writes one piece at a time while looking back at the audio, as Whisper does; or a transducer, which mixes the two and suits live speech. Streaming models send early guesses that can change before they are marked final.
▶ Watch: How Speech-to-Text Works (2026) (4:31)
Why does speech-to-text get names and numbers wrong?
Models favour the words they heard most, so rare names and brands get swapped for similar-sounding words, and numbers such as fifteen and fifty are easy to confuse. The video covers six weak spots: rare words, numbers, background noise, thin phone audio, accents and mixed languages, and words invented during silence. Each has a fix, from custom vocabulary and reading numbers back to a voice activity detector that only passes real speech. And measure the words that matter, such as names and numbers, not just the overall word error rate.
▶ Watch: Why Speech-to-Text Gets It Wrong (4:24)
What is speaker diarization?
Diarization works out who spoke when, labelling voices Speaker 0, Speaker 1 and so on. It is different from speaker identification, which matches a voice to a known person. The classic pipeline finds the speech, turns each segment into a voiceprint and groups similar voiceprints; end-to-end models do it in one network and handle people talking over each other, up to a fixed number of speakers. Accuracy is measured with the diarization error rate (DER). On phone calls you often don't need it: record the agent and the caller on separate channels.
▶ Watch: Speaker Diarization Explained (7:00)
How do voice agents deal with noise and echo?
To a voice agent, every sound in the room could be speech: steady noise like fans, sudden noise like a door, other voices, its own voice echoing back, and the reverberation of a room. Noise causes false interruptions, wrong words and wrong actions. The clean-up chain runs echo cancellation first, then noise suppression, then voice isolation. The catch: audio that sounds cleaner to you is not always easier for speech-to-text, so raise the strength gradually, measure errors with it on and off, and never stack two noise removers.
▶ Watch: Noise and Echo in Voice Agents, Explained (7:23)
3. Speaking: text-to-speech
How does text-to-speech work?
The classic recipe has four steps: normalise the text (so "$200" becomes "two hundred dollars"), turn letters into phonemes, plan the prosody (rhythm, pitch and pauses), then produce the sound wave. Newer models predict audio tokens the way a chatbot predicts words, or refine noise into speech. For a voice agent, the number that matters is time to first audio, so the voice starts playing while the rest is still being made. Test any voice on your own product names, numbers and dates.
▶ Watch: How Text-to-Speech Works, Step by Step (5:42)
How are TTS models trained?
TTS models learn from recordings of real people paired with the exact words spoken: studio recordings, audiobooks, internet speech or synthetic speech. A single classic voice can be trained on about 24 hours of audio, while large flexible models use from 95,000 hours (F5-TTS) to a million hours (CosyVoice 3), and raw internet audio goes through a cleaning pipeline first. Data is also where the hard questions are: before you choose a voice, ask where the data came from, whether the speakers agreed, whether you can use it commercially, and what stops someone cloning a voice without permission.
▶ Watch: How TTS Models Are Trained (6:08)
How do speech-to-speech models work?
A speech-to-speech model takes sound in and gives sound out, with no transcript passed between models. An audio codec cuts speech into short slices described by numbers from fixed lists called codebooks, and the model predicts the next slice the way a text model predicts the next word. Kyutai's Mimi codec, used by Moshi, makes about 100 audio tokens per second of speech, against 3 or 4 text tokens for the same words. Many models write their answer as text alongside to stay smart, yet they still trail text models on long tasks: in one customer service test, voice agents completed 31 to 51% of tasks, against 85% for a text model that reasons first.
▶ Watch: How Speech-to-Speech AI Models Work (9:00)
4. Real-time conversation
Where does voice agent latency come from?
People leave about 200 milliseconds between turns; many voice AI builders aim to reply in under 800. That budget goes to the network, deciding the user has finished, speech-to-text, the language model's first token, text-to-speech's first audio and playback. In a published example, the language model and deciding the user had finished took the most. Teams win time back with turn detection models and streaming, so the steps overlap like an assembly line. Measure the slow turns too: the one turn in 20 that takes over a second and a half is what makes an agent feel unreliable.
▶ Watch: Voice AI Latency: Where the Time Goes (4:30)
How does a voice agent know when to talk?
Turn-taking is two jobs: end-of-turn detection (noticing the user has finished) and barge-in (stopping when the user talks over the agent). Silence timers alone either cut people off or feel slow, so turn detection models listen to the last few seconds for cues in the words and in pitch, loudness and pace. Every detector trades cutting people off against making them wait. When interrupted, the agent should stop fast, forget what it never said, and ignore a cough or an "mm-hm", which means "keep going".
▶ Watch: Turn-Taking in Voice AI, Explained (4:10)
What is WebRTC, and why do voice agents use it?
WebRTC is a free toolkit built into browsers for live voice, video and data. It favours speed over perfection: instead of pausing for lost packets, as WebSockets do, it skips them, because a tiny gap disturbs a conversation less than a freeze. A call takes four steps: signalling, finding a route (STUN and TURN), encryption, then audio. The AI sits on a server, reached directly or through a hub (an SFU) that passes the audio around.
▶ Watch: How WebRTC Works, Explained Simply (4:28)
How does a phone call reach an AI voice agent?
Through SIP, a plain-text protocol that sets up, manages and ends calls. Businesses connect over a SIP trunk, a bundle of virtual phone lines over the internet; the call follows a short script (invite, ringing, OK, bye) while the audio travels separately. The call then reaches the AI in one of two ways: the phone provider streams the audio to your server over a WebSocket, or the SIP trunk connects straight to a system that speaks SIP, such as LiveKit or OpenAI's Realtime API. Phone audio is thin (8,000 samples a second), so test with real calls.
▶ Watch: SIP Explained for AI Voice Agents (5:06)
5. Building the agent
How do you prompt a voice agent?
Write for the ear: "You're speaking, not writing." Keep each turn to one idea in one or two sentences, ask one question at a time, leave out formatting such as bullets and links, and say dates in full. Expect messy live transcripts from speech-to-text, so have the model read names and numbers back. Have the agent say it is an AI, and remember that a prompt is not a lock: anything it must never do should also be blocked in your code.
▶ Watch: How to Prompt a Voice Agent (4:36)
How do voice agents use tools during a call?
Through function calling: each tool is like a form with a name, a description and blanks that the model fills in; your code runs it and hands the result back for the spoken answer. Tools take time, so agents fill the silence with a short preamble ("Let me check that for you"), progress sounds or spoken updates. Before anything that books, cancels or charges, the agent should read back what it heard and confirm, because a misheard name or date becomes a wrong action. And a tool result is information, never orders: text inside it can try to hijack the agent.
▶ Watch: How Voice Agents Use Tools Mid-Call (8:09)
Can voice AI run on the device?
Running on the device brings privacy, offline use, speed and no cloud fees, which suits apps, cars, wearables and kiosks (phone calls reach a server anyway). Speech-to-text and text-to-speech fit on devices today. The language model is the hard part: fine for short tasks, weak on broad knowledge, and limited by memory, battery and heat. So many products split the work: small models on the device, the big model in the cloud.
▶ Watch: On-Device Voice AI: What Runs Locally (6:23)
6. Testing and measuring
How do you test a voice agent?
Voice agents are hard to test: a language model can answer the same input differently, and many parts can fail. The answer is a layered plan: test each part on its own (the model on text transcripts, speech-to-text on real recordings), run simulated callers, and grade calls with an LLM judge that you check against people. Measure task success in your own system, not by what the agent said. Turn every failed call into a test, rerun the set whenever the prompt, model or provider changes, and keep listening to real calls after launch.
▶ Watch: How to Test a Voice Agent (2026) (4:18)
How do voice AI benchmarks work, and where do they mislead?
Every benchmark has a test set, a metric and conditions, and changing any of them can reorder the ranking. Speech-to-text is scored by word error rate (WER); text-to-speech relies on human ratings and arenas; turn-taking and whole-agent tests such as Tau-bench are newer. They mislead when the test comes from a vendor whose own model tops it, when the audio is cleaner than real phone calls, or when a correct answer stands in for how a call feels. The benchmark that matters most is a few dozen of your own calls.
▶ Watch: How Voice AI Benchmarks Work (4:51)
7. Cost, build or buy, and the law
How much does a voice AI agent cost per minute?
A phone agent carries five bills per minute of call: the phone line (about 0.5 to 1.5 cents), speech-to-text (0.2 to 2 cents), the language model (well under a cent for a small model with prompt caching, 2 cents or more without), the voice (a fraction of a cent to about 3 cents) and the platform (about 1 to 5.5 cents). Added up at mid-range list prices from September 2026, that is about 8.5 cents a minute, or 43 cents for a five-minute call, and about 5 cents a minute on a cheaper platform. Compare the cost per successful call, not just per minute.
▶ Watch: Voice AI Agent Cost Per Minute (2026) (4:59)
Should you build or buy a voice AI agent?
There are four roads: a finished agent that a vendor runs with you (such as Decagon, PolyAI or Sierra), a platform you configure (such as Bland, ElevenLabs, Retell or Vapi, often around five cents a minute plus model and phone costs), an open-source framework you code (such as LiveKit Agents or Pipecat), or your own stack. The rule of thumb: a platform for low volume, no audio engineers and a fast launch; a framework for high volume, strict data rules and full control; a finished agent for a large contact centre that wants results. On cost alone, the video's example only favours a framework from about 375,000 minutes a month.
▶ Watch: Build or Buy a Voice AI Agent? (7:43)
Is your AI voice agent legal?
The rules often follow the person on the call, not just where your company is, and five matter most: telling people they are talking to an AI (the EU AI Act requires it by the start of the conversation, and some US states such as Maine have their own rules), consent for recordings and transcripts, consent before outbound calls, written consent for cloned voices, and data privacy. The simplest habits: say in the first sentence that it is an AI and that the call is recorded, and always offer a way to reach a human. This is not legal advice.
▶ Watch: Is Your AI Voice Agent Legal? 5 Rules (5:12)
Where to go next
Every layer in this guide has products behind it. Browse speech-to-text, text-to-speech and voice cloning tools in the Tech section of the library, the platforms and frameworks for building agents in the Platform section, and testing tools in Eval.



