Voice AI comes with a pile of acronyms: STT, TTS, VAD, WebRTC, SIP. Voice AI Explained is our series of 22 short videos on how voice AI works, one question at a time, in plain language. Together they run a little over two hours, and most are under six minutes. This guide puts them in learning order, with a short answer to each question: read the answer here, then watch the video for the full explanation.
1. The big picture
How does a voice agent work?
A voice agent does what a person does in a conversation: it listens, thinks and speaks. In the classic pipeline, speech-to-text turns the caller's words into text and notices when they have finished, a language model decides what to say, and text-to-speech speaks the answer. Agents in apps and browsers usually connect over WebRTC, which carries better audio than WebSockets.
▶ Watch: How Voice Agents Work: STT, LLM, TTS, WebRTC (11:44)
What is the voice AI stack?
The stack is the set of layers a voice product is built from: models (speech-to-text, language models, text-to-speech, or one speech-to-speech model), the plumbing that moves the audio (WebRTC for apps, telephony for phone calls), open-source frameworks to assemble a pipeline yourself, managed platforms you configure, finished vertical agents, and the evaluation tools that check them.
▶ Watch: The Voice AI Stack Explained (2026) (4:19)
Should you build a cascaded agent or use a speech-to-speech model?
A cascaded agent chains three models like a relay team: speech-to-text, a language model, then text-to-speech. You can swap each part, check the text before it is spoken and see where an error came from. A speech-to-speech model hears and answers in one step, which keeps the tone, hesitation and humour that a transcript loses.
▶ Watch: Cascaded vs Speech-to-Speech Agents (3:58)
2. Listening: speech-to-text
How does speech-to-text work?
A microphone only measures air pressure, often 16,000 times a second. Speech-to-text slices that audio into short overlapping windows and draws it as a spectrogram; an encoder network turns each moment into numbers describing the sounds, and a decoder writes the words using one of three methods: CTC, encoder-decoder or transducer.
▶ Watch: How Speech-to-Text Works (2026) (4:31)
Why does speech-to-text get names and numbers wrong?
Models favour the common words they heard most, so rare names and brands get swapped for similar-sounding words, and numbers such as fifteen and fifty are easy to confuse. The video covers six places speech-to-text fails on calls and a fix for each, from custom vocabulary and keyword boosting to reading numbers back and checking their format.
▶ Watch: Why Speech-to-Text Gets It Wrong (4:24)
What is speaker diarization?
Diarization works out who spoke when, labelling voices Speaker 0, Speaker 1 and so on. It is different from speaker identification, which matches a voice to a known person. The classic pipeline finds the speech, turns each segment into a voiceprint and groups similar voiceprints; newer end-to-end models do it in a single network.
▶ Watch: Speaker Diarization Explained (7:00)
How do voice agents deal with noise and echo?
To a voice agent, every sound in the room could be speech: steady noise like fans, sudden noise like a door, other voices, its own voice echoing back, and the reverberation of a room. Noise causes false interruptions and wrong words. The fix is a clean-up chain: echo cancellation, then noise suppression, then voice isolation.
▶ Watch: Noise and Echo in Voice Agents, Explained (7:23)
3. Speaking: text-to-speech
How does text-to-speech work?
The classic recipe has four steps: normalise the text (so "$200" becomes "two hundred dollars"), turn letters into phonemes, plan the prosody (rhythm, pitch and pauses), then produce the sound wave. Newer models predict audio tokens or refine noise into speech instead.
▶ Watch: How Text-to-Speech Works, Step by Step (5:42)
How are TTS models trained?
TTS models learn from recordings of real people paired with the exact words spoken: studio recordings, audiobooks, internet speech or synthetic speech. A single classic voice can be trained on about 24 hours of audio, while large flexible models use from 95,000 hours (F5-TTS) to a million hours (CosyVoice 3). Raw internet audio goes through a cleaning pipeline first.
▶ Watch: How TTS Models Are Trained (6:08)
How do speech-to-speech models work?
A speech-to-speech model takes sound in and gives sound out, with no transcript in between. An audio codec cuts speech into short slices described by numbers from fixed lists called codebooks, and the model predicts the next slice the way a text model predicts the next word. Kyutai's Moshi, for example, uses 12.5 slices per second.
▶ Watch: How Speech-to-Speech AI Models Work (9:00)
4. Real-time conversation
Where does voice agent latency come from?
People leave about 200 milliseconds between turns; voice AI builders aim to reply in under 800. That budget goes to the network, deciding the user has finished (endpointing), speech-to-text, the language model's first token, text-to-speech's first audio and playback. The language model and endpointing take the most, and teams win time back with turn detection models and streaming.
▶ Watch: Voice AI Latency: Where the Time Goes (4:30)
How does a voice agent know when to talk?
Turn-taking is two jobs: end-of-turn detection (noticing the user has finished) and barge-in (stopping when the user talks over the agent). Silence timers alone either cut people off or feel slow, so turn detection models listen to the last few seconds for cues in the words and in pitch, loudness and pace.
▶ Watch: Turn-Taking in Voice AI, Explained (4:10)
What is WebRTC, and why do voice agents use it?
WebRTC is a free toolkit built into browsers for live voice, video and data. It favours speed over perfection: instead of pausing for lost packets it skips them, because a tiny gap disturbs a conversation less than a freeze. A call takes four steps: signalling, finding a route (STUN and TURN), encryption, then audio.
▶ Watch: How WebRTC Works, Explained Simply (4:28)
How does a phone call reach an AI voice agent?
Through SIP, a plain-text protocol that sets up, manages and ends calls. Businesses connect over a SIP trunk, a bundle of virtual phone lines over the internet; the call follows a short script (invite, ringing, OK, bye) while the audio travels separately. The video also covers the limits of phone audio and DTMF keypad tones.
▶ Watch: SIP Explained for AI Voice Agents (5:06)
5. Building the agent
How do you prompt a voice agent?
Write for the ear: "You're speaking, not writing." Keep each turn to one idea in one or two sentences, ask one question at a time, leave out formatting such as bullets and links, say dates in full, and expect messy live transcripts from speech-to-text.
▶ Watch: How to Prompt a Voice Agent (4:36)
How do voice agents use tools during a call?
Through function calling: each tool is like a form with a name, a description and blanks that the model fills in; your code runs it and hands the result back for the spoken answer. Tools take time, so agents fill the silence with a short preamble ("Let me check that for you"), progress sounds, or spoken updates when it runs long.
▶ Watch: How Voice Agents Use Tools Mid-Call (8:09)
Can voice AI run on the device?
Running on the device brings privacy, offline use, speed and no cloud fees, which suits apps, cars, wearables and kiosks (phone calls reach a server anyway). Speech-to-text and text-to-speech fit on devices today; the language model is the hard part.
▶ Watch: On-Device Voice AI: What Runs Locally (6:23)
6. Testing and measuring
How do you test a voice agent?
Voice agents are hard to test: a language model can answer the same input differently, and many parts can fail. The answer is a layered plan: test each part on its own (the model on text transcripts, speech-to-text on real recordings), run simulated callers, grade calls with an LLM judge, then monitor real calls.
▶ Watch: How to Test a Voice Agent (2026) (4:18)
How do voice AI benchmarks work, and where do they mislead?
Every benchmark has a test set, a metric and conditions, and changing any of them can reorder the ranking. Speech-to-text is scored by word error rate (WER), often on cleaner audio than real phone calls; text-to-speech relies on human judges, and turn-taking has tests of its own.
▶ Watch: How Voice AI Benchmarks Work (4:51)
7. Cost, build or buy, and the law
How much does a voice AI agent cost per minute?
A phone agent carries five bills per minute of call: the phone line (about 0.5 to 1.5 cents), speech-to-text (0.2 to 2 cents), the language model (under 1 cent for a small model with prompt caching, 2 cents or more without), the voice (0.2 to 3 cents) and the platform's fee. Prompt caching cuts the model's cost three to five times.
▶ Watch: Voice AI Agent Cost Per Minute (2026) (4:59)
Should you build or buy a voice AI agent?
There are four roads: a finished agent that a vendor runs with you (such as Decagon, PolyAI or Sierra), a platform you configure (such as Bland, ElevenLabs, Retell or Vapi, often around five cents a minute plus model and phone costs), an open-source framework you code (such as LiveKit Agents or Pipecat), or your own stack. Each trades control against effort and price.
▶ Watch: Build or Buy a Voice AI Agent? (7:43)
Is your AI voice agent legal?
The rules follow where the person you call is, and five matter most: telling people they are talking to an AI (the EU AI Act requires it by the start of the conversation, as do some US states such as Maine), consent for recordings and transcripts, rules for outbound calls, voice cloning, and data privacy. The simplest habit: say it is an AI in the first sentence. This is not legal advice.
▶ Watch: Is Your AI Voice Agent Legal? 5 Rules (5:12)
Where to go next
Every layer in this guide has products behind it. Browse speech-to-text, text-to-speech and voice cloning tools in the Tech section of the library, the platforms and frameworks for building agents in the Platform section, and testing tools in Eval.
