How Voice Agents Work: STT, LLM, TTS, WebRTC & Speech-to-Speech

The Voice AI Pipeline

A voice AI agent operates similarly to a human conversation, which involves listening, understanding, thinking, and speaking. The standard voice AI pipeline consists of three main components:

• Speech Detection / Speech-to-Text: Understands what is being said and detects when the speaker has finished.
• Large Language Model (LLM): Serves as the "brain" to process the input and determine the appropriate response.
• Text-to-Speech (TTS): Converts the LLM's text response into spoken audio.

Transport Layers: WebRTC vs. WebSockets

To connect a user to a voice agent, developers typically use one of two transport protocols:

• WebRTC: Generally preferred for voice AI because it supports higher bandwidth (48 kHz), resulting in cleaner transcription, better LLM processing, and higher-quality audio output.
• WebSockets: Often results in lower audio quality, which can degrade the performance of the speech-to-text, LLM, and text-to-speech models.

Architectural Approaches

There are two primary paradigms for building voice AI agents:

• Cascaded Pipeline: This approach connects separate speech-to-text, LLM, and text-to-speech models. Rather than processing sequentially, modern cascaded pipelines use a continuous streaming loop where the LLM streams chunks of its response to the TTS model before it has finished thinking the entire sentence, reducing latency.
• Speech-to-Speech Unified Model: This approach bakes speech-to-text, LLM, and text-to-speech capabilities into a single unified model. It offers lower latency and a simpler architecture but comes with higher costs and significantly less flexibility for customization or regional adjustments.

Analyzing Failure Scenarios

To build better user experiences, developers should consider how technical failures map to human conversation difficulties. For example, high latency is comparable to thinking slowly, a text-to-speech failure is like having a sore throat, and speech-to-text errors are similar to not hearing clearly.