Why Voice AI Agents Fail - Voice AI Space Conference NYC

Voice AI agents often fail because developers treat them as text-based systems with audio layered on top. In reality, voice agents are concurrent systems rather than turn-based ones, which introduces unique engineering and implementation challenges.

Four Key Failure Modes of Voice AI Agents

• Turn-taking complexity: Unlike text-based systems where input is explicitly sent, voice users pause to breathe or think. Agents struggle to distinguish between a temporary pause and the end of an utterance. A potential solution is using a small, fast parallel model (such as a 2-billion parameter model) to classify whether the silence represents a complete thought.

• Ambiguous backchannels: Simple words like "yes" or "hmm" can serve as either direct instructions or conversational fillers (backchannels), depending on the context. Engineers can address this by counting words or using classifiers to determine the user's intent, though using large language models (LLMs) for this adds latency and cost.

• System disagreement: Because voice systems rely on multiple connected components (such as Automatic Speech Recognition and Text-to-Speech), they can struggle when a user interrupts or changes instructions mid-process. Implementing "cooperative cancellation", where a token is passed through the system to halt background processes: helps the agent handle interruptions and determine how far to roll back the conversation.

• Premature goodbyes: It is difficult for an agent to determine when a conversation has actually ended. An agent might interpret a polite acknowledgment like "that makes sense" as a signal to end the call prematurely.

The Latency and Capacity Trade-off

Optimizing voice agents requires balancing latency and model capacity. Smaller models provide the fast response times necessary for a natural conversational flow but have limited context windows, making them prone to losing track of longer conversations. Larger, more robust models can handle long-horizon conversations but introduce latency that degrades the user experience.

Rasa's Voice-Native Approach

To address these challenges, conversational platforms are shifting from text-first frameworks to voice-native architectures. Rather than relying on pure agentic wrappers, this approach utilizes control gates and deterministic rules alongside generative AI to steer conversations effectively. A course detailing these failure modes and engineering solutions is available on GitHub under "Why Agents Fail."