My name is... my name is...: A Linguistic Framework for Debugging Voice AI Failures - Voice AI NYC

The Challenge of Voice AI Failures

Voice AI systems frequently experience failures during basic interactions, such as when users attempt to spell out their names or provide account numbers. These failures often stem from the system mishearing similar sounds (like "M" and "N"), mispronouncing names, using confusing terminology, or interrupting the user. Such issues lead to user frustration, task failures, abandoned calls, and a persistent preference for human agents over voice agents.

Human Communication as a Joint Activity

Human communication is a real-time, subconscious, and automatic joint activity characterized by interactive alignment. For Voice AI to succeed, it must also function as a joint activity. Communication operates across three linguistic levels, sounds, words, and interaction: and two primary processes: listening and speaking.

A Linguistic Framework for Voice AI

To diagnose and resolve failures, Voice AI systems must be evaluated across a multi-layered linguistic framework:

• Listening:

  • Sounds: Recognizing speech accurately, which involves managing signal-to-noise ratios, voice activity detection, noise suppression, speaker identification, prosody, and accents.

  • Words: Understanding the user's vocabulary, including named entities, out-of-vocabulary terms, alphanumerics, jargon, dialects, and multilingual elements.

  • Interaction: Detecting turns accurately, managing latency, and waiting for the appropriate timing to respond.

• Speaking:

  • Sounds: Pronouncing speech clearly with natural voice quality, proper phonemes, and prosody without hallucination.

  • Words: Selecting appropriate vocabulary that the user can easily comprehend.

  • Interaction: Delivering information with the correct timing and content alignment, adhering to conversational principles like Grice's maxims, and utilizing interactive grounding.

Interdependence and Orchestration

The components of this framework are highly interdependent. Sounds are derived from words, words are expressed through sounds, and both exist within conversational turns. Consequently, component-specific benchmarks do not guarantee a positive user experience. Successful Voice AI requires the holistic orchestration of all these layers in real-time. To assist with this, ServiceNow has introduced EVA (End-to-End Voice AI Evaluation), a benchmark that measures both accuracy (EVA-A) and experience (EVA-X).

Key Takeaways

• Voice AI must be treated as a real-time joint activity rather than a simple sequential pipeline.

• Systems must address user needs across multiple layers simultaneously, with the potential to integrate additional modalities like vision, gesture, and emotion.

• Organizations should study linguistics and employ linguists to design better conversational partners.

• Because speakers adapt and languages evolve over time, Voice AI systems must be designed to accommodate long-term linguistic changes.