Voice AI Space Singapore - Michelle Duke @ Twilio

Conversational AI represents the third generation of voice technology, succeeding first-generation assistants like Siri and second-generation assistants like Alexa. Unlike its predecessors, which were limited to one-off commands, conversational AI is natural, effortless, and capable of spanning multiple topics and maintaining context over time. It allows for hands-free, interactive, and accessible communication without the need for buttons, forms, or menus.

Challenges of Voice AI

Implementing voice AI is technically challenging because human conversations are naturally messy. Real-world speech includes interruptions, pauses, varying accents, and different pacing. Additionally, applying voice AI to complex domains like gaming (e.g., Magic: The Gathering) introduces further complications, such as intricate rules, informal "house rules," and strategic ambiguity (distinguishing between what a player can do versus what they should do).

How Voice AI Works

Building a voice AI application typically requires several interconnected components:

• Speech-to-Text (STT): Translates spoken audio into text so the system can process it.
• Large Language Model (LLM): Analyzes the text to understand the user's intent and generate an appropriate response.
• Text-to-Speech (TTS): Converts the generated text response back into spoken audio for the user.

Developers must also program complex logic to handle interruptions, language detection, and accent variations. To simplify this process, Twilio offers a tool called Conversational Relay, which packages STT, TTS, and interruption handling into a few lines of code, allowing developers to focus on their application's core logic and data.

Addressing Latency

Latency is a common issue in voice AI applications. Several strategies can be used to minimize or manage delay:

• Geographic Deployment: Deploying the application on servers physically close to the user and the AI models.
• Token Streaming: Processing and delivering the response incrementally as it is being generated, rather than waiting for the entire response to complete.
• Multiple AI Agents: Running multiple agents simultaneously, where one agent engages the user with conversational filler (e.g., acknowledging the question) while another retrieves the requested information in the background.