Voice AI Space Singapore - Beng Hui Ong @ Agora

Agora's Conversational AI Capabilities

Agora provides software and SDK solutions that enable developers to build real-time AI agents. The conversational AI workflow operates by capturing and encoding audio, which is then transmitted through Agora's global infrastructure (SD-RTN). This audio is processed by a conversational AI engine using a cascade model that integrates Speech-to-Text (STT), a Large Language Model (LLM) or Retrieval-Augmented Generation (RAG) system, and Text-to-Speech (TTS). The TTS can also be linked to a digital avatar. The resulting audio and video are sent back through the global infrastructure to the user, facilitating natural interactions that emulate human conversation through features like turn detection, silent detection, and variations in tone and speed.

Workshop Demonstration

A live demonstration illustrates the integration of these technologies to create a virtual assistant. The demo utilizes Agora STT, OpenAI as the LLM, Minimax TTS to emulate a specific colleague's voice, and the Akool Avatar service to replicate her physical appearance. During the interactive demo, the speaker asks the digital avatar about Singapore, and the avatar responds in real time. The code template for replicating this workshop demo is open-source and hosted on GitHub.

Open-Source AI Hardware Gadget

The speaker also presents an open-source hardware device designed to demonstrate physical AI integration. This mobile 4G device features:

• Moving eyes and an accelerometer to detect physical movement.
• A camera that allows the device to see the user.
• A voice interface connected to Agora's conversational AI.

In a brief demonstration, the speaker activates the device and asks it for information about Singapore, which it successfully retrieves and speaks aloud. This hardware showcases how AI agents can be deployed in diverse physical shapes and forms.