What's new in AI Audio at DeepMind? - Voice AI Space Conference NYC

Overview of Google DeepMind's AI Audio Models

The presentation highlights the rapid development and release of Google DeepMind's frontier and open models, focusing on their advanced audio capabilities. Key models discussed include the Gemini 3 family (such as Flash and Flash Light), the open-source Gemma 4 models (which can run on-device and natively support multimodal inputs like audio, images, and video), and specialized media models like Gemini 3.1 Pro Image (referred to as "NanoBanana" during benchmarking).

Audio Understanding with Gemini

Unlike traditional Automatic Speech Recognition (ASR) systems that only transcribe text, Gemini models function as reasoning and thinking engines. They understand more than just the literal words spoken. Using a single audio prompt, the model can perform multiple tasks simultaneously:

Provide a concise summary of the audio content.
Generate a detailed transcript with precise timestamps.
Identify and label different speakers by name.
Detect multiple languages within the same audio stream (including English, German, French, Japanese, and Mandarin Chinese) and provide English translations.
Analyze and identify the speaker's emotional tone (such as happy, neutral, or sad).

Expressive Speech Generation (TTS)

The Gemini 3.1 Flash TTS model represents Google's most expressive speech generation technology. Because it leverages Gemini's deep audio understanding, users can prompt the model using a highly descriptive structure to craft specific performances. This prompting structure includes:

Audio Profile: Defining the persona, character identity, age, and background.
Scene: Setting the physical environment and overall vibe (e.g., a bustling food market or a large auditorium).
Director's Notes: Providing performance guidance on style, breathing, pacing, articulation, and accent.
Audio Tags: Inserting inline modifiers within the transcript to trigger specific vocal behaviors like whispering, shouting, or laughing.

Demos of this technology include a high-pitched Irish tech evangelist persona and a Singaporean hawker center regular speaking in a distinct Singlish accent, both showing highly realistic laughter and pacing compared to their flat base voices.

Real-Time Multimodal Interaction (Gemini Live)

The Gemini 3.1 Flash Live model enables real-time, bidirectional speech-to-speech communication via a stateful WebSocket API. This allows developers to stream text, audio, and video frames to the model, which then processes the inputs and responds instantly with audio and text transcripts. During a live demonstration using the ai.studio/live playground, the speaker showcases several capabilities of Gemini Live:

Computer Vision: The model successfully looks through the speaker's camera to identify and read the text on his cap ("ai.studio") and describe his outfit.
Real-Time Grounding: Using the Google Search tool call, the model retrieves live weather information for New York City to advise the speaker on whether he needs an umbrella.
Accent and Language Switching: The model natively switches from English to German while maintaining its assigned Irish-accented voice ("Puck").

Native Audio vs. Cascaded Pipelines

The speaker compares native audio models (which process sound-to-sound directly without converting to text first) with traditional cascaded pipelines (ASR followed by an LLM, followed by TTS):

Native Audio: Offers extremely low latency and preserves emotional nuances, accents, and environmental context. However, they currently act as "black boxes," making them harder to control.
Cascaded Pipelines: Remain highly relevant for enterprise use cases because they provide observability, safety checks, and the ability to intercept and rewrite text before it is spoken.