Sony just put its name in the all-in-one voice model race. On November 10, its Speech & Language AI Lab in Tokyo published Walrus, an 8B model that transcribes speech, translates it, speaks the translation back and answers questions about audio, all with one set of weights. Sony says the translated speech keeps the original speaker's voice. One catch before you go looking for it: it's a paper and a demo for now. No weights, no code, no license.

TL;DR

  • One model for speech recognition, speech translation (to text and to speech), text-to-speech, emotion recognition, audio captioning and audio Q&A.

  • Built on Google's Gemma 4 E4B, trained on more than 600M samples across 20+ tasks and 100 languages.

  • Two versions: Commercial-Ready (commercially usable data only) and Research (adds non-commercial data, scores higher).

  • Biggest leads, per Sony: mixed-language speech and speech translation. Weaker on emotion recognition and audio captioning.

  • All benchmarks are Sony's own. Nothing to download yet.

Sony and voice: a longer story than you'd think

Walrus isn't Sony's first time near a microphone. Here's the short version:

So Sony has been working on voice tech for more than 25 years. What's new with Walrus is that Sony is building its own general-purpose speech model, the same category as Qwen-Omni, Phi-4-Multimodal and Gemma.

For everyone: what Walrus actually is

Most voice products today are a chain of separate tools: one that turns speech into text, one that translates, one that turns text back into speech. Each handoff can lose something, like tone of voice, and adds delay.

Walrus puts those jobs in one model. You tell it what you want in plain words, such as "Translate the Japanese speech into English text, and then speech", and it does it. Sony says the translated speech keeps the original speaker's voice and emotion, and reports that Walrus does this better than Qwen3-Omni and SeamlessM4T v2. Nobody outside Sony has measured Walrus yet. The same Sony team did publish a way to measure it earlier this year: COMPASS, a benchmark that scores cascaded and end-to-end speech-to-speech translation systems on 46 metrics across eight dimensions, including speaker preservation and naturalness. Co-author Alkis Koudounas told us it was published at EMNLP 2026.

Three demo examples show where this helps day to day:

  • Getting names right: give it a list of terms and it fixes a misheard drug name in a medical transcript.

  • Mixed languages: a sentence that jumps between English and Japanese comes out correctly in both scripts.

  • Captions that hear everything: transcripts with timestamps, who's speaking, and labels for music and sound effects.

Why should a non-specialist care? Dubbing, live translation, accessibility captions and customer service calls in several languages could get simpler and cheaper when one model does the whole job.

Why you can't use it yet: Sony hasn't released the model. The only public page is the demo site, and there's no Walrus on Hugging Face.

For the technical crowd: how it's built

Architecture. Walrus is decoder-only, initialized from Gemma 4 E4B (about 4B effective parameters, about 8B counting embeddings). Sony left the architecture unchanged and fine-tuned every parameter, including the USM audio encoder, on purpose: any gain can be traced to data, not model design.

Speech output. Walrus generates discrete speech tokens. A single-layer VQ codebook of 8,192 entries quantizes layer-20 features from W2v-BERT 2.0, downsampled from 50 Hz to 25 Hz. That's 13 bits per token, or 325 bits per second. A VQ decoder plus a Vocos-based GAN vocoder (building on UniCATS / CTX-vec2wav) turns tokens into audio. The vocoder is conditioned on layer-8 W2v-BERT features from the source audio, which is how speaker identity and prosody carry over into translated speech.

Task prompting. No task tokens. Each task samples natural-language instructions from a template pool during training, so the same weights switch tasks from the prompt.

Three-stage curriculum.

Stage

Steps

LR

What changes

1. Foundation

87,500

2e-6

Broad multilingual ASR, MT, speech translation, S2ST, TTS, emotion, captioning, audio QA. ASR weighted 5.0, TTS 0.25

2. Expansion

135,000

2e-6

Adds multi-talker ASR, diarization, timestamps, code-switching, closed captions, per-dataset weights. Output = Commercial-Ready

3. Specialization

150,000

1e-6

Adds non-commercial and synthetic entertainment speech. ASR down to 4.0; TTS, S2ST, captioning and emotion up to 2.0 (audio Q&A stays at 1.0). Output = Research

Overall mix: recognition 45%, translation 27%, understanding 21%, generation 7%. Training took about 30 days on 128 H100s (around 92,000 GPU hours), batch size 2,048.

Keyword biasing. Keywords come from named-entity recognition (kept if they appear in at least two utterances). Each training list mixes true keywords with distractors, with list sizes drawn from 0, 10, 20, 50 or 100, so the model learns to ignore irrelevant terms.

Synthetic code-switching data. 13 language pairs. Gemma 4 26B writes mixed-language sentences across 25 everyday domains; Qwen3-TTS or an in-house Chatterbox variant voices them segment by segment with one reference speaker. Clips are filtered at NISQA 3.5 or higher and at most 5% error under Whisper Large-v3 Turbo.

Multi-talker data. Real meetings (AMI, AliMeeting, AISHELL-4) plus simulated mixtures (LibriMix, LibriSpeechMix, FastMSS), 2 to 4 speakers, seven languages, clips up to 30 seconds.

How does it compare?

Benchmark (metric)

Best prior system, per Sony

Walrus Commercial-Ready

Walrus Research

FLEURS ASR (WER, lower is better)

4.00 (Phi-4-MM)

7.54

5.33

VoxPopuli ASR (WER)

11.78 (Gemma 4)

8.38

7.84

LibriSpeech clean/other (WER)

1.57/3.13 (Audio Flamingo 3)

1.88/4.99

1.65/3.85

CS-FLEURS code-switching, read (CER)

32.30 (Gemma 4)

26.69

7.23

CoVoST-2 speech translation X→En (BLEU, higher is better)

39.33 (Phi-4-MM)

40.28

42.63

CVSS-C speech-to-speech X→En (ASR-BLEU)

36.02 (Qwen2.5-Omni)

43.27

43.63

IEMOCAP emotion (accuracy)

63.80 (Audio Flamingo 3)

41.44

49.55

All figures from Sony's technical report. "Best prior" is the strongest of Gemma 4, Phi-4-MM, Audio Flamingo 3 and Qwen2.5-Omni (4B to 10B models). Nothing has been reproduced independently.

The standout is code-switching: 7.23 CER against 32.30 for the best prior system. Against the larger 30B Qwen3-Omni, Sony reports 94.7% word accuracy on FLEURS (93.8% for Qwen3-Omni, 92.0% for Whisper Large-v3), 43.63 vs 41.27 ASR-BLEU on CVSS-C speech-to-speech translation, and 19.32 vs 37.04 cpWER on two-speaker overlapping speech (LibriMix, which Sony notes is part of Walrus's own training data, so that test favours Walrus). Qwen3-Omni stays ahead on FLEURS speech-to-speech translation in both directions: 33.60 vs 32.28 into English and 25.67 vs 22.94 out of English.

Where it falls short, in Sony's own words and numbers: emotion recognition (emotion data was only about 0.3% of training), audio captioning (0.39 CIDEr vs 0.50 for Audio Flamingo 3), and real conversational Japanese, where the error rate is "still high in absolute terms". The Commercial-Ready model, the one you'd be able to ship, trails the best prior system on LibriSpeech, MLS, FLEURS, emotion, captioning, both audio understanding tests (MMAU-Pro 50.18 vs 52.20, MMSU 55.18 vs 61.40) and Sony's internal entertainment test. The biggest gain between the two versions is on code-switching (26.69 to 7.23 CER), which Sony credits to more synthetic code-switching data in stage 3. Sony also notes it did not separate the effect of new data from the new task weights.

What's missing

  • Weights, code and a license, and any word on whether they're coming.

  • Latency and inference cost. The report gives training compute but no real-time speed figures, which voice agent builders will want to see.

  • Independent results. Every number above comes from Sony.

What it means if you build voice agents

For now, Walrus is a recipe, not a product. The curriculum, the task weights and the synthetic code-switching and multi-talker pipelines are described in enough detail to reuse if you fine-tune your own speech models. If Sony does ship the Commercial-Ready weights, the number to test on your own audio is the gap between the two versions.

Sources