Customizing Speech Models for Any Industry - Voice AI Space Amsterdam

Overview of Resonate

Resonate is a speech-to-text company focused on building customizable ASR models for specialized domains. The speaker explains that traditional customization methods, such as fine-tuning or fixed hotword lists, often fail when the model is deployed in new acoustic environments, locations, or real-world usage conditions.

The core idea is that modern speech recognition should not only transcribe what it hears but also reason about what could plausibly have been said based on context. This is especially important in domains like healthcare, where the system may need to recognize thousands of complex medical terms, medication names, or specialized vocabulary.

Main Functions

Customizable Speech-to-Text:
Resonate builds speech-to-text systems that can adapt to specific domains and vocabularies. Instead of relying only on generic transcription, the system can prioritize terms that are more likely in a particular context, such as medical conversations.

Context Biasing:
The platform uses context biasing during decoding. This allows the model to choose the most likely words from a large vocabulary by considering the surrounding context, rather than treating every possible word equally.

System Architecture

The speaker describes speech models using the analogy of “ears” and “brains.” The encoder acts as the ears, listening to and processing the raw audio signal. The decoder acts as the brain, interpreting the audio and producing the final text output.

Different model architectures balance these parts differently. For example, Parakeet models are described as having very strong “ears” but smaller “brains,” which can lead to spelling variations or mistakes when handling specialized terms like medication names.

The presentation also compares models such as Whisper and Parakeet, showing that the best base models often outperform custom architectural tweaks. Resonate’s approach is built around using strong base models and improving them with promptable ASR and context biasing rather than relying only on fine-tuning.

Key Learnings

• Fine-Tuning Does Not Always Generalize:
  Fine-tuning a speech model may work in one environment but fail in another. A model trained on one acoustic setup, location, or user group may perform poorly when deployed somewhere else.

• LLMs Can Help but Also Hallucinate:
  Large Language Models can act like a “brain” that corrects transcription mistakes, but they can also invent terms that were never spoken. This is dangerous in specialized domains where accuracy matters.

• Context Biasing Is More Scalable:
  Instead of hardcoding a few hotwords or forcing the model to memorize every possible term, context biasing lets the model select from a large vocabulary based on the conversation context. This makes it useful for domains with huge terminology lists, such as healthcare.

• Strong Base Models Matter Most:
  The speaker emphasizes that state-of-the-art base models often perform better than custom architectures. The best path is usually to start with a powerful foundation model and then guide it intelligently with context and prompts.

Technical Details and Q&A

Technology Stack:
Resonate works with speech-to-text model architectures such as Whisper and Parakeet. The company also owns compute clusters in Amsterdam, which it uses to train foundational speech models.

Encoder and Decoder Design:
The encoder processes the audio and captures acoustic information, acting as the model’s “ears.” The decoder interprets the encoded audio and generates the final transcription, acting as the model’s “brain.”

Reasoning / Agent Logic:
Resonate trains models to predict what could have been said, not only what was directly heard. This allows the system to consider plausible alternatives and choose the best transcription based on context.

Customization Approach:
Earlier customization methods included short prompts, hotwords, and fixed dictionaries. The modern approach uses context biasing, which can handle much larger vocabularies and make more intelligent decisions during decoding.

Healthcare Use Case:
In healthcare, the model may need to choose from very large vocabularies, such as 200,000 medical terms. Context biasing helps narrow down the most likely candidates based on the conversation, reducing errors without requiring the model to memorize every term perfectly.

Quality Assurance:
Quality is measured by transcription accuracy in real-world conditions, especially when dealing with specialized vocabulary. The goal is to avoid hallucinations, spelling inconsistencies, and failures caused by new acoustic environments.

Important Keywords and Definitions

• Context Biasing:
  A decoding technique where the model is guided to prioritize specific words or terms based on the surrounding context.

• Hallucination:
  When a model generates words or terms that were not actually present in the audio.

• Decoder:
  The part of the speech model that interprets audio features and produces the final text output. It acts as the “brain.”

• Encoder:
  The part of the model that processes raw audio into useful representations. It acts as the “ears.”

• ASR:
  Automatic Speech Recognition, the technology used to convert spoken language into written text.

• Parakeet:
  A speech model architecture mentioned in the talk. It has strong audio-processing ability but may struggle with spelling consistency for specialized terms.

• Hotwords:
  Specific words or phrases given priority so the model is more likely to recognize them.

• Customization:
  The process of adapting a speech model to a specific domain, vocabulary, or business use case.

• Analogy:
  Think of a speech model like a detective with strong hearing but a limited memory. It may hear the clues clearly but still guess the wrong suspect if it lacks context. Context biasing is like giving the detective a focused suspect list based on the case, helping it choose the most likely answer without inventing something that was never there.