How Speech-to-Text Works: From Sound Waves to Words (2026)

From Sound Waves to Numbers and Pictures

A microphone does not hear words; it only measures air pressure moving up and down. To process this, a device measures the pressure thousands of times per second—often 16,000 times a second for speech-to-text—resulting in 16,000 numbers for every second of speech.

To make these numbers easier to read, the audio is sliced into tiny, overlapping segments of about 25 milliseconds. For each slice, the energy at each pitch is measured. Aligning these slices chronologically creates a spectrogram, which is a visual representation of sound where time runs horizontally, pitch runs vertically, and brightness represents loudness. Many models use a Mel spectrogram, which spaces pitches to mimic human hearing.

The Encoder

A neural network called the encoder analyzes the spectrogram. It examines each moment in context with the surrounding sound and translates it into numbers that describe the sounds heard and how they fit together. Encoders learn these patterns by training on massive datasets of speech paired with written transcripts.

Three Ways to Write the Words

Because the encoder only describes what it hears without writing actual words, another process must translate those descriptions into text. There are three primary methods to achieve this:

• CTC (Connectionist Temporal Classification): The model guesses a letter, word fragment, or blank space at every moment, then merges duplicates and removes blanks. While fast and highly suitable for live applications, it makes guesses independently without considering previously written words.
• Encoder-Decoder: A decoder network writes the text sequentially, one piece at a time. At each step, it uses attention to look back at both the audio and the words it has already written. This method excels at punctuation, spelling, and context, but because it processes audio in larger chunks (up to 30 seconds), it is typically better suited for recordings than live calls.
• Transducer: This approach combines the previous two methods. It processes audio moment-by-moment like CTC, but ensures each new word depends on the text written so far, like a decoder. This allows it to transcribe speech in real time while still utilizing context.

Streaming and Live Transcripts

For live applications like voice agents, speech-to-text systems generate partial (or interim) results—early guesses that can change as more audio is processed. Once a segment of text is finalized, it is marked as final. Voice agents can use partial results to prepare but should only execute actions based on final text.