Noise and Echo in Voice Agents, Explained

Kinds of Noise and Their Impact

To a voice agent, every sound in a room is a candidate for speech. There are five main types of noise:

• Steady noise, such as fans, traffic, or hums.
• Sudden noise, like doors slamming, dogs barking, or coughing.
• Other voices, such as a TV or other people talking, which are the hardest to filter because they resemble the target voice.
• Echo, which is the agent's own voice returning through the caller's speaker and microphone.
• Distance, where sound bouncing off walls creates reverberation.

Noise disrupts voice agents by causing false interruptions in turn-taking, losing or inserting words in transcripts, and sending misheard names or numbers to backend tools.

The Clean-Up Chain

Clean-up typically occurs in sequential steps:

1. Echo cancellation: Subtracts the agent's own voice. This works best on the caller's device, close to the microphone.
2. Noise suppression: Lowers non-speech sounds like fans or traffic.
3. Voice isolation: Isolates one main speaker and removes background voices. Often, a single model handles both suppression and isolation.

For devices with multiple microphones, beamforming can be used to focus listening in a single direction. On phone calls, noise removal typically runs on the agent's server or within the phone network.

How AI Noise Removal Works

AI models analyze sound using a spectrogram, evaluating thin slices of audio every 10 to 20 milliseconds. The model applies a mask to decide how much of each frequency band to keep, preserving speech while turning down noise. Because live calls require real-time processing, models can only look ahead by 10 to 20 milliseconds. Open-source examples of these models include DeepFilterNet and RNNoise.

Machine vs. Human Hearing

Audio that sounds cleaner to human ears is not always easier for speech-to-text systems. Cleaning processes introduce artifacts (distortions) that can degrade machine recognition. Studies show that original noisy audio can sometimes outperform cleaned audio in speech-to-text accuracy. Consequently, developers should gradually adjust clean-up strength and measure word error rates on their own calls rather than relying solely on human perception.

Integration and Best Practices

Vendors differ on where to place clean-up in the pipeline. For example, AI-coustics runs voice detection on the original audio side-by-side with clean-up, while Krisp places turn-taking models after voice isolation. AssemblyAI suggests using cleaned audio for voice detection but original audio for transcription.

Two key rules are widely agreed upon:

• Do not stack multiple noise removers on the same audio stream.
• Phone audio is narrow (sampled at 8 kHz), so developers should use telephony-specific models.

Far-Field Audio and Reverberation

When a microphone is far from the speaker, it captures delayed copies of the voice bouncing off walls, known as reverberation. To measure performance in these environments, the Far-field Speech Recognition Leaderboard evaluates models by placing clean speech into simulated furnished rooms with added noise. Testing models specifically for far-field conditions is crucial if the physical microphone is positioned far from the user.