How Voice Agents Handle Noise and Echo: Noise Cancellation (2026)

Kinds of Noise and Their Impact

To a voice agent, every sound in a room is a candidate for speech. There are five main types of noise:

Steady noise, such as fans, traffic, or hums.
Sudden noise, like doors slamming, dogs barking, or coughing.
Other voices, such as a TV or other people talking, which are the hardest to filter because they resemble the target voice.
Echo, which is the agent's own voice returning through the caller's speaker and microphone.
Distance, where sound bouncing off walls creates reverberation.

Noise disrupts voice agents by causing false interruptions in turn-taking, losing or inserting words in transcripts, and sending misheard names or numbers to backend tools.

The Clean-Up Chain

Clean-up typically occurs in sequential steps:

Echo cancellation: Subtracts the agent's own voice. This works best on the caller's device, close to the microphone.
Noise suppression: Lowers non-speech sounds like fans or traffic.
Voice isolation: Isolates one main speaker and removes background voices. Often, a single model handles both suppression and isolation.

For devices with multiple microphones, beamforming can be used to focus listening in a single direction. On phone calls, noise removal typically runs on the agent's server or within the phone network.

How AI Noise Removal Works

AI models analyze sound using a spectrogram, evaluating thin slices of audio every 10 to 20 milliseconds. The model applies a mask to decide how much of each frequency band to keep, preserving speech while turning down noise. Because live calls require real-time processing, models can only look ahead by 10 to 20 milliseconds. Open-source examples of these models include DeepFilterNet and RNNoise.

Machine vs. Human Hearing

Audio that sounds cleaner to human ears is not always easier for speech-to-text systems. Cleaning processes introduce artifacts (distortions) that can degrade machine recognition. Studies show that original noisy audio can sometimes outperform cleaned audio in speech-to-text accuracy. Consequently, developers should gradually adjust clean-up strength and measure word error rates on their own calls rather than relying solely on human perception.

Integration and Best Practices

Vendors differ on where to place clean-up in the pipeline. For example, AI-coustics runs voice detection on the original audio side-by-side with clean-up, while Krisp places turn-taking models after voice isolation. AssemblyAI suggests using cleaned audio for voice detection but original audio for transcription.

Two key rules are widely agreed upon:

Do not stack multiple noise removers on the same audio stream.
Phone audio is narrow (sampled at 8 kHz), so developers should use telephony-specific models.

Far-Field Audio and Reverberation

When a microphone is far from the speaker, it captures delayed copies of the voice bouncing off walls, known as reverberation. To measure performance in these environments, the Far-field Speech Recognition Leaderboard evaluates models by placing clean speech into simulated furnished rooms with added noise. Testing models specifically for far-field conditions is crucial if the physical microphone is positioned far from the user.