How to Test a Voice Agent: Simulated Callers and Evals (2026)

Why Testing Voice Agents is Difficult

Testing a voice agent is more challenging than testing a website because language models are non-deterministic, meaning the same input can yield different answers. Additionally, voice agents rely on multiple moving parts—such as speech-to-text, the language model, the voice, the network, and turn-taking—any of which can fail. Because conversations branch when users interrupt, change their minds, or go off-script, simply calling an agent a few times is insufficient for a complete test plan.

A Layered Testing Approach

• Test the individual parts: Start by testing the language model using text transcripts rather than audio, which is faster and cheaper. This verifies if the model calls the correct tools with the right details. Speech-to-text should be tested on real call recordings to ensure names and numbers are captured accurately, and the voice output should be checked for correct pronunciation of product names using pronunciation dictionaries.
• Use simulated callers: Test entire conversations by using another AI to simulate a caller with a specific persona and goal (e.g., a hurried caller on a noisy phone line). These simulations can be run by voice to test the entire system or by text to test the model. While hundreds of these tests can run simultaneously, simulated callers are often too cooperative, so it is important to mix in real call recordings.

Key Metrics and Evaluation

Testing should focus on task success by verifying in the internal system—not just what the agent says—that the action was completed correctly. Other critical metrics include latency, cut-offs, missed interruptions, and whole-call signals like how often callers ask for a human, hang up early, or repeat themselves.

To evaluate hundreds of transcripts, developers can use a language model as a judge to grade calls against a checklist. Because the AI judge can make mistakes, its performance should be periodically verified against human grading, and hard database checks should be used whenever possible.

Regression Testing and Monitoring

Every failed call should be turned into a new test case to prevent future regressions. The entire test suite should be run whenever there is a change to the prompt, model, voice, or provider. Because providers update their models, developers should pin exact model versions and run tests on a regular schedule, such as every night.

After launching, developers must continuously monitor real calls using dashboards to track task success, slow turns, and transfers. Listening to a sample of real calls weekly helps identify new failures, which should then be fed back into the test suite as new test cases.