To test an AI voice agent, run calls through the whole call path before it answers a single customer: the phone line, speech recognition, every branch of the flow, the systems it reads from and writes to, and the handover to a person. Write down the expected result for each test call, compare what actually happened, fix, and run the set again. Then repeat the same set after every change to the flow, because a change to one step can break another. A demo in the browser proves the idea. Only test calls over a real phone line prove the agent is ready for your callers.

In short: what a test plan covers

• The full call path, not just the conversation: audio, recognition, flow logic, backend, speech output and handover.
• A test set built from real call reasons, with the edge cases written in on purpose.
• A written pass criterion per test call, checked in the backend as well as by ear.
• Regression tests after every change, in a test environment, before the change reaches callers.
• A watch period after go-live, where completed, transferred and dropped calls per step tell you what the tests missed.

Why testing a voice agent is different from testing a chatbot

A chatbot receives clean text. A voice agent receives a phone call, and most of what goes wrong happens before the language model sees a word.

• Phone audio is narrowband. Google’s Speech-to-Text documentation notes that “most legacy telephony audio, for example, use sample rates of 8000 Hz, which may give less accurate results”, and advises sending audio at its original sample rate rather than resampling it (Google Cloud). A test in the browser, with a good microphone, hides that.
• Callers pause, interrupt and talk over the agent. The system has to decide when a caller has started and stopped speaking. Speech services expose this as voice activity events, with timeouts for when speech never starts or ends (Google Cloud). Whether your flow handles a long pause or a caller who interrupts is something you only see on a call.
• Silence is noticeable. For network delay alone, the ITU guidance quoted by Cisco treats up to 150 ms one way as acceptable for most applications (Cisco, on ITU-T G.114). Recognition, reasoning and speech synthesis add their own time on top. Why that matters is covered in latency in AI voice agents.
• The agent acts on live data. It verifies callers, looks up records and writes results back. A call that sounded perfect but changed the wrong record has failed.

If you are still choosing between a text and a voice channel, AI voice agent vs chatbot sets out the differences.

What to test: six layers of the call path

Layer	What goes wrong	How to test it	Passes when
Phone line and audio	Echo, clipped first words, poor quality on mobile	Call in from a landline, a mobile with weak signal and a noisy room	The first word of the greeting and of the caller is never lost
Speech recognition	Names, numbers, postcodes, dates and licence plates misheard	Read the hard inputs aloud, in different accents and speeds	The value the agent confirms matches what the caller said
Intent and flow	Wrong reason recognised, a branch that loops or dead-ends	One test call per branch, plus callers who change topic halfway	Every branch reaches its intended end state
Backend and CRM	Wrong record found, nothing written back, timeouts	Test records in a test environment, including a record that does not exist	The backend shows exactly the change the call should have made
Handover	Context lost, transferred to the wrong team, caller repeats everything	Ask for a person at different steps, and trigger the fallback paths	The colleague sees who is calling and why before saying hello
Speech output and timing	Long silences, wrong pronunciation, talking over the caller	Listen to recorded test calls end to end	No silence a caller would notice, every name and number spoken correctly

The hard inputs deserve their own tests. How voice agents handle them is covered in posts on address validation, licence plate recognition, dates and verifying callers.

Build the test set from real calls

The best test set starts from the reasons customers call today, not from the flow diagram.

1. List your top call reasons and the way callers phrase them, including the detours and half sentences. Check with your privacy officer which recordings and transcripts you may use for this.
2. Write a happy path for each reason, then the edge cases: an unknown customer number, two questions in one call, a caller who corrects themselves, silence, “I want to speak to someone”, a backend that does not answer.
3. Use test data, not production records. Create test customers in a test environment, so a failed test cannot change a real account.
4. Write the expected result next to each test call: which branch, which value captured, what written to which system, and how the call ends.
5. Have customer service colleagues call in. They know how customers talk and will find the places where the flow chafes faster than the people who built it.

Measure recognition accuracy where it matters

For speech recognition there is a standard measure: word error rate. Microsoft’s documentation defines it as the number of inserted, deleted and substituted words divided by the number of words in a human-labeled transcript, and describes a WER of 5 to 10% as good quality, 20% as acceptable and 30% or more as poor. To measure it you need audio plus human-labeled transcripts, and Microsoft suggests 30 minutes to 5 hours of representative audio (Microsoft Learn).

For a voice agent, the overall WER matters less than the accuracy on the words the flow depends on. A misheard filler word costs nothing. A misheard digit in a policy number sends the call down the wrong path. Score the values the agent confirms back to the caller separately.

Pass criteria: decide before you test

A test without a written expected result becomes an opinion. For each test call, record:

• Outcome: completed by the agent, transferred to a person, or dropped. These are the same three outcomes you will track once the agent is live.
• Path: did the call follow the branch it should have?
• Data: is the captured value correct, and is the change in the backend correct?
• Handover: if transferred, did the context arrive with the right team?
• Experience: any silence, repetition or mispronunciation a caller would notice?

Decide in advance what has to pass before go-live. Most teams treat a wrong backend write or a lost handover as a blocker, and a slightly awkward phrasing as something to fix in the next release.

Test again after every change

Call reasons change, and so do flows. A new branch for one call reason can change how the agent recognises another. That is why the same test set runs again after every change:

1. Make the change in a test environment, never directly on the live line.
2. Run the full regression set, not only the tests for the step you changed.
3. Compare against the last approved run. A test that passed before and fails now is the first thing to look at.
4. Approve, then put it live, and know how you would go back to the previous version if the live numbers say otherwise.

How a release process for voice agents can work, with versions and a way back, is described in AI agent version control.

Compliance checks belong in the test set

• Check your opening message (AI disclosure, call recording notice) with your legal or privacy team before go-live.

After go-live: let real calls show what the tests missed

No test set is complete. Start with one call reason or a share of callers, and widen when the numbers hold. In the first weeks, look per step of the flow at how many calls were completed, transferred and dropped: the step where callers hang up or ask for a person is where your test set had a gap. Add those calls to the regression set. How to read these numbers is explained in how to measure an AI voice agent.

Frequently asked questions

How do you test an AI voice agent? Call it over a real phone line with a test set built from your actual call reasons. For each test call, write down the expected branch, the values to capture, the change in the backend and how the call should end, then compare with what happened. Cover the happy path and the edge cases for every step, and repeat the set after every change.

How many test calls do you need before go-live? Count coverage, not calls. Every branch of the flow needs at least one happy path and its edge cases, every hard input (names, numbers, dates) needs tests in different voices, and every route to a person needs a handover test. For recognition accuracy, Microsoft suggests 30 minutes to 5 hours of representative audio with human-labeled transcripts.

Can you test a voice agent without real customer data? Yes, and you should. Use test customers in a test environment, so a failed test cannot change a real account. Real call recordings are useful for building realistic test phrases; check with your privacy officer which recordings and transcripts you may use.

What do you need to retest after changing a flow? The whole regression set, not only the changed step. A new branch or prompt can change how the agent recognises other call reasons, so compare every result with the last approved run before the change goes live.

What is the difference between testing and monitoring a voice agent? Testing happens before callers are affected, with calls you designed and expected results you wrote down. Monitoring happens after go-live, on real calls: completed, transferred and dropped per step of the flow. Monitoring shows the cases your test set missed, and those calls go back into the test set.

How teams test on AssistYou

AssistYou is the platform customer service teams use to build reliable AI voice agents, and our team helps you set it up. Your team designs and adjusts the flow in the Flow Designer and sees every call in the portal, and every change is tested before it goes live. After go-live, Agent Statistics shows how many calls the AI Voice Agent completed, transferred to a person or dropped, with drop-offs per step directly on the flow in the Flow Designer. See the AI Voice Agent product page, or contact our sales team to walk through a test plan for your own call reasons.

← Back to blog