Building a voicebot that actually understands customers on the phone means getting five things right: speech recognition (ASR) that copes with 8 kHz telephone audio, language understanding (NLU) that identifies what the caller wants, a response time that feels like a conversation, a connection to your CRM and backend so the voicebot can resolve something, and a release process in which every flow is tested before it takes a live call. The model is rarely the problem. The chain around it decides whether your voicebot works.

This article is the build guide. If you first want to know what a voicebot is and how it differs from a menu system, read what is a voicebot.

The call path: what happens between “hello” and the answer

Every turn in a phone conversation passes through the same chain. If you are deploying a voicebot for a business, you need to know where each link can fail.

1. Voice activity detection (VAD). Decides when the caller starts and stops speaking. Cut off too early and you lose half a postcode. Too late and the caller hears silence.
2. Automatic speech recognition (ASR). Turns speech into text. This is where the phone channel’s biggest trap sits (see below).
3. Language understanding (NLU or a language model). Extracts what the caller wants (the intent) and the details that belong to it, such as a customer number, a date or an address.
4. Dialogue and logic. Decides the next step: ask a question, look something up in a system, confirm, or transfer.
5. Text-to-speech (TTS). Turns the answer into speech and plays it.

End-to-end audio models also exist, combining recognition, understanding and response in a single model. They sound fluent, but you lose sight of the intermediate steps: you see less clearly what was heard, why a decision was made and which data was used. In a contact center where a call ends in a change to a customer record, that visibility is not optional.

Why telephone audio makes speech recognition hard

Most telephony delivers audio at a sample rate of 8,000 Hz. Google notes that “most legacy telephony audio” uses that rate, “which may give less accurate results” (Google Cloud). Microsoft measures speech recognition in word error rate (WER), “the industry standard for measuring model accuracy”, and rates contact center audio among the hardest scenarios: “Low, 8 kHz, could be two people on one audio channel, could be compressed” (Microsoft Learn). For call centers Microsoft expects a WER below 30%, against sometimes below 10% for a voice assistant on 16 kHz audio.

In practice, “fifteen” and “fifty” or “F” and “S” sound almost identical over the phone. A voicebot also has to catch postcodes, house numbers with suffixes and product names. Four techniques help:

• Give the ASR context. If the voicebot just asked for a postcode, recognition expects that format. That sharply reduces errors.
• Run several ASR engines in parallel. Different engines make different mistakes. Comparing their output lets you pick the most likely answer each time.
• Apply logical correction to transcripts. A postcode in the wrong format does not exist. Neither does a date of birth in the future. Check what is possible before you use it.
• Use confidence thresholds. When recognition is uncertain, have the voicebot confirm: “I have 3013 AK, is that right?”

How this works for addresses is covered in address validation with AI Voice Agents.

Latency: budget your response time per step

A conversation feels natural as long as the pause before an answer stays short. For network delay alone, ITU recommendation G.114 treats one-way delay up to 150 ms as “acceptable for most user applications” and above 400 ms as “unacceptable for general network planning purposes” (Cisco on ITU-T G.114). Everything your voicebot does comes on top of that.

So treat latency as a budget you spread across the chain:

• End-of-turn detection. How long does the VAD wait after the last sound before it closes the turn?
• Recognition. Does the ASR stream, or wait for the sentence to finish?
• Understanding and logic. How long does the language model take, and how long the lookup in your CRM or backend?
• Speech. Does TTS start playing while the rest of the sentence is still being generated?

Measure the time between the end of the caller’s speech and the voicebot’s first audible word, per step and on real calls. An average hides the slow outliers, so look at the slowest calls too. If a system lookup takes longer, have the voicebot say so (“let me pull up your details”) rather than go quiet. A layer-by-layer deep dive is in latency in AI voice agents.

Designing intents and flows

A voicebot that tries to do everything at once does nothing well. Start with a limited set of intents that together cover a large share of your calls, and design a flow for each with a clear end. In AssistYou you design those flows in the Flow Designer.

• Keep intents small and distinct. “Change address” and “report a move” are the same thing to a caller. Merge them. “Request an invoice” and “dispute an invoice” are not. Keep them apart.
• Only ask for what you need. Every detail the voicebot asks for costs time and adds a chance of a recognition error.
• Confirm what matters. An amount, a date, an IBAN or a cancellation is always confirmed before it is processed.
• Design the exit first. What happens when the voicebot does not understand, when the caller gets angry or when a system does not respond? After two failed attempts the caller reaches an agent, with the details already collected.
• Let the caller interrupt. People talk over a voicebot. A flow that cannot handle that feels like a menu.

The handoff to an agent is a design question of its own. What the colleague needs to see is covered in designing the handoff. If you are rebuilding an existing menu, start with moving from IVR to AI voice agents.

Writing back to your CRM and backend

A voicebot that only transfers calls saves little. The value is in calls the voicebot finishes itself, and for that it has to read and write data in the systems you already run. AssistYou’s published customer stories show what that looks like:

• Carglass verifies callers against data in Salesforce CRM (Carglass).
• Unigarant has the AI Voice Agent handle identity verification, retrieve the customer file and complete whole processes, such as cancelling a policy (Unigarant).
• Trevvel connects the AI Voice Agent to Google Maps, so a caller can say “Erasmus MC” without a full address (Trevvel).

A few rules make write-back reliable:

• Read first, write after. Fetch the record after verification, so the voicebot knows what the call is about.
• Confirm only after success. Say “your address has been changed” only once the backend has confirmed the change.
• Plan for a system that does not answer. A timeout should end at an agent or a callback, not in silence.
• Log every action. Which data was read, what was changed, and in which call. You need that for complaints and for every audit.

More on connections in integrating AI agents with your CRM.

Versioning and the path from test to production

A voicebot is never finished. Opening hours change, a new product brings new questions, a line in a flow turns out to confuse callers. Every change touches live calls. This is what a platform should give you before you deploy a voicebot on it:

• Every change as a version. You should be able to see which version of a flow was live at a given moment and what changed compared with the previous one.
• A test environment separate from production. You build and test changes in an acceptance environment (UAT). Callers notice nothing.
• Promotion of an approved version. Production receives exactly the version that was tested, not a copy that was edited afterwards.
• One-step rollback. If something goes wrong, you restore the previous version without waiting for a new release.
• A change log. Who changed what, when and why. In regulated sectors that is not a nice-to-have.

How such a pipeline fits together is covered in version control for AI agents.

Testing a flow before it takes a live call

A demo on a laptop says little about how a flow performs with real callers on the phone. Testing properly means:

1. Test over the phone, not in the browser. Only then do you hear the effect of 8 kHz audio, background noise and a weak mobile connection.
2. Build a test set from real call reasons. Use the way customers actually phrase their question, including detours, accents and half-finished sentences. Check with your privacy officer which recordings and transcripts you may use for this.
3. Provoke the edge cases on purpose. An unknown customer number, a caller who switches topic halfway, a system that does not respond, someone who says “agent” straight away.
4. Check what happened in the backend. A call that sounded fine but changed the wrong record is a failed test.
5. Have customer service colleagues call in. They know how customers talk and where the flow chafes.
6. Start with part of the traffic. Put the flow live for one call reason or a share of callers first, and expand when the numbers hold.

Measuring after go-live

After go-live the real work starts. Agree beforehand which numbers you track, so you can compare each new version with the previous one:

• Fully resolved. What share of calls does the voicebot finish without an agent?
• Correctly routed. Does a transferred call reach the right team?
• Drop-off per step. At which question in the flow do callers hang up or ask for a person?
• Repeat calls. Does the same customer call back about the same question within a few days?
• Cost per contact. What does a resolved call cost now, compared with before? See what is cost per contact.

Also listen back to a sample of calls every week. Numbers tell you where a flow chafes, calls tell you why. Why a voicebot without analysis is a risk is covered in AI voice agents without analytics.

Frequently asked questions

What do you need to build a voicebot? Speech recognition that handles telephone audio, language understanding that identifies intents and details, text-to-speech, flows with a clear exit to an agent, a connection to your CRM and backend, and a release process with versions, a test environment and a way to roll back.

How fast does a voicebot need to respond? Fast enough to feel like a conversation. For network delay alone, ITU-T G.114 treats up to 150 ms one-way delay as acceptable for most applications. Everything the voicebot does comes on top of that. So measure the time between the end of the caller’s speech and the voicebot’s first word, per step in the chain.

Why does a voicebot understand less on the phone than in a demo? Because telephone audio is usually 8 kHz, against 16 kHz or more from a microphone. Microsoft rates contact center audio among the hardest scenarios for speech recognition. Giving context, running several ASR engines, logical correction and confirming when unsure make the difference.

Can a voicebot update data in our CRM? Yes, when it is connected to your CRM or backend. Carglass verifies callers with data from Salesforce CRM, and at Unigarant the AI Voice Agent completes processes such as cancelling a policy. Confirm a change to the caller only once the backend has processed it.

How do you test a voicebot before it goes live? Over the phone, with a test set built from real call reasons, with edge cases you provoke on purpose, and with a check of what happened in the backend. Then put the flow live for part of the traffic and expand when the numbers hold.

Deploying a voicebot in your contact center

AssistYou’s AI Voice Agent answers, verifies and routes calls in English, Dutch and German, for contact centers in energy, insurance, mobility and banking, among other sectors. See how it works on the AI Voice Agent product page and how pricing is structured on the pricing page.

Want to know which of your calls a voicebot can take over first? Contact sales.

← Back to blog