The problem
The voice agent demo was genuinely impressive. Natural, fast, handled the script. Three weeks after launch your customers are talking over it, it is mishearing names, and some calls just end.
I see this pattern constantly with voice, more than with any other kind of AI system. The gap between a working demo and a reliable phone system is enormous, and almost nobody scopes for it.
After 4 years building these, here is what actually separates the deployments that survive.
Latency is the whole experience
Most voice agents in the wild run 2 to 3 seconds of round trip delay. The caller finishes talking, waits, then hears the agent start.
Two seconds does not sound like much written down. On a phone call it is the difference between a conversation and an interrogation. Humans expect a response gap of roughly 200 milliseconds. Anything past about 800 and the caller starts to feel that something is wrong, even if they cannot name it.
And here is the part that catches people out: vendor latency numbers are frequently measured on one segment of the pipeline, not the full end to end delay a caller actually experiences. The impressive number in the sales deck is often the model inference time alone, excluding speech recognition, network hops, text to speech and telephony.
Ask specifically for end to end, caller-finishes-speaking to caller-hears-audio, measured on your actual telephony path. The number usually changes.
What breaks once real calls start
The demo was one call, on good network, with a cooperative speaker reading a happy path.
Production is:
None of these are exotic. They are Tuesday. But none of them appear in a demo.
The architecture question that decides it
Voice systems get assembled from parts: speech to text, an LLM, text to speech, telephony. It is tempting to pick the best component in each category.
In production, that modular approach is where latency, state and integration failures compound. Every boundary adds delay and another thing that can drop. Four components with four network hops is four places for the call to die.
The systems I have seen hold up minimize hops aggressively, use streaming everywhere so processing starts before the caller finishes speaking, and keep conversational state somewhere that survives a dropped connection.
That last one matters more than people expect. If state lives only in the socket, a blip means the caller has to start over, and they will not. They will hang up and call a competitor.
What I build in from the start
A scoped job, not an open conversation
The voice agents that work do one narrow thing well. Book, qualify, route, confirm, take a simple order.
Open ended conversational agents on the phone are still a bad bet for most businesses. The failure modes are too visible and the caller has no patience.
A fast, obvious path to a human
Not buried. Not after four failed attempts. If the agent is not confident, it hands off, and the handoff carries the context so the customer does not repeat themselves.
An agent that transfers 20% of calls cleanly is worth far more than one that tries to handle 100% and infuriates a quarter of them.
Interruption handling as a requirement
People interrupt. If the agent cannot stop talking mid sentence and listen, it will feel broken regardless of how good the rest is. This is a non negotiable in scoping, not a nice to have.
Load testing before launch
Run concurrent calls at your realistic peak, not your average. Latency behaviour under load is completely different from single call behaviour, and peak is exactly when you can least afford it.
Recording and review in week one
Listen to real calls in the first week. Not metrics, actual audio. You will hear problems in twenty minutes that no dashboard will surface for a month.
The test before you sign
Ask the vendor or builder for a recording of a real customer call with a real problem, not a scripted demo. Ask for the end to end latency on your telephony path. Ask what happens when the socket drops.
If you get a demo instead of answers, you are looking at a prototype.
Voice is genuinely valuable when it is scoped tightly and engineered for the ugly cases. It is also the AI deployment most likely to embarrass you in front of customers, because unlike a bad chatbot reply, a bad call is something a person experiences in real time and remembers.