What is voice latency?
The gap between a caller finishing a sentence and the agent starting its reply. It is the single number that decides whether a call feels natural.
Voice latency — Voice latency is the delay between a caller finishing a sentence and the agent beginning its reply, summing speech recognition, model response and speech synthesis. Under about 800 milliseconds feels natural; over 1.5 seconds feels broken.
Also known as: turn latency
Why it matters
Human conversation has a rhythm: people reply within a few hundred milliseconds of each other, and a longer gap reads as hesitation, distraction or a bad line. Voice latency is the sum of every step a voice agent takes between hearing the last word and speaking the first word of its reply, transcription, model reasoning, and speech synthesis, plus whatever the phone network itself adds. Under roughly 800 milliseconds, the pause reads as normal conversational rhythm. Past about 1.5 seconds, callers start talking again, assuming the line dropped or the agent missed them.
Latency is also where the three underlying systems fight each other. A larger, smarter language model takes longer to respond; a more expressive text-to-speech voice can take longer to start streaming audio. Every choice made to improve accuracy or voice quality has to be weighed against what it costs in the time before the caller hears anything.
Where the delay comes from
- 1
Network and telephony
The call itself, carrier and region, adds time before any AI component sees a word.
- 2
Speech-to-text
Transcribing the caller's audio to text the model can read.
- 3
Model response
The language model reasons over the conversation and produces its reply.
- 4
Text-to-speech
Converting the reply to audio, ideally streaming the first words before the whole sentence is generated.
- 5
The caller hears it
Total elapsed time is what the caller experiences as the agent's response speed.
The mistake to watch for
Questions
- How is voice latency different from time to first token?
- Time to first token measures only how quickly a language model starts producing its response. Voice latency is the full round trip a caller experiences: transcription, model response and speech synthesis together, plus the telephony leg, so it is always higher than time to first token alone.
- What voice latency counts as acceptable?
- Under roughly 800 milliseconds tends to read as a natural conversational pause. Past about 1.5 seconds, callers commonly start speaking again, assuming the agent missed them or the call dropped.
- Does a bigger language model always mean higher latency?
- Usually, yes, though not always by much. Larger models generally take longer to produce a response, which is one reason voice agent platforms often default to smaller or faster models for the parts of a call where speed matters most.