Latency budgets for natural conversation

What the caller is actually measuring

The number that matters is not model inference time or transcription time. It is the gap between the caller finishing their sentence and hearing the first syllable of a reply. Everything inside that gap is your budget, including the parts you don't control.

Rough thresholds, from listening to a lot of calls rather than from a paper:

• Under about half a second reads as a normal conversation. Nobody remarks on it.
• Around a second is noticeable but tolerable — it reads as someone thinking, which is not a failure.
• Past roughly a second and a half, callers start doing the thing that breaks everything: they assume they weren't heard, and repeat themselves. Now you have two utterances to reconcile and the agent is a turn behind.
• Past three seconds, most callers conclude the line has dropped.

The repeat-themselves threshold is the important one. It is where added latency stops being an annoyance and starts producing actual conversational failures.

Where the time goes

Roughly: detecting that the caller stopped speaking, transcribing what they said, deciding what to do, generating a reply, and synthesising speech. Network transit sits between each one.

End-of-speech detection is the stage most teams under-invest in. Waiting too long adds dead air to every single turn; waiting too little cuts callers off mid-thought. It's the highest-leverage tuning in the whole pipeline, and unlike the others you cannot buy your way out of it — no vendor sells you a better guess about whether a human has finished talking.

It is also the only stage whose cost is paid on every single turn regardless of what the caller said. A 300ms end-of-speech window on a twelve-turn call is three and a half seconds of pure dead air, spent doing nothing.

The stages, and what each one is worth optimising

In rough order of how much return you get per hour spent:

1. 1End-of-speech detection. Highest leverage, no vendor fix, paid every turn. Tune it against your own recordings and your own caller demographics — the right window for a slow, considered speaker is wrong for a rapid one.
2. 2Response streaming. Large, one-off win. Start speaking on the first clause rather than the complete reply.
3. 3Tool calls. Unbounded and invisible until production. This is where the outliers live.
4. 4Speech synthesis. Streaming matters more than raw speed; time-to-first-audio is the metric, not total generation time.
5. 5Transcription. Usually already fast enough on a streaming model. Optimise last.
6. 6The language model itself. Everyone starts here. It is rarely the problem, and switching to a weaker model to save 100ms usually costs you more in extra turns caused by worse answers.

Start speaking before you've finished thinking

Streaming a reply as it's generated, rather than waiting for a complete response, removes a large chunk of perceived delay at no cost to quality. The caller hears the first words while the rest is still being produced.

This is close to free, and it changes what you are optimising. Total generation time stops mattering much; time to the first speakable clause becomes the number. A reply that takes two seconds to finish generating but starts playing after 400ms feels fast, because the caller is listening for the whole of the remaining 1.6 seconds.

There is a trap in it. Streaming means committing to your opening words before knowing how the sentence ends, so a reply that begins "Yes, I can see that booking —" cannot gracefully become "— actually I can't find it." Anything the agent might have to retract shouldn't be streamed ahead of the fact that determines it.

Tool calls are where budgets blow up

Tool calls are where budgets blow up. A CRM lookup that takes two seconds needs to happen while the agent is talking, not in silence after it stops.

The reason this bites so hard is that tool latency is not yours. Your pipeline can be tuned to 600ms and one slow third-party endpoint will put a four-second hole in the middle of a call. Worse, it will do it intermittently — fine in testing, terrible on the afternoon their API is degraded.

Three things help, in order of how much:

• Overlap the call with speech. Acknowledge, then look up while the acknowledgement is playing. "Let me pull that up" is not filler if the lookup genuinely happens during it.
• Fetch early where you can predict. If the flow almost always needs the customer record, request it as soon as you have an identifier, not at the point of use.
• Set a hard timeout with a real fallback. Not a retry — a different sentence. An agent that says "our system is being slow, can I take your number and have someone confirm?" has handled the failure. One that waits has not.

Budget tool calls at their 95th percentile, not their median. The median is what you see in development; the 95th is what your callers get.

Measure the total, at the tail

Instrument the whole round trip per turn — end of caller speech to first byte of outbound audio — and record the per-stage split inside it. Without the split you cannot tell a slow model from a slow lookup. Without the total you will optimise a stage that was never the problem.

Then look at percentiles rather than averages. Average latency hides exactly the calls that go wrong: a pipeline averaging 700ms with a long tail past three seconds produces a meaningful number of conversations where the caller talks over the agent, repeats themselves, or hangs up. Those calls are the ones worth reading, and they are invisible in a mean.

The uncomfortable part of doing this properly is that it usually reveals the slow stage is something you chose for a good reason — a validation step, a compliance check, a lookup that makes the answer correct. Latency work is mostly deciding which of those to move off the critical path, not deleting them.