Engineering
Latency budgets for natural conversation
Under 500ms feels like a conversation. Over a second feels like a hold queue. A walk through where the time actually goes, and which parts are worth optimising.
Conversational latency is a budget, not a number. Every stage in the round trip spends part of it, and the caller only experiences the total.
Writing the budget down per stage makes the tradeoffs concrete and stops teams from optimising the stage that was never the bottleneck.
Where the time goes
Roughly: detecting that the caller stopped speaking, transcribing what they said, deciding what to do, generating a reply, and synthesising speech. Network transit sits between each one.
End-of-speech detection is the stage most teams under-invest in. Waiting too long adds dead air to every single turn; waiting too little cuts callers off mid-thought. It's the highest-leverage tuning in the whole pipeline.
Start speaking before you've finished thinking
Streaming a reply as it's generated, rather than waiting for a complete response, removes a large chunk of perceived delay at no cost to quality. The caller hears the first words while the rest is still being produced.
Tool calls are where budgets blow up. A CRM lookup that takes two seconds needs to happen while the agent is talking, not in silence after it stops.
