Designing voice agents that survive interruptions

Why turn-taking is harder than it sounds

Human conversation does not alternate cleanly. Two people overlap constantly, and most of that overlap is not an attempt to take the floor — it is acknowledgement, agreement, or thinking aloud. Linguists separate these, and the distinction turns out to be the whole engineering problem:

• Backchannels — "mm-hm", "right", "okay", "yeah". The caller is signalling that they're still listening and want you to keep going. Stopping here is actively wrong.
• Barge-in — a genuine attempt to take the floor, usually because the agent has said something wrong, is going too slowly, or has asked something already answered.
• False starts — the caller begins speaking, then abandons it. Half a word, then silence.
• Environmental noise — a door, a cough, a television, another conversation in the room, a car horn.

Detect intent to interrupt, not just sound

Naive barge-in triggers on any incoming audio above a threshold, which means a cough or a door closing stops the agent mid-sentence. Callers experience this as an agent that keeps flinching. We gate on speech-likeness and a short duration threshold before yielding the turn, so background noise doesn't derail a sentence the caller wanted to hear.

Duration is the cheapest signal that separates a backchannel from a real interruption. "Mm-hm" is short and falls inside the window. "No, that's not what I — " runs past it. Tuning that window is the single highest-leverage adjustment in the whole behaviour: too short and the agent talks over people who are trying to correct it, too long and every "okay" derails it.

The window is not one number for every deployment either. A caller on a hands-free car speaker in traffic needs a more conservative threshold than someone on a desk phone in a quiet office, because the noise floor is doing more of the work. Deployments that skew mobile want different tuning from ones that don't.

A useful rule of thumb: if you can't decide whether something is a backchannel or a barge-in, the caller usually can't either. Resuming is the recoverable mistake; stopping dead is not.

Yield fast, then decide

When speech does clear the gate, the agent stops talking immediately. That part is not a judgement call — a caller who is speaking and can still hear the agent's voice will assume they weren't heard and start again louder, and the call degrades from there.

But stopping playback is not the same as abandoning the turn. The agent keeps what it was about to say. If the interjection resolves into a filler word or a false start, it resumes from where it left off rather than restarting the sentence or, worse, moving on as though it had finished. Restarting a long sentence from the top is its own kind of failure: the caller hears the same opening clause three times and concludes the thing is stuck in a loop.

Practically this means three pieces of state have to survive the interruption: what has already been spoken, what remains unspoken, and whether the remainder is still relevant given what the caller just said. The third is a model decision. The first two are bookkeeping, and getting them wrong produces most of the "it repeats itself" complaints.

Keep state outside the sentence

The facts an agent has collected — the policy number, the address, the appointment slot — live in structured state, not in the transcript it happens to be reading. An interruption discards the phrasing, never the progress.

This is what lets a caller say "actually, make it Thursday" four turns later and have the agent update one field instead of re-running the whole booking flow. The correction touches the slot and nothing else; the name, the address and the service type stay exactly as captured.

Agents that keep their state implicitly in the conversation history cannot do this. Ask one to change a detail from four turns ago and it will often re-elicit everything after it, because the only record of what it knows is a transcript it has to re-read. That is the difference between an agent that handles a correction and one that punishes you for making one.

It also changes what a dropped call costs. When state is structured and written as it is gathered, a call that disconnects mid-flow leaves behind everything the caller had already given you rather than a half-finished transcript nobody will read.

The failure modes worth testing for

Most interruption bugs do not show up in a scripted test, because whoever writes the script unconsciously waits their turn. These are the ones we actively replay:

• The caller answers the next question before it is asked — "and my postcode is…" — and the agent asks for it anyway.
• The caller says "no, wait" mid-confirmation and the agent completes the booking regardless.
• A backchannel resets a long explanation to its first clause.
• The caller goes silent for four seconds mid-sentence and the agent treats the pause as end-of-turn, then answers half a question.
• Two people in the room talk to each other and the agent tries to participate.
• A correction lands on the field the agent has already moved past, and the update silently doesn't apply.

Every one of these is in our recordings, which is where we found them. None of them were in the test plan first.

What this buys you on a real line

Handling interruption well doesn't announce itself. Callers don't notice the mechanism — they notice that they didn't have to fight the thing, and that repeating themselves wasn't necessary. On inbound lines that shows up as fewer abandoned calls partway through a flow. On support calls it shows up as fewer transfers requested out of frustration rather than genuine need.

It is also the difference between a demo that impresses and a deployment that survives its first week. The messy call is not the exception in production. It is the volume.