Voice AI conversation repair: recovering a misheard turn

A caller reads out a postcode, the agent hears one character wrong, and instead of noticing, it quietly books an engineer to the wrong address. Or the reverse: the caller says something perfectly clearly, and the agent replies "sorry, I did not catch that" three times in a row until the caller sighs and asks for a human. Both failures come from the same gap. The agent misheard a turn, and it had no graceful way to recover.

Every enterprise voice AI deployment eventually meets this moment. In McKinsey's State of AI (November 2025) about 88% of organisations report using AI somewhere, yet only around 6% capture material earnings impact, and a surprising amount of that gap lives in the unglamorous handling of the turns that go wrong. A voice agent that understands nine turns out of ten still fails the caller on the tenth if it cannot repair. Conversation repair is the design discipline that decides what happens on that tenth turn, and it is what separates a usable agent from a frustrating one.

This guide is written for the people who own that outcome: the operations and CX leaders deciding whether a voice AI agent is ready for real callers, and the engineers who have to build the recovery behaviour underneath it. We will cover how an agent detects it has misheard, the difference between explicit and implicit confirmation, how to re-prompt without a dead-end loop, when to stop repairing and escalate, and where the compliance line sits when a misheard value gets written into a system of record.

This guide is shipped by the team behind Dilr Voice, enterprise voice AI built for regulated deployments. Or see DATS, our five-stage AI consulting system.

What is conversation repair in a voice AI agent?

Conversation repair is how a voice agent recovers after it has misheard or misunderstood a turn, rather than pushing ahead on a wrong reading. Dilr Voice treats repair as a distinct layer: detect a low-confidence turn, decide whether to confirm or re-prompt, re-ground the caller, and escalate if recovery fails. The term comes from conversation analysis, where repair is the ordinary machinery people use to fix trouble in speaking, hearing and understanding.

The idea is not new. In their 1977 paper in the journal Language, Schegloff, Jefferson and Sacks described the organisation of repair in conversation and showed a strong human preference for self-correction over being corrected by the other party. A good voice agent borrows that instinct: it should notice its own trouble and invite a clean redo, instead of forcing the caller to interrupt and fight it.

It helps to say what this post is not about, so the boundaries are clear. Measuring whether the agent understood a turn at all is a separate job, covered in our guide to voice AI natural language understanding. Knowing when a caller has finished speaking is endpointing and turn-taking. Getting a specific alphanumeric reference or account number right, with read-back and phonetic disambiguation, is its own discipline in our reference number capture guide. This post starts one step later: a turn has already been misheard, and the question is how the agent recovers.

Why will a voice agent always mishear some turns?

Because misrecognition is a property of speech, not a bug you can patch out. Even at the research frontier, Microsoft reported a 5.1% word error rate on the Switchboard telephone benchmark in 2017, roughly matching the 5.9% error rate of professional human transcribers. That is a curated benchmark. On real, un-scripted calls the rate is materially higher, so a Dilr Voice deployment has to assume some turns will be heard wrong.

Switchboard is conversational telephone speech, which is already harder than clean dictation, and it is still a tidy dataset with known speakers and reviewed transcripts. Live enterprise calls add everything the benchmark strips out: background noise, strong regional accents, cross-talk, hesitations, and callers who change their mind mid-sentence. Two related guides go deeper on the inputs that push the error rate up, on accent and dialect recognition accuracy and on the way a caller experiences a stalled turn in our note on dead air and latency masking.

The practical consequence is a design one. If misrecognition is inevitable, then repair is not an edge case to bolt on at the end, it is a core behaviour to design from the first sprint. An agent evaluated only on its accuracy on a happy-path script will look excellent in the demo and collapse in week one of production, when the tenth turn finally goes wrong and there is no recovery path underneath it.

How does a voice AI agent detect that it has misheard?

The agent detects trouble through a confidence gate: a rule that reads the recogniser's own uncertainty before it acts. Modern platforms expose this directly. Amazon Lex returns both a transcription confidence and an intent confidence score between 0.0 and 1.0, plus up to four alternative intents, so Dilr Voice can check how sure the system is before it commits to a reading and, when it is unsure, choose to confirm rather than guess.

The critical nuance is that a confidence score is not a probability of being correct. Amazon's own documentation is blunt about it: "The confidence scores that Amazon Lex V2 returns are comparative values. You should not rely on them as an absolute score." In other words, the number tells you the top reading beats the runner-up, not that it is right. Two readings scoring 0.75 and 0.72 signal genuine ambiguity that no single threshold resolves, which is exactly why the threshold is a design decision your team owns, not a value the vendor can set for you.

Good detection therefore reads more than one signal. It looks at the gap between the best and second-best interpretation, whether the uncertain part is the whole intent or a single slot, and how costly acting on a wrong reading would be. A weak reading of a chatty aside can be let go. A weak reading of a payment amount, a date or an identity check cannot. The table below shows the kind of prescriptive rules a Dilr Voice deployment configures. These are design rules, not measured findings, and every team should tune them to its own call types.

What is the difference between explicit and implicit confirmation?

Explicit confirmation asks the caller to verify a value directly, for example "I have the fourteenth of March, is that right?" Implicit confirmation folds the value into the next question so the caller can object if it is wrong, such as "and for the fourteenth of March, what time suits you?" Dilr Voice uses implicit confirmation to keep low-risk calls fast, and reserves explicit confirmation for the high-cost turns where a silent mistake is expensive to unwind.

The trade-off is friction against safety. Every explicit confirmation adds a turn, and blanket confirmation of everything is how you get the robotic, interrogation-style call that drives people to zero out for an agent. The skill is selective confirmation: cheap turns pass on an implicit check, and only the values that would cause a wrong booking, a wrong charge or a data protection problem earn a full explicit stop. This is where the AI operating model consulting work usually lands, because the list of which values are critical is a business decision, not an engineering one.

Implicit confirmation also does quiet compliance work. By repeating back what it captured, the agent gives the caller a natural chance to catch an error before it propagates, which supports keeping records accurate at the point of capture. It is a lighter touch than an explicit challenge on every field, and on most calls it is the right default for an enterprise voice agent that has to stay both accurate and bearable to talk to.

How should a voice agent re-prompt without a "sorry, I did not catch that" loop?

By making the re-prompt targeted and progressive instead of generic and repetitive. A blanket "sorry, I did not catch that" throws away everything the agent already heard and forces the caller to start over. A repair-aware agent re-grounds instead: it restates what it does know, names only the missing or uncertain piece, and asks a narrower question. Dilr Voice re-prompts on the specific slot, so the caller fixes one detail rather than repeating the whole turn.

The pattern below is the repair loop a Dilr Voice deployment follows on any uncertain turn, whatever the caller was trying to do.

Progressive prompting matters because the second attempt should not be a carbon copy of the first. If a caller's town was misheard, the agent can widen its help on the retry: offer to take it letter by letter, suggest the two nearest matches it did hear, or accept a postcode instead. Each step gives the caller a different, easier route to the same answer rather than the identical dead end, which is the whole point of designing a voice agent around recovery instead of around a demo script. A repair loop that varies its strategy recovers calls that a fixed apology loop simply loses.

When should a voice agent stop repairing and escalate to a human?

When repair has failed a small, fixed number of times, or when the turn is too consequential to keep guessing at. A well-designed Dilr Voice agent counts its repair attempts and hands off after two or three, rather than trapping the caller in an endless retry. It also escalates immediately, without exhausting the counter, when the uncertain value carries legal, financial or safety weight and a wrong reading would do real harm.

Escalation is where regulation becomes concrete. The EU AI Act's Article 14 human oversight requirement expects a person to be able to intervene in higher-risk systems, and under the UK's reformed regime the automated decision-making safeguards in Articles 22A to 22D of the UK GDPR, in force from 5 February 2026, mean an agent should not slide from a failed repair into a solely automated decision with significant effects on someone. In plain terms, when the agent is not sure and the stakes are high, the correct move is a human, not another attempt.

The handoff itself has to be clean, or you have just moved the frustration rather than solved it. The human who picks up needs the context the agent already gathered, which is why repair and escalation sit next to live agent-assist and whisper coaching in a mature deployment. Passing a warm, summarised handover, rather than a cold "please hold," is the difference between a rescued call and a lost customer, and it is a core part of any serious AI execution office rollout.

What are the compliance risks when a voice agent mishears?

The main risk is that a misheard value gets written into a system of record and then persists as inaccurate personal data. A turn that is misheard and repaired inside the call is never stored, so it raises no accuracy issue at all. The real exposure is the write-through case, where Dilr Voice commits a wrong number, a wrong appointment date or a mis-keyed amount to a CRM or case system without confirming it first.

That is exactly where the UK GDPR accuracy principle bites. Article 5(1)(d) requires that personal data be:

"accurate and, where necessary, kept up to date; every reasonable step must be taken to ensure that personal data that are inaccurate, having regard to the purposes for which they are processed, are erased or rectified without delay"

Read carefully, that duty is about the accuracy of a record, not about perfect transcription, which is why confirming critical values explicitly before they are saved is the strongest business reason to build repair properly rather than hope the recogniser is right. The Information Commissioner's Office treats data accuracy as a live obligation, so a value that was heard wrong and never checked is a defect you are meant to fix, not an acceptable cost of automation.

There is a disclosure dimension too. A caller who does not realise they are talking to a machine will not push back on a misheard detail the way they would with a person, so transparency and repair reinforce each other. Naming the assistant as AI and giving the caller an easy way to correct it are two halves of the same duty of care, which is why compliance and conversation design cannot be treated as separate projects.

What is the best confirmation strategy for a voice agent in 2026?

There is no single best strategy, and any vendor who claims one is selling you a script, not a system. For most enterprise Dilr Voice deployments the strongest default is selective confirmation: implicit confirmation on low-risk turns to keep the call natural, explicit confirmation only on critical values, and a hard escalation rule when repair fails. That balance protects both accuracy and the caller's patience, which a blanket confirm-everything policy destroys.

The honest concession is that this is not always the right answer. If your calls are short, single-purpose and low-stakes, for example a simple opening-hours query, heavy repair logic is over-engineering, and a lighter tool from a platform like Vapi, Retell AI, Synthflow or PolyAI may ship faster and serve you perfectly well. A competitor who bolts a shallow bot onto Twilio will beat a carefully governed deployment to launch every time, precisely because they skipped the repair and escalation design. The trade-off is that they also skip the recovery behaviour, and it shows the first time a caller is misheard on something that matters.

So the best strategy is the one calibrated to your actual call mix and risk. The way to find it is not a feature comparison but a scoping exercise: list your call types, mark which values are critical, and design repair around those. That is what a fixed-fee placement diagnostic produces, and it is why we start there rather than with a platform choice. Integrations with Salesforce and HubSpot then determine where confirmed values land, which closes the loop back to the accuracy duty above.

Does adding confirmations make calls longer?

Yes, every confirmation adds a turn, so blanket confirmation of every field noticeably lengthens calls and annoys callers. The point of a repair strategy is to spend that time only where it pays off. Dilr Voice confirms critical values explicitly and lets low-risk turns pass on an implicit check, so the extra seconds go to the amount or the date, not the small talk. Selective confirmation keeps calls both accurate and brisk.

Can lowering the confidence threshold fix misrecognition?

No, and treating it as a dial to turn is a common mistake. As Amazon's documentation warns, confidence scores are comparative, not absolute, so a lower threshold just makes the agent act on weaker readings and hide more errors. The real fix is better repair: detect uncertainty, confirm or re-prompt on the turns that matter, and escalate when recovery fails. Dilr Voice tunes thresholds per call type rather than chasing one global number.

Want to see this in production? Try Dilr Voice live, book an AI placement diagnostic, see our DATS methodology, or read about our approach to placing AI inside enterprise systems.

To place this in the wider voice AI picture, browse the voice AI blog category, and for what happens after the call, see how a post-call summary and wrap-up records the repairs and confirmations for audit. If you are still deciding whether to deploy at all, about Dilr.ai explains who we are and how we work.

Written by the Dilr.ai engineering team, practitioners who ship enterprise AI in production. Follow us on LinkedIn for shipping notes, or subscribe via the RSS feed.