Voice AI post-call summaries: an accuracy guide

When a human agent finishes a call, they write a few lines into the customer record: what the caller wanted, what was agreed, what happens next. That note, not the recording and not the raw transcript, is what the next person acts on. It is read by a colleague picking up the case, by a manager building a forecast, and sometimes by a regulator asking what the customer was told. The record is the asset. The call is just the event that produced it.

Voice AI changes who writes the note, not how much it matters. A voice agent can draft a faithful summary in the second after the call ends, or it can draft a fluent, confident, plausible summary that quietly invents a detail nobody said. The second kind is worse than no note at all, because it reads as authoritative and gets trusted. In McKinsey's State of AI (November 2025), 88% of organisations report using AI somewhere, yet only around 6% are capturing material financial impact. A large part of that gap is operational trust: teams do not act on outputs they cannot rely on, and an unreliable call summary is a small, daily version of exactly that problem.

This guide is about the after-call synthesis layer specifically: how to make a voice AI post-call summary that is grounded in what was actually said, how to decide which fields a human still checks, and why UK data protection law treats an inaccurate summary as a legal exposure, not just a quality issue. We will deliberately hand the mechanics of disposition coding and CRM write-back architecture to their own guides and stay on the one question that decides whether the note can be trusted: is it true to the call?

This guide is shipped by the team behind Dilr Voice, enterprise voice AI built for regulated deployments. Or see DATS, our five-stage AI consulting system.

What is a voice AI post-call summary, and how does it differ from the live transcript?

A voice AI post-call summary is the short, synthesised record a voice agent writes after a call ends: the caller's intent, the outcome, any commitment made, and the next action. Dilr Voice produces it from the call transcript, but it is not the transcript. The transcript is the verbatim source; the summary is an interpretation of it, condensed for a human to read later. That act of interpretation is exactly where faithfulness can break.

The distinction matters because the two layers have different failure modes and different controls. A real-time transcript can be wrong through mishearing, and you fix that with better speech recognition and confidence scoring on the words. A summary can be wrong even from a perfect transcript, because summarisation compresses, and compression forces choices about what to keep, what to drop, and how to phrase an inference. A note that says "customer agreed to the upgrade" when the caller said "let me think about it" is a summarisation failure, not a transcription one. Treat them as two problems, because the fixes do not transfer between them. The summary sits near the end of the pipeline described in our enterprise voice AI agents guide, after the words are captured and before the record is closed.

Why do AI call summaries hallucinate, even when handed the full transcript?

Large language models hallucinate in summarisation even when the source document is supplied, because they generate the most probable continuation rather than only the supported one. Grounding the model in the transcript reduces invention sharply, but it does not remove it. The evidence is direct: on grounded summarisation, where the model is given the exact source text, the best systems still fabricate content at a measurable rate, and the harder and longer the source, the higher that rate climbs.

Vectara's next-generation Hallucination Leaderboard (19 November 2025) measures precisely this: it asks models to summarise more than 7,700 documents of up to 32,000 tokens, spanning law, medicine, finance and other domains, and scores how often the summary contradicts the source. On this harder benchmark the strongest model hallucinated in 3.3% of summaries, and several frontier models, including well-known ones, exceeded a 10% hallucination rate. The lesson for a contact centre is not "pick a better model and stop worrying". It is that even a grounded summary of a single phone call has a non-zero invention rate, so the invention has to be designed against, measured, and caught before it reaches the record.

This is also why the format of the summary matters. A free-text narrative gives the model the most room to add a smooth but unsupported sentence. A structured summary, where the agent fills named fields such as outcome, commitment and follow-up date, constrains it to the shape of a record and makes each claim easy to check against the transcript. Structured extraction does not make hallucination impossible, but it shrinks the surface where it can hide.

How do you keep a post-call summary grounded, and which fields still need a human?

You keep it grounded by constraining the model at every stage rather than trusting one clever prompt. Pull the summary only from the transcript, extract named fields instead of free prose, generate under instructions that forbid adding facts not present in the source, and attach a confidence signal to each field. Then route the low-confidence and high-stakes cases to a person before the note is trusted. The pipeline below is the shape we deploy; each stage narrows what the summary is allowed to say.

Not every field carries the same risk, so not every field earns the same scrutiny. A miscaptured callback time is an annoyance; a fabricated statement that a customer was told a product was suitable is a compliance event. The practical rule is to make the review threshold a function of consequence: auto-write the low-stakes fields, and require a human to confirm anything that records advice given, consent obtained, a complaint, a vulnerability, or a financial commitment. The same logic that governs when a call escalates to a human governs when a summary does, and pinning that threshold down is exactly what the AI operating model we set with each client is for, before anything goes live. Getting that threshold right is the difference between a summary layer that saves time and one that manufactures risk at scale.

Is an AI-written call summary personal data you must keep accurate?

Yes. A post-call summary about an identifiable caller is their personal data, and the UK GDPR accuracy principle applies to it in full. That principle is not softened by the fact a machine wrote the note; if anything the duty is sharper, because the volume is higher and the errors are systematic rather than one-off. The obligation sits in Article 5(1)(d), and the Data (Use and Access) Act 2025 left it untouched when its wider reforms commenced on 5 February 2026, so it reads today exactly as it always has.

"accurate and, where necessary, kept up to date; every reasonable step must be taken to ensure that personal data that are inaccurate, having regard to the purposes for which they are processed, are erased or rectified without delay"

UK GDPR, Article 5(1)(d)

Two consequences follow for a summary layer. First, the caller can exercise their Article 16 right and obtain rectification of an inaccurate note "without undue delay", so you need a correction path, not just a generation path. Second, the ICO draws a line you must respect in the wording itself. Its guidance on the accuracy principle says personal data must not be "incorrect or misleading as to any matter of fact", and that where a record contains an opinion, it "must make clear that it is an opinion". A voice agent that writes "caller was frustrated" is recording an inference, and the ICO's AI guidance is explicit that such records "should indicate that they are statistically informed guesses rather than facts". In practice that means the summary must separate what was said from what was inferred, and label the inference. This is where compliance and good engineering point the same way: the privacy notice and transparency obligations, the EU AI Act's interaction-disclosure rules, and the DPA 2018 all reward the same discipline of a record that does not overstate its own certainty.

What does inaccurate post-call data actually cost an enterprise?

It costs the thing every AI programme is trying to buy: trust in the output, which is what turns adoption into value. When a record is wrong often enough, people stop relying on it and rebuild context by hand, and the automation's promised saving evaporates. The macro picture makes the point plainly. Most enterprises now use AI, but only a small fraction convert it into measurable financial impact, and the operational record is one of the quiet places where that conversion fails.

We are deliberately not re-deriving the time-and-money case for automating after-call work here, because our after-call work guide already covers the disposition coding, wrap-up automation and the labour saving in full. The point that belongs to this guide is narrower and more important: those savings only bank if the note is faithful. An after-call work programme that is fast and wrong is not a saving, it is a debt that comes due the first time a manager forecasts on invented data or a regulator reads a summary that says something the caller never heard. Speed without faithfulness is negative value. That is why the DATS methodology treats summary accuracy as an acceptance criterion, not a nice-to-have.

How should the summary write back to Salesforce or HubSpot without corrupting the record?

Carefully, and never by blind overwrite. The write-back should append the summary as a new, timestamped, attributed entry rather than replacing existing verified fields, so an AI-generated line can never silently destroy a human-confirmed one. Low-confidence fields should land as drafts that a person clears, high-stakes fields should wait for review, and every written field should carry provenance: which call, which model, and whether it was auto-written or human-confirmed. That provenance is what makes the record auditable later.

The integration plumbing itself, how the agent authenticates to Salesforce or HubSpot, how it maps fields, and how it stays idempotent across retries, is a genuine engineering topic that our CRM and telephony integration architecture covers end to end, and the same tool-calling discipline described in our tool-calling architecture guide applies to a write-back call as much as to a lookup. What matters for accuracy is the contract on top of that plumbing: the CRM is the system of record, the transcript is the system of truth, and the write-back must preserve the ability to reconcile one against the other. A summary that lands in HubSpot with a link back to the transcript segment it came from is defensible; one that lands as an orphaned paragraph is not. Telephony providers such as Twilio can supply the call metadata that ties the two together, and building that link at write time is far cheaper than reconstructing it during an audit.

The same diagnostic logic underpins our AI operating model consulting, which sets the governance for who owns summary quality once the system is live rather than leaving it to whoever last touched the CRM.

What is the best voice AI for accurate post-call summaries in 2026?

There is no single best platform for every case; the right answer depends on whether your advantage comes from owning the summary logic or from having it delivered and governed for you. Builder platforms such as Vapi, Retell AI, Bland AI and Synthflow give you direct control of the summarisation prompt, the grounding constraints and the evaluation harness, which suits a team with in-house machine-learning evaluation capability and one narrow, high-volume call type to optimise. Managed platforms such as Dilr Voice and PolyAI put those same faithfulness controls, the confidence gating, the review thresholds and the provenance, inside the delivery rather than on your backlog.

The honest concession is that a builder platform can win outright when you have the eval muscle in-house and a single call type, because you can tune faithfulness harder than any generalist will for you, and you keep the summary logic as proprietary IP. Where managed delivery wins is breadth and assurance: many call types, several regulated lines, and an obligation to show an auditor how accuracy is measured and maintained. If you can staff continuous faithfulness evaluation yourself, build. If summary accuracy has to be demonstrable across a portfolio from day one, buy it as a governed capability. That trade, not a feature list, is the decision, and it is the one our DATS five-stage methodology is designed to make for you with evidence rather than vendor claims.

Should the post-call summary be structured JSON or free text?

Prefer structured fields for anything the business acts on, and add a short free-text note only as a supplement. Structured extraction, filling named fields such as outcome, commitment and follow-up, constrains the model to the shape of a record and makes every claim checkable against the transcript. Free text gives the model room to add a fluent but unsupported sentence, so if you keep it, keep it clearly secondary and never let it carry a fact the structured fields do not.

How do you measure summary faithfulness once you are in production?

Sample calls continuously and score each summary against its own transcript, not against a human's memory of the call. A faithfulness check asks one question per claim: is this supported by the source text? Track the rate of unsupported claims by call type and by field, the way Vectara's leaderboard scores summaries against source documents, and treat any drift upward as a regression. Dilr Voice runs this on a rolling sample so the number is live, not a launch-day artefact.

Who is liable if an inaccurate AI summary leads to a wrong decision?

The controller is. Under UK GDPR the organisation processing the personal data is accountable for its accuracy, and "the AI wrote it" is not a defence to an inaccurate record. That is why the review threshold and the provenance trail matter commercially and not just technically: they are how you demonstrate the "reasonable steps" the accuracy principle requires, and how you show a regulator or a court that a wrong summary was an isolated caught-and-corrected error rather than an ungoverned system writing fiction into customer files. We keep the rest of this territory current across our voice AI coverage.

Want to see this in production? Try Dilr Voice live, book an AI placement diagnostic, see our DATS methodology, or read about our approach to placing AI inside enterprise systems.

Written by the Dilr.ai engineering team, practitioners who ship enterprise AI in production. Follow us on LinkedIn for shipping notes, or subscribe via the RSS feed.