Voice AI dead air: masking latency without faking it

A voice agent can pass every accuracy test in the lab and still lose the call in the first awkward pause. The caller finishes a sentence, and the agent goes quiet for two seconds while a language model composes its reply. The caller does not think the agent is reasoning. The caller thinks the line has dropped, and starts talking again, right over the top of the answer that was about to arrive. In human conversation the gap between one speaker finishing and the next starting is astonishingly short: across ten languages, researchers found the most common gap between turns falls between 0 and 200 milliseconds (Stivers et al., PNAS, 2009). Silence much longer than that reads as a fault.

This is the dead-air problem, and most teams meet it only in production. It is not about how fast the agent understands a request, which is natural language understanding, and it is not about detecting when the caller has stopped speaking, which is endpointing and turn-taking. Dead air is about what the agent does with its own thinking time: the unavoidable seconds while a model reasons, a knowledge base is queried, or a payment gateway responds. Fill that gap badly and the call collapses. Fill it dishonestly and a user-experience problem becomes a compliance one.

This guide covers where the silence actually comes from in a voice pipeline, how long you have before a caller reacts, and how to cover the gap without inventing activity that is not happening. It is written for teams past the demo, running real inbound traffic on an enterprise voice AI platform where every dropped call is a lost customer.

This guide is shipped by the team behind Dilr Voice, enterprise voice AI built for regulated deployments. Or see DATS, our five-stage AI consulting system.

What is dead air on a voice AI call, and why does it lose callers?

Dead air is the silence a caller hears while a voice agent processes their request and has not yet begun to speak. On an enterprise line it opens after the caller stops talking, while the language model reasons or a backend system responds. Dilr Voice treats dead air as a first-class failure mode, because a gap approaching two seconds reads to the caller as a dropped line and triggers either a hang-up or an interruption that derails the turn.

The reason the gap is so costly is that callers do not grade it against a computer's clock. They grade it against a human one. Human turn-taking is close to instant, so any silence that stretches past a second or so is read as trouble long before a model has actually failed. On our own outbound programmes we see hang-up rates climb sharply once dead air passes roughly two seconds, and the same threshold holds on inbound calls. The agent has not made a mistake at that point. It has simply been quiet for too long, and the caller has drawn the obvious conclusion.

Dead air is also distinct from the two problems it is most often confused with. It is not a barge-in issue, which is about handling a caller who interrupts, and it is not an endpointing issue, which is about knowing when the caller has finished. Dead air is the agent's own silence, on its own turn, and it needs its own controls.

How long can a voice agent stay silent before the caller reacts?

Not long at all. Human turn-taking research places the natural gap between speakers at 0 to 200 milliseconds, and conversation analysts describe a standard maximum silence of about one second before a pause is heard as marked and uncomfortable (Jefferson, 1989). On a phone line, where there are no visual cues that the other party is still there, tolerance is shorter still. Past roughly two seconds of dead air, most callers assume the connection has failed.

There is measured support for setting the trigger well inside that window. A study identifying the temporal threshold of tolerance for silent gaps after a request found that listeners begin to hear a gap as signalling trouble once it passes roughly 600 to 700 milliseconds (Roberts and Francis, 2013). The practical lesson for a voice agent is that the first cover signal cannot wait until the silence is obviously broken. It has to begin around the point a human would already have said something, well under a second, not at the two-second cliff where the caller is reaching for the end-call button.

This is why a mature agent works to a budget of thresholds rather than a single timeout. The first few hundred milliseconds are free. Somewhere before one second the agent should have acknowledged the turn. By two seconds it must have either delivered the answer or explained the wait. Designing to those bands is the core of good conversation design, and it is why the DATS five-stage methodology treats latency behaviour as a design decision, not a tuning afterthought.

Where does the silence come from in a voice AI pipeline?

The silence is rarely the voice engine. A voice agent runs three stages in series: speech to text, a language model, then text to speech. Modern streaming text-to-speech returns its first audio in well under 100 milliseconds, so the audible gap is almost always the model's reasoning step or a slow call to a backend system. Dilr Voice instruments each stage separately, so the team can see exactly which component produced the dead air rather than guessing.

The numbers make the point. Deepgram reports that real-time voice agents need first audio inside about 300 milliseconds to sound natural, and the leading speech engines clear that comfortably: ElevenLabs Flash returns around 75 milliseconds, and both Deepgram Aura-2 and Cartesia Sonic land near 90 milliseconds (Deepgram, 2026). Speech recognition on a strong provider adds a couple of hundred milliseconds. The stage that swings is the language model, whose time to first token ranges from a few hundred milliseconds to well over a second depending on the prompt, the context length and the load.

That distribution has a clear implication: latency work should start with the model and the retrieval path, not the voice. If you have not yet profiled where your round-trip time goes, our guide to voice agent latency benchmarks sets out how to measure each stage, and the AI placement diagnostic does the same profiling on a live flow before any masking logic is added. You cannot honestly cover a gap you have not located.

The same diagnostic logic underpins our AI operating model consulting, which sets the latency and fallback standards a voice programme is held to before it scales past a pilot.

How should a voice agent fill the gap without lying?

With graded, honest signals that match the length of the wait. A short backchannel such as "mm-hmm" or "let me pull that up" covers a sub-second gap. An explicit holding line, "I am checking your account now", covers one to two seconds. A progress update covers anything longer. Dilr Voice escalates through these cover phrases automatically, and every one is true: the agent says it is checking something only when a retrieval or a backend call is genuinely in flight.

The escalation matters because a single filler strategy fails at both ends. Backchannels alone leave a five-second database call feeling abandoned, while a formal "please hold" on every half-second pause makes a fast agent sound slow and robotic. The honest-masking ladder below is what a production agent walks as the wait extends, stopping the moment the answer is ready.

Two engineering choices make the ladder cheap to run. The first is streaming: if the model streams its response and the text-to-speech engine speaks the opening words as they arrive, the audible gap shrinks to the model's time to first token, and much of the ladder is never needed. The second is a small library of pre-synthesised cover phrases, so a backchannel plays instantly without waiting on its own synthesis round-trip. Both are standard in the way we build enterprise voice AI agents, and both reduce dead air before a single filler phrase is spoken.

When is a filler phrase honest, and when is it stalling?

The line is truthfulness. A cover phrase is honest when it describes something the agent is actually doing: retrieving a record, waiting on a gateway, or reasoning through a request. It becomes stalling, and then deception, when it invents activity to buy time. An agent that says "let me just check with my colleague" on a system with no colleague is not masking latency. It is misleading the caller, a more serious problem than a slow reply.

That boundary is also where regulation starts to bite. The EU AI Act sets a transparency baseline for any system that speaks to people:

"Providers shall ensure that AI systems intended to interact directly with natural persons are designed and developed in such a way that the natural persons concerned are informed that they are interacting with an AI system, unless this is obvious from the point of view of a natural person who is reasonably well-informed, observant and circumspect."

Article 50(1) does not ban filler phrases. It bans the impression that a machine is a person, and a filler that fabricates human activity walks straight into that impression. The safe rule we apply across every Dilr Voice deployment is simple: a cover phrase may describe a process, never a person. "I am looking that up" is a process. "My manager is just confirming" is a person, and it is off limits unless a person really is involved. Keeping that line clean is part of why regulated buyers put a voice agent through an operating model review before it goes near a live queue.

What should happen when the wait is genuinely too long?

When a backend stalls past the point any cover phrase can hold, the agent should stop pretending and act. Dilr Voice sets a silence-timeout on every turn: past a defined threshold the agent acknowledges the delay, offers to continue or reroute, and if the system is still unresponsive it routes to a person or books a callback rather than leaving the caller in dead air. A timeout ending in a graceful handover beats a flawless filler.

Designing that exit is not an edge case, it is the safety valve that lets you run cover phrases confidently in the first place. The escalation and human handover path has to be instant and context-preserving, so the caller does not repeat everything they have already said. It also has to interact cleanly with barge-in, because a caller who has waited too long will often interrupt the very progress update meant to reassure them, and the agent must yield rather than talk over them. Getting the timeout, the handover and the interruption behaviour to agree is exactly the kind of production detail the DATS methodology and our AI execution office are built to close out.

The macro picture is a useful reminder of why these details decide outcomes. Around 88% of enterprises now use AI, yet only about 6% capture material earnings impact from it (McKinsey, The State of AI, 2025). The gap is almost never the model. It is the unglamorous production work, and on a voice line dead air is one of the first places that work either shows up or does not.

What is the best way to handle voice AI latency in 2026?

The best approach reduces latency at the source first and masks only what remains. In practice that means streaming the model's response, choosing a fast text-to-speech engine, caching common answers, and shortening prompts, before layering any cover phrases on top. Masking a two-second gap is far harder than removing it. There is no single best tool for every team, so the honest verdict depends on how much real-time audio engineering you want to own.

Build-it-yourself platforms such as Vapi, Retell AI and Bland AI give you full control of the filler logic and the transport layer, and paired with a fast stack (a streaming model, a low-latency engine like ElevenLabs or Cartesia, and orchestration over LiveKit, Pipecat or Twilio) they can be tuned to very tight response times. The trade-off is that you own every part of the honest-masking ladder yourself. Managed platforms such as Dilr Voice and PolyAI ship that ladder already built and tuned, with the timeout, handover and disclosure behaviour wired together.

The scenario where build wins is a specific one: a team with dedicated real-time audio engineers, a single simple call flow, and the appetite to maintain the latency stack as models change underneath it. For that team the control is worth the maintenance. For almost everyone running several regulated flows, where the filler logic has to stay honest, the timeout has to hand over cleanly, and the disclosure has to satisfy the EU AI Act, a managed platform removes a class of failure the team would otherwise rediscover in production. If you are weighing the two, our view on where AI actually belongs in your systems is the place to start, and you can read more about Dilr.ai and how we make that call.

Does streaming the model's response remove the need for filler?

It removes most of it, not all. When a language model streams its answer and the text-to-speech engine speaks the opening words as they arrive, the audible gap collapses to the model's time to first token, often a few hundred milliseconds, which sits inside the natural turn window. Dilr Voice streams by default. Filler is still needed for the slower cases: a retrieval that must complete before the first word is known, or a backend call that blocks the turn.

Should a voice agent play hold music or background sound during a wait?

Rarely on a conversational turn. Hold music signals a formal queue, not a two-second pause, and playing it mid-conversation tells the caller they have been parked, which is worse than a brief honest holding line. A light, natural acknowledgement carries more reassurance and keeps the exchange feeling like a conversation. Dilr Voice reserves any hold treatment for genuine queueing, such as waiting for a human agent, and covers in-turn thinking time with spoken cover phrases instead.

Can backchannels like "mm-hmm" be overused?

Yes, and overuse is its own failure. A backchannel on every micro-pause makes an agent sound nervous and inattentive, and it can trip the caller's own turn-taking, prompting them to stop and start awkwardly. The fix is to gate backchannels on elapsed silence rather than sprinkling them, so they appear only when a gap is actually opening. Tuning that frequency by call type is part of how we prepare every Dilr Voice deployment before it takes real traffic.

Want to see this in production? Try Dilr Voice live, book an AI placement diagnostic, see our DATS methodology, or read more voice engineering notes in the voice AI blog.

Written by the Dilr.ai engineering team, practitioners who ship enterprise AI in production. Follow us on LinkedIn for shipping notes, or subscribe via the RSS feed.