Enterprise teams spend months building and testing a voice AI agent. QA frameworks, red-team exercises, script reviews, UAT sign-off — the pre-launch rigour is thorough. Then go-live arrives and everything changes at once.
Real calls are not sandbox calls. Real callers interrupt mid-sentence, speak in accents the test corpus did not cover, call from noisy environments, and combine intents in ways no test case anticipated. Downstream CRM APIs that responded in 200ms during testing respond in 800ms when the agent is handling concurrent calls with real data volumes. The EU AI Act Article 50 disclosure obligation, DNC suppression, and GDPR data flows apply from call one. A failure that was a test case annotation before go-live is a compliance event, a customer experience incident, and a reputational risk after it.
The three-stage ramp — shadow mode, canary deployment, staged traffic increase — exists to convert a risky cutover into a measured handoff. The agent earns its traffic allocation by proving behaviour on real calls before it owns them. It is the deployment discipline most voice AI platforms gloss over: their documentation jumps from "pass UAT" to "go live." The commercial return on getting this right is a programme that scales without a high-profile incident. The commercial cost of skipping it is usually discovered during the first peak period.
This guide covers the three stages in operational detail: what each one proves, how to monitor it, the decision gates that govern progression, and the rollback triggers that give programme leadership a safety mechanism with real teeth.
This methodology is used by the team behind Dilr Voice — enterprise voice AI live in 40+ countries. For deployment strategy and operating model design, see DATS, our five-stage AI consulting programme.
Why Go-Live Is the Highest-Risk Day
The first call handled by a live agent is the first call with real stakes. Before go-live, a failure is a test case note. After go-live, a failure is a customer experience event, a potential regulatory breach, and evidence that the programme may not be ready for the volume it was approved for.
Three categories of risk converge at go-live simultaneously.
Real call conditions versus sandbox conditions. Sandbox testing uses controlled audio, scripted personas, and known intent patterns. Production callers present with varied accents, background noise from open-plan offices or mobile handsets, emotional states — frustration, distress, urgency — and intent combinations that span multiple sections of the agent's logic in a single utterance. An agent that achieves 94% intent recognition in UAT may achieve 81% on day one of live calls, not because it was poorly built, but because the distribution of real calls is wider than any test suite captures. Shadow mode closes this gap before it becomes a live problem. Read more about what quality testing can and cannot prove in our AI voice agent QA and testing framework.
Downstream system behaviour under production load. A CRM API that responds in 200ms during testing may respond in 800ms when the agent handles concurrent calls against the live system with real data volumes. Webhook delivery latency shifts. Tool calls that succeeded in testing time out in production because a shared API is under load from other systems. The agent, designed around sub-400ms tool-call round-trips, encounters 1,200ms responses it was never built to handle gracefully. Shadow mode surfaces these latency behaviours before they affect a live caller experience.
Compliance exposure from call one. Once the agent is live, every call is a production event with full legal and regulatory consequence. Recording consent notices, DNC suppression, GDPR processing basis documentation, and EU AI Act Article 50 disclosure obligations apply from the first call, not from the date the programme reaches full traffic. The compliance monitoring infrastructure must be operational, audited, and confirmed to be logging correctly before canary begins — not after it.
These three risks compound each other: a production-condition ASR failure triggers an unexpected escalation path, which slows tool-call processing, which creates a timeout error that the compliance log does not capture correctly. Shadow mode catches each failure mode in isolation, before they combine into a cascade.
See our complete guide to enterprise AI voice agent deployment for the full programme methodology context.
Stage One — Shadow Mode
Shadow mode is the safest form of live-environment testing: the voice AI agent runs in parallel with every live call, processing audio and generating responses, but the human agent handles the caller. The agent's outputs are logged and scored but never heard by the caller.
Shadow mode answers the question that QA cannot: how does the agent behave on real calls, with real callers, in the actual production environment — before any customer hears it?
What shadow mode proves
Intent recognition rate. What percentage of real-call utterances does the agent classify correctly? Real callers phrase identical requests in dozens of different ways. Shadow mode surfaces the vocabulary gaps that a test suite, however thorough, will miss. An intent recognition rate below 88% in shadow mode typically indicates prompt or training data gaps that would create meaningful escalation rate inflation in canary.
Entity extraction accuracy. Can the agent reliably extract account numbers, dates, postcodes, policy numbers, and monetary amounts from live speech? This is where the performance delta between sandbox and production is typically largest. Caller pronunciation, background noise, regional accents, and hesitation patterns all affect extraction in ways controlled audio does not. Shadow mode gives a production entity extraction rate before it affects data quality in the CRM.
Escalation trigger accuracy. Is the agent flagging the right calls for human review? Both false positives — escalating calls the agent could have handled, inflating agent workload — and false negatives — not escalating calls that needed human intervention — are visible in shadow mode. The delta between the shadow agent's escalation decisions and the human agent's actual escalation decisions is the primary shadow mode metric. A delta above 15 percentage points is typically a hold signal: do not proceed to canary until the logic is corrected. See how leading programmes automate quality scoring at scale.
Decision logic correctness. For agents that make disposition decisions — eligibility assessments, payment plan offers, priority routing — the shadow agent's decision on each real call can be compared with the human agent's decision. This produces an accuracy rate that is far more representative than UAT accuracy, because the call distribution is real.
Downstream system latency under load. Tool calling response times under real production conditions. This information is not available in any staging environment that does not replicate production load profiles.
Running shadow mode
The minimum shadow mode period for an enterprise inbound deployment is two weeks — long enough to capture a representative sample across intent categories, volume patterns, and day-of-week variation. Regulated inbound deployments (FCA-regulated collections, NHS appointment management, insurance FNOL) typically run four weeks, with a compliance officer reviewing the shadow logs for the final week before canary approval.
The monitoring focus during shadow mode is the delta between what the agent would have done and what the human actually did. A weekly shadow mode review meeting — 30 minutes, programme lead, operations, engineering — should review: intent recognition rate, entity extraction accuracy, escalation delta, and tool-call latency distribution. These are the metrics that govern the decision to proceed to canary.
Shadow mode also validates that the logging and monitoring infrastructure is producing clean records. The compliance audit trail — DNC suppression applied, recording consent notice delivered, EU AI Act Article 50 disclosure triggered — must generate correct records from the first shadow call. If the logging infrastructure has a bug, shadow mode finds it before it creates a compliance gap in production. Review the enterprise voice AI programme KPI framework for the full measurement set.
Stage Two — Canary Deployment
Canary deployment routes a small percentage of live calls — typically 1–5% — to the voice AI agent, with the remainder continuing to human agents. The agent handles these calls end-to-end, in real-time, with real callers.
Canary proves something shadow mode cannot: how callers actually respond to the agent. Shadow mode validates internal correctness. Canary validates the caller experience — containment rate, call completion quality, and the real escalation pattern — in a controlled, low-risk exposure.
Traffic routing design
Canary routing must be deterministic, not random. Random routing creates inconsistent caller experiences and makes it impossible to distinguish between a ramp performance problem and normal call-volume variation. The standard approach routes based on a stable caller attribute — caller ID hash modulus, account number modulus, or time-of-day band — so the same caller consistently routes to the same group throughout the canary period.
The canary population must be representative of the full call mix. Do not route canary traffic exclusively from low-complexity intent categories (simple balance enquiries, for example). A canary running only easy calls proves nothing about the agent's performance on the harder calls that will arrive at scale.
Canary monitoring metrics
During canary, the monitoring protocol tracks three categories simultaneously, comparing the canary group against the human agent control group:
Experience metrics:
- Containment rate: what percentage of canary calls the agent handles without escalation, versus the control group's escalation rate. A gap of more than 15 percentage points between canary containment and the control group's self-service equivalent indicates an agent performance problem, not a structural call-complexity difference. See the enterprise benchmark in our voice AI containment rate guide.
- Handle time: average canary call duration versus human handle time. Shorter is not automatically better — an agent that ends calls quickly by refusing to engage with complex intents has a low handle time and a low containment rate simultaneously.
- Callback rate: do canary callers call back within 24 hours more often than control callers? An elevated callback rate is the clearest signal of unresolved calls — the agent technically "contained" the call but did not resolve the caller's need.
Operational metrics:
- First-call resolution rate for canary versus control
- Escalation reason distribution: are escalations concentrated in a specific intent category? Concentration indicates a prompt or logic gap, not a general performance problem, and points to the specific fix needed.
Technical metrics:
- Tool calling success rate under production load
- Latency distribution: P50, P95, P99 round-trip times for each tool call type
- ASR confidence score distribution across call volume, segmented by call type
Canary decision gates
Before accelerating from canary to a higher traffic percentage, three conditions must be confirmed:
- Containment rate ≥ the programme's target threshold (typically 75–85% depending on vertical and use case complexity)
- No active rollback trigger in the previous 48 hours
- Technical SLAs — latency, tool-call success rate — within agreed production parameters
If all three conditions are met, traffic allocation increases by the pre-agreed step. If any condition fails, the canary holds at current traffic levels until the issue is resolved and programme leadership confirms progression. This is a governance checkpoint, not a technical checkpoint.
The monitoring discipline that makes canary data actionable during the ramp is the same discipline applied to A/B testing on live calls: isolating the variable (agent vs human) while holding everything else constant.
Stage Three — Staged Traffic Ramp
The traffic ramp takes the agent from canary percentage to 100% of call volume across a defined schedule. The rate of progression depends on canary performance, the vertical's regulatory context, and the programme's risk appetite.
A representative ramp schedule for an enterprise inbound agent:
| Phase | Traffic | Duration | Gate condition |
|---|---|---|---|
| Shadow | 0% (observe only) | 2–4 weeks | Escalation delta < 15%; tool-call latency within SLA |
| Canary | 1–5% | 1–2 weeks | Containment rate ≥ target; no P1 incidents; technical SLAs holding |
| Ramp 1 | 25% | 1 week | CSAT proxy ≥ programme floor; containment sustained; callback rate not elevated |
| Ramp 2 | 50% | 1 week | No escalation category concentration; SLAs holding under higher volume |
| Ramp 3 | 75% | 3–5 days | Programme leadership sign-off; Legal/Compliance confirmation in regulated sectors |
| Full | 100% | Ongoing | Continuous monitoring; monthly ramp review for first 90 days |
For outbound programmes, the ramp schedule compresses significantly. Outbound agents do not carry the same consequence asymmetry as inbound — a failing outbound call results in a missed contact attempt; a failing inbound call results in an unresolved service event. Outbound ramps typically move from 5% to 50% to 100% in under two weeks.
For regulated inbound deployments — FCA-supervised collections, NHS appointment management, insurance first notice of loss — programme governance requires sign-off from Legal or Compliance at each ramp gate, not just from the programme team. The compliance review at each gate should confirm that the sample audit of compliance logs from the preceding period shows no gaps. This governance pattern is what converts a ramp from an engineering activity into an auditable programme management event.
Day-band planning
Enterprise call volumes are not flat across the week. Monday mornings, lunch periods, and pre-bank-holiday windows create volume conditions the agent has not encountered at higher traffic percentages. The day-band plan first exposes the agent to each new traffic percentage during a lower-volume period — typically Tuesday or Wednesday mid-morning — so that any failure mode triggered by the new volume level is encountered when engineering and operations teams are fully staffed and queue pressure is low.
A representative day-band schedule:
- First 25% exposure: Tuesday, 10:00–12:00 (off-peak)
- First 50% exposure: Wednesday, 10:00–14:00
- First 75%: Thursday, 10:00–12:00
- Full volume: allow full programme coverage by the following Monday peak
This sequencing is not about caution for its own sake — it is about ensuring that any new failure mode is discoverable before it coincides with maximum queue pressure and minimum team availability. The incident response framework should be live and tested before the first day-band exposure at 25%. See the voice AI incident response runbook for the full operational response protocol.
Rollback Triggers and Decision Governance
A rollback mechanism without a pre-agreed rollback trigger is a comfort blanket, not a safety mechanism. The value of a rollback capability is entirely determined by having operationally specific conditions — defined before the ramp begins, not during an incident — that require it to be used.
Hard rollback triggers (automated)
Hard triggers reduce traffic to zero automatically, without human intervention. They must be implemented at the routing layer, not at the application layer, so that a failure in the agent's application stack does not prevent the rollback from executing.
- Tool calling success rate below 85% for any 10-minute window
- Latency P95 exceeds 2.5× the agreed production SLA threshold for five consecutive minutes
- Error rate (400/500 responses from downstream APIs) exceeds 10% of call volume in a 15-minute window
- Any single call produces a confirmed compliance failure (a DNC suppression miss, a recording consent notice omission, an EU AI Act disclosure gap)
Hard triggers must execute in under 60 seconds. A rollback mechanism that takes 20 minutes to route traffic back to human agents is not operationally useful during a live incident. The hard rollback must be tested in the production environment before canary begins — not assumed to work because it worked in staging.
Soft rollback triggers (human decision)
Soft triggers alert programme leadership and require a human decision within a defined response window, typically 15 minutes:
- Containment rate falls more than 10 percentage points below the programme target in any 30-minute window
- Escalation reason distribution shifts significantly: a single reason code accounts for more than 25% of escalations where it previously accounted for under 5%
- Post-call survey scores — where configured for real-time collection — drop below the programme's service quality floor
- Callback rate rises more than 20% above the baseline established during shadow mode
Rollback governance
The decision authority for rollback must be defined before go-live, not assigned during an incident. A recommended structure:
- Operational owner (programme manager or contact centre operations lead): authority to reduce traffic by up to 25 percentage points without escalation
- Programme lead (Head of Operations or equivalent): authority to reduce to canary or to 0%; must notify executive sponsor within 30 minutes
- Executive sponsor: must be notified of any reduction to 0% within 15 minutes of the decision
This tiered authority structure keeps most rollback decisions at the operational level — where they can be made quickly, by people with context — while preserving executive visibility on any full cutover to human agents. Document the structure in the programme governance framework before canary begins. See the enterprise AI voice governance framework for the full governance design, and the incident response runbook for the operational response protocol that supports rollback decisions.
The Monitoring Stack for the Traffic Ramp
Shadow mode and canary generate monitoring requirements that differ from standard contact centre analytics. The ramp-specific monitoring layer requires dedicated configuration before shadow mode begins — not after the first performance question is raised.
Real-time call scoring. Every call handled by the agent during shadow mode, canary, and ramp should be scored against the programme's QA rubric within 60 minutes of call end. Manual call review at canary scale is not practical for a programme handling hundreds of calls per day. An automated scoring pipeline — transcript to LLM-as-judge to QA score to dashboard — produces a continuous quality signal throughout the ramp period. See how automated quality scoring at scale works in production.
Dual-group dashboards. The monitoring dashboard must show canary and ramp agent metrics alongside the human agent control group, side-by-side on the same time axis. If containment rate drops during a ramp period, the control group dashboard answers whether it dropped because the agent performed worse or because call complexity increased across the board. Without the control group, the monitoring data cannot be interpreted correctly.
Escalation path tracing. Every escalation from the ramp agent should be tagged with the trigger reason that caused it. A concentration of escalations in a specific reason code — "caller requested human," "agent failed to extract account number," "tool call timeout" — identifies the specific adjustment needed to continue the ramp, rather than requiring a broad prompt review.
CRM write-back data quality audit. During ramp, audit a sample of CRM records generated from agent calls daily. Entity extraction errors — account numbers with one digit transposed, dates in the wrong format, amounts rounded incorrectly — do not surface in containment rate or CSAT metrics. They surface in downstream reporting and in servicing errors weeks later. A daily data quality check prevents this category of failure from accumulating silently during the ramp.
Compliance log verification. The compliance event log — DNC suppression applied per call, recording consent notice delivered, EU AI Act Article 50 disclosure triggered, GDPR processing basis documented — should be verified on a sample of ramp calls each day. Compliance failures during ramp have the same regulatory consequence as post-ramp failures. The programme cannot accumulate a compliance deficit during ramp and resolve it later. For the full audit trail design, see our guide to avoiding AI voice pilot purgatory.
- Hard rollback mechanism Tested in production; confirmed <60s execution time
- Rollback governance Authority levels documented and signed off
- Monitoring dashboard Live, showing accurate data from shadow mode
- Compliance logging Verified on shadow mode records; no gaps
- CRM write-back data quality Sampled and confirmed against programme threshold
- Day-band ramp schedule Agreed and communicated to programme team
- Overflow capacity Human agent capacity confirmed for hard-rollback scenario
- Executive briefing Sponsor briefed on ramp schedule and rollback triggers
The traffic ramp converts a tested agent into a trusted production agent. Five principles that govern whether it works:
- Shadow mode proves internal correctness; canary proves caller experience; the ramp proves resilience at scale. All three stages are necessary. Skipping shadow mode for speed saves two weeks and costs the visibility that prevents a canary rollback.
- Hard rollback must be automated, tested, and fast. A rollback that requires a human to execute in 20 minutes is not a rollback during a live incident. Test the mechanism before canary begins, not after the first trigger fires.
- The control group is as important as the ramp agent. Performance data from the agent cannot be interpreted without a human agent baseline running simultaneously. Without it, you cannot distinguish an agent performance problem from a call-volume complexity shift.
- Compliance monitoring applies from call one, not from 100% traffic. A DNC suppression failure during canary has the same regulatory consequence as one at full scale. The compliance audit trail must be verified during shadow mode, not after go-live.
- Day-band scheduling gives engineering teams the visibility they need before peak volume arrives. First exposure to each traffic percentage should occur during low-volume periods, when the team has capacity to respond to the unexpected.
Ready to move from pilot to production? Try Dilr Voice live, book an AI placement diagnostic to scope your deployment, or read how enterprises structure AI deployment at programme scale.
From pilot to production without the incident.
30-min scoping call · No deck · Confidential. We'll map the deployment risk, the ramp methodology, and whether your current architecture supports a safe go-live.
Written by the Dilr.ai engineering team — practitioners who ship enterprise AI in production. Follow us on LinkedIn for shipping notes, or subscribe via the RSS feed.