A UK financial firm can now read, in the regulator's own words, what weak AI monitoring looks like. On 19 October 2026 the Financial Conduct Authority published AI in financial services: good practice and areas for improvement, drawing on supervisory work, the Mills Review and its AI Lab initiatives, including AI Live Testing. A week later the joint Bank of England and FCA survey put a number on the same weakness: 39% of respondents to its full questionnaire routinely monitored customer impact for their AI systems.
That gap is the subject of this guide, written for the Head of Regulatory Affairs or Chief Risk Officer at a bank, insurer or fintech who is not in an AI Live Testing cohort and still has to answer the same questions. It explains what the programme examines, what the October report says about live oversight, and which records a firm should hold once a model is serving customers. Testing is live only for firms the FCA has admitted, but the questions it asks are available to everyone.
The scope is deliberately narrow. The map of how each sector regulator applies the AI principles belongs to our risk framework guide, and senior manager accountability under SM&CR Phase 2 to our SM&CR post. Experimentation before deployment, which is the work of the FCA's Supercharged Sandbox, is covered here only to mark the boundary.
This guide is shipped by the team behind DATS, the five-stage AI consulting system from DILR.AI. Or see our AI operating model work, which covers governance, RACI and lifecycle, audit-ready by design.
What is FCA AI Live Testing?
FCA AI Live Testing is a pilot within the Financial Conduct Authority's AI Lab that supports firms deploying mature AI systems in live UK financial markets with real customers. Firms get regulatory support from the FCA and technical support from its partner, Advai. The programme is exploratory: its terms of reference say it does not cover AI auditing, technical certification or compliance with other frameworks.
The terms of reference draw the line against its sister service plainly. The Supercharged Sandbox helps firms in the discovery and experiment phase, while AI Live Testing concerns deployment in controlled live market environments. The practical test is whether real customers are touching the system. The programme sits beneath a wider policy choice: the FCA states that it does not plan to introduce extra regulations for AI and will rely on existing frameworks. Our enterprise AI consulting guide covers how that choice shapes a broader programme, and the fintech sector overview covers where AI pays first.
The programme grew out of an April 2025 engagement paper that drew 67 responses. The FS25/5 feedback statement records the problem respondents kept returning to: AI systems are probabilistic, and firms struggle to decide how much testing and ongoing monitoring is enough. Respondents asked whether 90% accuracy is sufficient against 95%, and whether matching a human process is good enough. Respondents noted that the answer depends heavily on the use case, and the terms of reference say the structure of testing depends on each use case's circumstances.
Who is in the AI Live Testing cohorts, and can a firm still apply?
AI Live Testing has run two cohorts, with seven firms named in the first and eight in the second, and Scottish Widows, part of Lloyds Banking Group, named on both lists. No third cohort had been announced on the FCA pages we checked on 8 November 2026, and second-cohort applications closed on 24 March 2026. A firm outside the programme should plan on building the evidence itself.
The first cohort, named in December 2025, was Gain Credit, Homeprotect, NatWest, Monzo, Santander, Scottish Widows and Snorkl. The second cohort, named on 21 April 2026, was Aereve, Coadjute, Barclays, Experian, GoCardless, Lloyds Banking Group (Scottish Widows), UBS and Palindrome. Its use cases include targeted support for investments, credit score insights, agentic payments, anti-money laundering detection and Know Your Customer. Our KYC review cost analysis looks at that last use case from a bank's side.
Meeting the criteria does not guarantee acceptance, and the programme is not a form of certification. The FCA's stated timetable is that testing concludes by the end of 2026, with an evaluation report in Q1 2027. FS25/5 described plans for two cohorts of approximately five to ten firms each. If the FCA opens another window, the eligibility bar is useful as a self-assessment whether or not a firm applies: a developed proof of concept, plans to deploy within months, identified AI-specific risks, some pre-deployment testing, and a way of showing how the model would be tested live against clearly defined risks. The terms also warn that non-financial use cases such as IT, sales, marketing or HR may be ineligible. Teams that cannot yet tick those boxes may be facing an AI operating model gap rather than a testing gap.
What does AI Live Testing actually examine?
AI Live Testing examines the whole AI system rather than the model alone. The FCA's terms of reference define that system as the model, its deployment context and core risks, governance and human-in-the-loop arrangements, evaluation techniques, and input and output controls. Participation runs through three phases: discovery, framework validation, and live system testing with shared evaluation of the findings.
The middle phase deserves most attention from firms outside the programme. Framework validation asks how well a firm's approach addresses the risks it identified, covering testing methods, governance, guardrails and monitoring, including data. In the final phase the firm monitors the system in live deployment against those same risks and metrics, and the FCA may ask to see the monitoring information. The sequence is the useful idea: metrics are chosen against named risks before launch, then the live system is held to them.
A firm can copy that sequence without any regulator in the room. Write down the intended outcome, name the risks that would defeat it, and pick one measure per risk before the system takes traffic. Where engineering teams need the mechanics, our piece on the evaluation harness covers the layer between a model and production, and canary and shadow traffic ramps show one staged rollout pattern from voice AI. DATS treats this as the Pilot to Production stage of its five-stage system.
The terms also note that the FCA may consider individual guidance, waivers or modifications where appropriate. Those tools are available to admitted firms in their specific circumstances, so a firm outside a cohort should not assume any relief from existing rules.
What did the FCA's October 2026 report say about AI monitoring?
The FCA's October 2026 report says firms should track real-world outcomes and adapt monitoring as risks evolve, and it criticises firms that watch only technical metrics or treat governance as a point-in-time exercise. The report's examples are illustrative and do not create new requirements, but they show what the FCA has seen working and failing across supervision and its AI Lab.
Section 3.2 of the report contains the sentence most worth pinning above a model risk committee: "Where firms monitored technical metrics rather than outcomes, they had less chance of meaningful oversight." It sits under a heading of narrow monitoring focus, alongside a second weakness of reactive or static monitoring, where firms could identify harm only after issues arose. Elsewhere the report records firms leaving long gaps between reviews of live outcomes. It sets no fixed timescale, and says each firm must determine what suits its use case.
The good-practice list is concrete enough to use as a specification. It includes continuous dashboards with thresholds and benchmarks, dedicated post-implementation monitoring groups, spot checks and automated scoring for drift or harmful outputs, logging and audit trails of AI interactions, and staff feedback on whether they used or modified an output and why. Our AI solutions page lists evaluation and observability for production agents among the problems we work on, for firms that want this built rather than specified.
One case study, from the report's section on products and services, shows how a technically successful control can still fail consumers. A firm used guardrails that successfully reduced model drift and unexpected responses, yet the tool then gave consumers overly generic outcomes and steered some towards inappropriate options. The FCA's reading is that the narrowly designed guardrail failed to consider what good consumer outcomes were. Firms remain responsible for the outcomes they deliver through AI-supported tools under the Consumer Duty, which is what binds them here, not the report. Our post on Consumer Duty in AI debt recovery applies that duty to one journey.
Where is the monitoring gap in UK financial firms?
The clearest reported monitoring gap in UK financial firms sits after deployment. Asked about controls for data risk in AI systems in the 2026 Bank of England and FCA survey, more than 90% of respondents each reported pre-deployment review, human approval and vendor due diligence, while ongoing monitoring, leakage testing and periodic review or recertification were each reported by around half.
The survey received 196 responses, of which 105 completed the full questionnaire, and 86% of respondents across both questionnaires reported using AI. The report sets that against 75% in 2024 but cautions that the two years do not cover the same population, and that its findings are evidence from participating firms and not a complete assessment of the sector. Unless stated otherwise, its figures come from the full questionnaire and use valid responses to each question. A separate question asked which metrics firms routinely monitored, and the answers show the same shape: as the report's authors note, measures focused on downstream impacts were less common.
Read against the FCA's wording, the ordering matters more than any single bar. User feedback, at 85%, tells a firm what people chose to report. Accuracy, at 72%, tells it how a model scored against a reference. Neither tells it whether a vulnerable customer was steered somewhere unsuitable and said nothing. Customer-impact metrics, at 39%, and risk metrics, at 34%, are where that question gets answered, and they are the least commonly reported.
This is a choice of measures, and the raw material often exists already: complaints, escalations, repeat contacts and outcome reviews. Linking those to the AI journey that produced them is an operating problem as much as a tooling one, which is where the DATS system starts. Our banking overview sets out where AI is already placed in that sector.
What evidence file should a firm outside the cohorts build?
A firm outside the cohorts should build a live evidence file for each material AI use case, recording the intended outcome, named risks, metric and threshold per risk, monitoring results, escalation decisions and changes made. The FCA's published examples and programme terms supply the content. The file should let a senior reviewer reconstruct what the system did and what the firm did about it.
The table below is our synthesis, not an FCA template. Each row draws on something the FCA has published, and a firm should scale it to the use case's risk. The FCA is explicit that there is no one-size-fits-all model.
| Record | What it should show | FCA anchor |
|---|---|---|
| Use case statement | Consumer need, target market, intended outcome, why AI was the right tool | Good practice, section 2.1 |
| Risk and metric register | Each named risk paired with a metric set before launch | Terms of reference, framework validation |
| Interaction log | Prompts, outputs, model versions, applied filters, escalation decisions | Good practice, section 2.2 |
| Outcome monitoring | Results across consumer groups, including vulnerability | Sections 2.1 and 3.2 |
| Threshold actions | What amber and red trigger, and who acts | Case studies M and P |
| Oversight record | Whether review was needed, whether it happened, what the reviewer did | Smaller firms example, section 3.3 |
| Model change log | Vendor model changes, impact planning, fallback tested | Section 4.2 |
Two rows carry more weight than their length suggests. The interaction log matters because the FCA links those records to complaints handling and to investigation when a consumer challenges an outcome, so a firm that cannot retrieve the model version behind a decision has a problem that surfaces late. The oversight record comes from one smaller firm that logged whether staff needed to review an output, whether they did, and whether they approved, overrode or escalated it. The FCA describes that as showing how oversight operated in practice. Our guide to keeping an AI tool inventory covers the register that this file hangs from.
The same discipline underpins our AI execution office, an embedded delivery model where production placements stay owned by the client. Teams earlier in the journey can use the wider compliance library to scope the regulatory side first.
How should thresholds and human review work in live AI monitoring?
The FCA's published examples suggest thresholds in live AI monitoring work best when agreed before launch, tied to the firm's risk appetite and linked to predefined actions, so a breach triggers a decision and not a discussion. The FCA's report gives examples of red, amber and green performance thresholds, and warns that human review can become a tick-box exercise when the reviewer's role and escalation criteria are unclear.
In the FCA's case study P, a firm translated its risk appetite into red, amber and green thresholds agreed between the business and model risk teams, with predefined actions and recurring concerns escalated to a governance board. In case study M, the relevant senior manager and business areas approved an incident playbook before sign-off, covering service interruption, unexpected outlier outputs and underperformance, with next steps ranging from enhanced monitoring to temporary shutdown. The structure is simple to copy. Deciding the numbers is harder, and respondents to FS25/5 noted that the right measures depend heavily on the use case.
Human review needs its own test. Case study O describes a firm where the reviewer role was unclear and became a tick-box exercise over time, with reviewers increasingly accepting AI-generated recommendations and some outputs proceeding despite low confidence. Case study N offers a practical counter: while establishing an affordability tool, one firm had reviewers assess each case without seeing the AI's proposed outcome, so human and AI judgements could be compared independently. Our work on human approval gate design goes deeper on reviewer mechanics.
Oversight functions need capability as well as a remit. The FCA lists second- and third-line functions lacking AI-specific expertise among its areas for improvement, with training that was generic or limited to awareness. Dilr Academy is our education line, and a firm should weigh any training provider against the role-specific needs the report sets out for senior stakeholders, technical teams and human reviewers. Where review happens during a live conversation, supervisor monitoring for voice AI covers intervention design, which this guide leaves aside.
How should firms monitor third-party AI models?
The FCA's good-practice examples point to monitoring third-party AI models by testing vendor assurances, obtaining notice of significant model changes and rehearsing a fallback. The 2026 survey found that for high-materiality third-party systems, 81% of respondents relied on vendor assurance and contractual arrangements, and 24% on deep knowledge of the system. The FCA lists untested reliance on vendor assurances as an area for improvement.
The FCA's position is short: "Firms can outsource AI services, but remain responsible for them." In the survey, applying model risk management principles to vendor-supplied models was the most commonly selected governance challenge, at 54% of respondents, and overseeing systems provided or materially influenced by third parties followed at 49%. Contracts remain necessary, and the report treats notice of significant model changes as good practice. They do not, on their own, tell a firm how a changed model behaves on its own customers.
Case study T shows the cost of skipping this. A firm built a product around a particular model with no alternative or transition plan, and had to redesign it when the model was retired. Case study Q shows the opposite design: a service built to switch models, with alternative server capacity elsewhere. For firms in scope of SYSC 15A, the report states that mapping AI dependencies into resilience planning is a requirement, so here the report is describing an existing requirement and not an illustrative example. Our posts on model supply chain assurance and disaster recovery and failover work through the mechanics for voice systems, and the same questions transfer to other production AI solutions.
What is the best way to prepare for FCA AI scrutiny in 2026?
The best way to prepare for FCA AI scrutiny in 2026 is to select your highest-impact live use case and rehearse the FCA's three phases on it internally, starting with the intended customer outcome. In our judgement that suits most mid-sized firms. A firm with mature model risk teams may need no outside help, and a multinational needing assurance across jurisdictions may be better served by a global firm such as Deloitte or KPMG.
We would use four criteria to judge any approach, including our own. Does it start from customer outcomes and not model scores? Does it produce records a senior reviewer can read without the build team present? Does it cover third-party models as rigorously as internal ones? And does it fit the firm's size? The FCA says smaller firms can govern AI effectively with targeted, proportionate controls, and it reports smaller firms using sample outputs, vulnerability markers in test cases and simpler risk tiers, so a proportionate version is a reasonable place to start.
A sensible order of work is to pick one use case, complete its use case statement and risk and metric register, set outcome measures and thresholds, then run a tabletop exercise on a threshold breach. The gaps that exercise exposes are the real backlog. Firms wanting a conversation about where to begin can talk to us, and our earlier analysis of FCA AI governance and the Treasury Committee response gives the policy background. Voice deployments can also look at Dilr Voice, and more about the team is on our about page.
Frequently asked questions
Does joining AI Live Testing mean the FCA has approved a firm's AI?
Joining AI Live Testing does not mean the FCA has approved a firm's AI system. The programme's terms of reference describe it as exploratory and state that it does not seek to cover AI auditing, technical certification or compliance with other frameworks. The FCA's October 2026 report says firms remain responsible for the AI systems they deploy, and the FCA can end participation at its discretion.
Does the FCA good and poor practice report create new rules for AI?
The FCA good and poor practice report does not create new rules for AI. Section 1.2 says its examples are illustrative, non-exhaustive and do not create new requirements or prescribe a single model. Existing obligations still bind firms, including the Consumer Duty, SM&CR and SYSC, and the report sets out publicly the practice the FCA regards as weak.
When will the FCA publish its AI Live Testing evaluation report?
The FCA has said its AI Live Testing evaluation report will be published in Q1 2027, after testing concludes by the end of 2026. An earlier version of the terms of reference pointed to the end of Q4 2026, so the April 2026 press release is the later statement. Firms should check the FCA's AI Lab page for any change.
Want to turn this into a working plan? See how DATS works, browse our industry guides, read about our approach to placing AI, or speak to a practitioner about your first live use case.
See what your live AI does to customers.
30-min scoping call · No deck · Confidential. We will tell you whether DATS fits, and which monitoring gap to close first.
Written by the Dilr.ai engineering team, practitioners who ship enterprise AI in production. Follow us on LinkedIn for shipping notes, or subscribe via the RSS feed.