The European Commission recently presented its action plan on Cybersecurity and Artificial Intelligence, which included a focus on AI Assurance, secure testing and governance.

Artificial intelligence is no longer something banks are merely experimenting with. In a recent keynote speech, Frank Elderson, Member of the Executive Board of the ECB, noted that their annual research reflects that more than 85% of banks under European banking supervision use artificial intelligence. Across Europe, AI is moving into live processes, customer interactions and increasingly into areas where systems can take actions rather than simply make recommendations.

That changes the assurance question significantly.

For decades, software testing in banking was built around a relatively straightforward principle. You knew what answer you expected, you ran the system, and you checked whether the result matched. That approach worked because software was largely deterministic. The same input produced the same output.

Generative and agentic AI break both assumptions.

I sat down with Samet Varlik, Head of Sales at Expleo Group, to explore what this means for banks as they move AI into production. One of his observations captured the challenge particularly well:

“an AI system that is right 96% of the time and obviously wrong on the remaining occasions could actually be safer than one that is right 99% of the time but produces highly convincing errors”.

In other words, the error rate alone tells us very little. Banks need to understand the nature of those errors, their potential consequences and whether anybody will notice when something has gone wrong.

From pass or fail to understanding risk

The first shift is in how banks define a successful AI system.

With traditional software, testing tends towards a binary outcome. Something either passes or fails. With AI, Samet argues that banks first need to define an acceptable range of behaviour and then assess how consistently the system stays within it. When it does leave that range, the questions become more important. How serious is the failure? Can it be detected? Can the system recover? Can the outcome be reversed?

Correctness is only one part of the picture.

Consistency, for example, could become a much bigger concern than many banks currently realise. Imagine two customers in essentially the same circumstances asking the same question using slightly different wording. The AI provides two different answers. Each might appear perfectly reasonable when viewed individually, but when placed alongside each other they could expose a fairness problem.

Traditional software testing rarely had to look for this because deterministic systems were not expected to behave differently simply because a question had been phrased differently.

AI assurance therefore needs to ask broader questions like:

• What situations have we actually tested?
• Does the system behave consistently when questions are rephrased? Does it know when it should stop and hand a case to a person?
• Can an important statement be traced back to information controlled by the bank?

Perhaps most importantly, Samet stressed that banks should stop thinking only about testing the model. Customers do not interact with a model in isolation. They experience the complete system, including the underlying data, instructions, controls, safety limits and human handovers and that whole system needs to be assured.

The important line is not between copilots and agents

Much of the current AI conversation revolves around terminology. We debate whether something is an assistant, a copilot or an agent and how autonomous it really is.

Samet suggested that banking may be looking at the wrong dividing line.

The more useful question is much simpler: can this AI change something for which the bank is responsible?

Can it amend a customer record? Move money? Create an obligation? Initiate a process that is difficult to stop?

Samet asked me to consider an AI system that drafts a customer letter and another that sends it to the customer. Technologically, those two capabilities may appear very similar. From an accountability perspective, they are completely different.

This distinction becomes particularly important when AI pilots move into production.

During the pilot, the AI makes a suggestion and a person checks it. Once the system scales, however, the business case may depend on people no longer reading every single output before sending it. At that point, the AI is no longer merely providing an input into a process; it has become a participant in it!

The risk can change suddenly rather than gradually.

And this is where treating AI agents exactly like human employees starts to become problematic.

An AI can make the same mistake 4,000 times before anyone notices

We often use human language to describe AI. We talk about digital employees, assistants and colleagues. That analogy is genuinely useful, but banks appear to underuse it.

Consider this: a new joiner doesn't get full access on day one. They get specific permissions, an approval limit, a second signature is required above a certain threshold, they get closer supervision at the start, a manager and an audit trail, and access can be withdrawn in an afternoon.

Banks already have all of that, but almost none of it has been applied to AI agents!

People make mistakes. So do AI systems.

The difference is scale.

As Samet pointed out, a person might misunderstand something and make a mistake once or twice before somebody notices. An AI agent could potentially make exactly the same mistake thousands of times in seconds before the problem becomes visible.

To remedy this, a bank might, for example, place limits on the number or type of actions an agent can perform, cap the total value it can affect within a given period and maintain an immediate stop mechanism; similar to the way we would treat a new joiner and potentially even stricter than we would treat a human, given the speed and scale that AI operates at.

This moves assurance beyond asking whether an AI produces a good answer. Banks need to test its behaviour, permissions and boundaries. Consider what exactly the AI is allowed to do and equally importantly, what is it not allowed to do?

Go-live can no longer be the end of assurance

Another assumption that AI challenges is that testing primarily takes place before deployment.

Samet was keen not to push this argument too far. Pre-production testing remains essential. If banks replace testing with monitoring, they risk watching a system they never properly understood in the first place.

The difference is that testing before launch now establishes a baseline rather than proving that the system is permanently safe. Conventional software is relatively stable unless somebody changes it, whereas an AI-enabled system has no equivalent finished state.

Banks therefore need to monitor three broad areas once AI is live: its inputs, its behaviour and its outcomes.

Behavioural monitoring might consider whether answers are still grounded in controlled information, whether similar questions continue to receive comparable responses and how often the system refuses a request or hands it to a person.

Apparently even positive changes need scrutiny. Samet told me that if an AI starts refusing fewer requests, that could suggest improved performance or it could equally indicate that its safety boundaries have weakened.

Outcome monitoring takes the bank closer to what customers and regulators will ultimately care about:

• Are employees increasingly overriding the AI?
• Are complaints rising?
• Are customers returning because the original answer was inadequate?
• Are transactions having to be reversed?

Banks will also need rigorous version control across models, data and dependencies. In an AI environment, saying that “nothing changed on our side” may become an increasingly inadequate explanation when system behaviour changes.

Does banking need an AI control room?

I asked Samet whether this continuous assurance requirement would eventually lead banks towards something resembling an AI control room.

His answer was yes, although not necessarily as a new standalone department.

Many of the required capabilities already exist across operational resilience, model risk, service management, compliance and technology. The missing element is often the ability to monitor the quality of what an AI system is actually saying and doing.

Traditional operational monitoring can tell you whether a service is available or whether response times are acceptable. AI assurance needs to answer a different question: are the outputs still as good as they were when we launched?

That requires business knowledge as much as technical knowledge.

The people assessing an AI used in collections, lending or customer service need to understand those processes and the customer outcomes involved. Knowing the testing platform is not enough.

For assurance professionals, Samet sees the role shifting away from writing scripts with predefined expected answers and towards designing experiments, creating realistic test data, hunting for unexpected failure modes and judging the severity of potential consequences.

Reporting will also need to evolve. Rather than a simple pass or fail, decision-makers will increasingly need information about risk, severity and confidence.

Accountability needs a name

Perhaps the most important part of our discussion concerned ownership.

AI touches technology, compliance, model risk, security, operations and the business, and the danger is not necessarily that nobody is paying attention; it is that many different teams are each looking at one piece of the problem, which is already a problem today.

Try this in your own bank: ask who has approved a live AI system?

In most cases, the answer will be a department or committee rather than an individual.

Samet believes accountability should follow the customer outcome.

He believes that if AI influences a lending decision, a payment, a complaint or a piece of advice, the person accountable for that business outcome should ultimately own the decision to deploy the AI.

His test is refreshingly simple: when something goes wrong, who takes the call?

Who explains what happened to the customer, to the regulator or to the executive committee? That person should have had a meaningful role in approving the system in the first place. Attaching a name to the decision also changes behaviour, as people become much more interested in edge cases, limitations and failure scenarios when they know they are personally accountable for the result.

If everybody owns a small piece of the risk, there is a danger that nobody owns the outcome.

Agree when you will switch it off before you switch it on

I finished by asking Samet what he would personally need to see before putting his own name behind an AI system being deployed to customers.

His answer provides a useful checklist for any bank approaching production:

• He would want clear boundaries around what the system can and cannot do,
• Evidence from real and difficult customer scenarios,
• Proof that somebody has deliberately tried to break it, and
• A clear understanding of how it can fail and how serious those failures could be.

He would also want evidence that human oversight works with real employees under real pressure, not simply an assumption that somebody will intervene because an approval button exists.

Then come the operational safeguards:

• Live monitoring,
• A functioning stop mechanism,
• A named owner and
• A backup process that has genuinely been tested.

But his final point may be the most valuable.

Before launching an AI system, the bank should agree on what circumstances would cause it to be switched off!

If nobody can describe those conditions in advance, Samet argues that the organisation has not really completed the decision to deploy it.

As AI becomes more autonomous, assurance can no longer be about proving that technology works on the day it goes live. It is becoming the discipline through which a bank establishes how much it trusts an intelligent system, what it permits that system to do and how quickly it can regain control when something goes wrong.

For banks, that may prove to be one of the defining governance challenges of the next phase of AI adoption.

It's always difficult to summarise an interview packed with insights, so while I've done my best above, you may prefer to watch or listen to the full interview below or follow along on your favourite podcast platform here (don't forget to subscribe for further insights from industry experts!).