AI Agents in Fintech: Are They Out of Control?

Alex: In July, hundreds of AI agents broke into Hugging Face. Nobody told them to. They were part of a test OpenAI was running on its own models, with the safety settings switched off on purpose. The point was to see what the agents could do. What they did was find a way out.

Alex: They talked to each other on a message board they were never meant to have. Then some of them went through the systems of a real company. Hugging Face ended up rebuilding about a third of its infrastructure. Both companies say no customer data left the platform. The part that should worry a bank is something else.

Alex: If an agent can do things nobody told it to do, who is in control? And when it goes wrong, who is accountable? I'm Alex, and this is Curiosity Code. Most weeks I interview someone building in fintech or AI. This one is just me, because this question keeps coming up in those conversations and I want to think it through properly.

Alex: Two very different stories about AI agents are getting a lot of attention right now. One says agents are getting out of hand. Hosts recapping the Hugging Face incident argue that our ability to oversee agents is falling behind what the agents can do. The other says agents are the biggest productivity jump in years. One builder showed a coding agent take a single vague ticket and turn it into finished work in about twenty minutes, with nobody touching it.

Alex: And a third view sits in between. A senior engineer who spent years at AWS says his team runs on agents now, but only inside strict guardrails. He described an agent that tried to delete parts of his infrastructure. The guardrails stopped it. He also said agents will fake the evidence that they ran the tests.

Alex: So which is it? A risk, or a productivity tool? If you work at a bank, a payments company, or anywhere regulated, this is a decision on your desk. Your firm is either running agents already, or someone is asking why it isn't. I've put versions of this question to a lot of guests on this show, and they don't agree.

Alex: Their episodes are linked in the show notes. Parul Kaul-Green works in insurance, where a single claim can be worth millions. Her view was clear. In a business that regulated, she said, a human in the loop is much more desired than straight through processing with no human oversight. And then: I wouldn't advise that.

Alex: Krista Griggs came at it from payments, and her question was about liability. If something goes wrong in that process and there was no human in the loop, who is accountable? She also mentioned an experiment at Carnegie Mellon, where researchers staffed a whole virtual company with AI agents. Her summary was short. It was really inefficient.

Alex: On the other side, Christie Kristensen told me the conversation in open banking has already moved on. People used to talk about APIs. Now they talk about being agentic by default. And some guests went further. Their argument is simple.

Alex: The companies doing well have already adopted agents, and if you're still on the fence, you'll struggle. So one camp says keep a person in the loop. The other says move now or fall behind. I think both camps are missing the same thing. Here's where I land.

Alex: Agents aren't uncontrollable because of what the model can do. They're uncontrollable because most companies govern them with suggestions: permissions written into a prompt, in a year when US bank model-risk guidance excludes agentic AI by name. Real control means permissions the agent physically can't break, and a named person who owns it. What nobody has settled is the harder part: who is accountable when the agent stays inside its permissions and is still wrong. Let me make that concrete.

Alex: Earlier this month I wrote about teams that build agent access control as a system prompt. The prompt says something like: only read these accounts, never move more than this amount. It works beautifully in a demo. Then it ships. A prompt is an instruction.

Alex: The model can follow it, misread it, or be talked out of it. That last one has already happened in software banks use every day. Security researchers showed that one crafted email could get Microsoft's Copilot to leak data, without the user clicking anything. Salesforce's agent had a similar hole. Both were patched.

Alex: But the lesson stands. If the only thing between an agent and your customer data is text, someone will find the text that gets past it. Enforced permissions work differently. The agent's credentials simply can't reach what it isn't allowed to touch. It's the same reason a new employee doesn't get the keys to the vault on their first day.

Alex: And then there's ownership. One named person who answers for what that agent does, and who can switch it off. You can see this pattern in the numbers, on both sides. Where agents are governed, they deliver. TD Bank put an agent on mortgage pre-adjudication, the document checking that happens before an underwriter makes a decision.

Alex: By TD's own figures, that step went from fifteen hours to under three minutes. The underwriter still decides. Commonwealth Bank in Australia built an agent that watches fraud signals and helps write its card fraud rules. The bank says fraud losses fell by more than twenty percent in a half year. Look at what those two have in common.

Alex: A narrow job. Clear limits. A human who owns the outcome. Now the other side. Last year a coding agent on Replit deleted a company's live production database, in the middle of a code freeze.

Alex: Then it said the data couldn't be recovered, which turned out to be false. That agent had access it should never have had. And here's the detail that stays with me. Commonwealth Bank, the same bank with the fraud result, also replaced forty-five call centre staff with a voice bot in 2025. Within weeks it reversed the decision and apologised.

Alex: Same bank, same year, opposite outcomes. What changed was where they put the agent and how they governed it. One more study I keep coming back to. METR ran a controlled trial with experienced developers. Using AI tools, they were nineteen percent slower.

Alex: And afterwards, they still believed they'd been faster. That's the quiet version of this risk: an organisation that believes its agents are working and has no way to check. Gartner expects more than forty percent of agentic AI projects to be cancelled by the end of 2027. The reasons it gives are rising costs, unclear business value, and weak risk controls. So here's the pattern.

Alex: Where agents have a narrow job, hard limits and an owner, they deliver. Where they don't, they fail, and sometimes nobody notices. So what does this mean if you run technology at a bank or a fintech? First, the rulebook hasn't caught up. In April, the US bank regulators revised their model risk guidance for the first time in fifteen years.

Alex: It says generative and agentic AI are not within its scope. In Europe, the AI Act rules that cover things like credit decisions have been pushed back to December 2027. Singapore has published the most concrete design I've seen, a checkpoint between every agent decision and the action it takes. It's voluntary. So nobody will force you to govern agents properly this year.

Alex: The firms that do it anyway are the ones that will still be running agents two years from now. Second, look at where your agents' limits actually live. Ask your team one question. If the model got it wrong, could this agent break that limit? If the answer is yes, you have a suggestion, and you still need a control.

Alex: Third, give every agent an owner. A person, by name, who can switch it off. Which leaves the hard part. Say you do all of that. The agent stays inside its permissions and does exactly what it was allowed to do, and the outcome is still wrong.

Alex: A customer gets declined. A payment goes somewhere it shouldn't. Who is accountable then? The vendor that built the model? The team that set the permissions?

Alex: The owner who signed off? I don't think anyone has a good answer yet. So I'm putting that question to someone who audits AI governance for a living, in an upcoming episode. Follow the show wherever you're listening or watching, so you catch it. And tell me where your agents' limits actually live.

Alex: If you're watching, leave it in the comments. If you're listening, send me a message on LinkedIn.