AI Agents: Who's Accountable for Mistakes? | Erich Barlow

Alex: Hello, everybody, and welcome to the Curiosity Code podcast. I'm your host, Alex Khomyakov, and my guest today is Erich Barlow, a Chartered Fellow of the BCS working in cybersecurity and AI governance. He is a doctoral candidate and he does lots of research in AI and AI standardization. He will speak a lot about that on this episode. And also he writes about what he calls synthetic intelligence, which is quite interesting.

Alex: It's not a machine imitating human thought, but machine intelligence as its own real thing, arrived at by a different route. The way a synthetic diamond is still a diamond. And he's here to talk about what happens when agents start acting with little or no intervention from the company that deployed them, who is accountable when one makes mistakes at computer speed and why the answer is harder than it looks. Welcome to the show, Erich.

Erich: Thanks, Alex. Good to be here.

Alex: Great. I've been waiting for this conversation for a long time and like, you know, AI governance is something that keeps lots of people awake who are implementing AI. And my first question to you is so were agents already acting on their own in financial services with little or no intervention from the company that deployed them?

Erich: So agent behavior already exists in fraud monitoring that we're already seeing, AML, alert triage, customer service automation, even claims processing. So we're already seeing that happening within the industry. Also underwriting support collections, payments, routing. So you're seeing a lot of automation already being deployed within the organizations that employ AI agents. They're also reviewing those decisions.

Erich: So organization may set up a policy and operating boundaries, but they also trans agent increasingly decides how to achieve the objectives within those constraints. So you're seeing organizations set up boundaries for those agents within their organizations, what autonomy they have, how they affect what they can do. So you're seeing those already. And, you know, a lot of firms will put that in place, but they don't necessarily communicate that out to their customers or clients. So you're already seeing it in place.

Alex: Yeah. And we talk a lot about how AI agents impact the industry and different implementations that already helping businesses, helping customers. But as we all know, this is artificial intelligence and it's not perfect. It does make mistakes. And in your opinion, who is responsible for these mistakes?

Alex: So when the mistakes happen at business level. So who is responsible?

Erich: So that's a very good question. We get that quite a bit, especially. Unfortunately, the responsibility still is not transferred to the AI system. It actually maintains with the deployer of those or the accountability remains with the institution that deployed it from a governance perspective. Responsibility is distributed across the executive leadership, business owners, the technology teams, model owners or vendors and the risk functions.

Erich: Regulators and customers will not accept that the AI did it. It's kind of funny when when it sounds when you say it out loud, the AI did it. It's kind of like blaming it on a small child. But that's where the responsibility lies. It lies with the deployer or the organization as a whole and how they deployed it.

Erich: So, yeah, and regulators, you just can't you can't point to the AI system. It's how it was set up and configured. So that's where that remains.

Alex: Right. So you've never seen the case so far where the answer like it's just the model did it was accepted by regulators.

Erich: Yeah. Yeah. You see that every once in a while and you'll hear customers and clients that deploy an AI agent and they're like, it's doing something that we we didn't tell it to do or it's acting up and it kind of feels like a small child, like a little toddler when when they do something wrong, you know, he's doing it. It's it's him. But it actually comes to their how they set it up.

Erich: They may have missed something. They might have misconfigured. And that seems to be kind of that root cause of those issues that happen.

Alex: But in general terms, your opinion is that leadership of the business is ultimately responsible for that. Right. Absolutely. Yeah. Does that change if the agent came from a vendor?

Alex: you know, rather than being built in house.

Erich: Yeah. Yeah. I can see where you're going with that. Yeah, it it still remains with the with the organization that deployed it. Ultimately, they're held responsible for anything that happens with that AI agent and how it interacts with the the industry or the customer or client base.

Erich: So if it's acting up, it still remains a human responsibility.

Alex: OK, let's move on to practical aspects of running agents. Can you share from your experience, what does managing an agent actually look like week to week for a team that has already deployed one?

Erich: Yeah, yeah. So it's like managing a high speed digital employee. Right. Teams review performance metrics expectations. So it's similar to how you would measure a new employee coming on onto your team.

Erich: So you're looking at how it's performing, what it's doing. You're looking at how how it reacts. If it's in a security space, you might want to look at security incidents and how those are managed. Are they different than where it was before? So you're looking for key performance indicators, KPIs when you're deploying an agent.

Erich: The other thing is, is you want to look at the software management or software activity for the agent as well, how it's performing as long as it's performing within the parameters. So it's more like a new employee, making sure it's catching on, making sure it's doing what you've asked it to do and how it's been deployed.

Alex: Right. And then in terms of like managing that employee, so in our conventional hierarchy of organization, you would always have a supervisor or a manager who will be occasionally checking what are you doing from your experience, who is actually reading those logs or traces left by agents and making sure.

Erich: Yeah. Yeah. There's actually a little bit of a goal to have kind of an intermediary agent now developed that will actually do some of that management. It's kind of funny that you pointed that out. It's kind of an automation that's being developed in some spaces right now to be able to kind of manage overall agents.

Erich: So you have like a mid level manager that's also AI generated. You asked me about the logs and everything. So usually that's the group or the technology team that's deployed. The agent will take a look at those and do evaluation based on the content or what they're seeing. And you can make determinations as to how to tweak your AI agent, what needs to happen to help it perform better.

Erich: So it's usually more. You're looking at how it relates response rates to logs, how it responds to queries, where it gets the information, that sort of thing. So, yeah, there's a lot to look at.

Alex: So going further into that, you write about systems that technically achieve the goal that were given and they actually cause harm doing it. So let's talk about that. Perhaps you can share some cases from financial services like bank or lender. I'm looking for your perspective and more fintech direction if you can share that.

Erich: Yeah, there's there's a lot of different things that have happened or I can share kind of thing. Mostly when financial tech organizations or finance organizations deploy technology, it's usually around the space of helping support customer delivery of services. And so that's where the focus with a lot of artificial intelligence is at right now. They're looking at improving performance, cutting back on fraud. So you'll see it within the fraud detection space, as I mentioned previously, where it's looking at holistically about withdrawals, like you withdraw money at 230 in the morning where you don't normally do that sort of thing, or you're looking at online transactions at weird hours or sites that you go to.

Erich: So it's looking it's it's doing pattern recognition. Organizations run into developing and implementing those AI solutions in a manner that allows them to react to those potential threats in a machine speed. Right. So they want to be able to do that quickly and provide accurate. The thing about it is that sometimes they overreact where you have out of your normal patterning.

Erich: We'll do something and it will say, OK, I need additional content so it may freeze the transaction. I've personally seen that. A lot of organizations will do that automatically as part of their approach to preventing potential fraud. And they want to just double check, OK, did you mean to do this twice? Did you do this for a certain amount?

Erich: Do you know that you did this? I know a lot of us have automations when it comes to making payments, and that also comes into play as well where automated payments can also trigger an AI in a way that is unexpected. So that's kind of where organizations are deploying them, mostly fraud, mostly in areas where it can help prevent potential threats or help the customer. You're already seeing that across the board, everything from communication, chat bots. Those are still AI and those are still very limited, but they definitely are employed by financial institutions as a means to help provide better services to their clients.

Erich: Right.

Alex: But I mean, if we talk about just that application level, support, chat bots, how much harm that could make to the business? Not that much, probably frustrated clients, lost business opportunity or transaction or something like that.

Erich: Yeah, you'd be surprised when somebody goes on to a site and you have a chat bot, chat bot collects quite a bit of data. They also collect metadata about the user. So you're collecting information about the system, you're collecting information about the browser. So it helps even that short interaction with the user will provide a lot of information for the organization who deploys the AI. And usually you have to opt in to that sharing of that information.

Erich: So the chatbots can provide a lot of support for the organization when you have a customer or client that's looking on the site for specific information and may not be finding what they're looking for. So it's very beneficial that way. So having the chat bots there as an interface between the user or the customer client in the business is definitely a a space where it excels at. But it also helps them find the information they're looking for quickly, probably more efficiently. It may be able to answer questions that you would normally have to go into a branch or an office to find out.

Erich: So it's very beneficial that way, especially when it's employed in a manner that allows it to do that.

Alex: How does governance layer usually look like for this kind of applications?

Erich: Yeah, so governance layer for for that really depends on where it's deployed. If it's a third party SaaS solution, which is probably about 90, 95 percent of the solutions that are out there, what that means is that means a third party cloud service provider has developed an AI and is deployed in a public cloud. So we're talking Azure, Google Cloud Services, AWS. And so they'll deploy it in the cloud and that allows the vendor of the AI to to manage that more effectively. And so that's the way most organizations are employing it.

Erich: Very few organizations will develop that internally or provide an internal resource in order to develop the AI solutions. They just don't have the expertise needed. And so they run into the skill set roadblock where they need to have additional skills, but they just can't. They can't do it. So a lot of organizations, especially financial technology organizations, will employ third party AI solutions as part of their development of AI.

Alex: So it's an interesting situation where, as we spoke at the beginning of the episode, the responsibility lies on senior management team, while the technology, in most cases, as you said, is elsewhere. So using third party vendor with its own governance. So in essence, the vendor is controlling what AI is doing. And the business only receives the results. Yeah, but the responsibility is on the business side.

Alex: It's interesting. Yeah.

Erich: think of it like a grocery store, right? You're buying a bunch of groceries and you're shopping around. Well, the you bring it home. Right. So the grocery store, that's the vendor, the deployer is the grocer, but they make a bad meal.

Erich: Now, you're not going to blame the cow. You're not going to blame the farmer. You're going to blame the cook because the cook is the one that made the bad meal. Well, that's kind of the way the AI governance within the space is looking at it is it's how you deploy it. So if you set it up incorrectly, that governance responsibility, that's still maintained with the deployer who deployed it, not the vendor, because the vendor could give you everything correct.

Erich: But you don't have something set up like you think it is. And that can cause an issue. And so therefore you're still responsible. You're still responsible ultimately for how things go. If they go bad, if they go right, good, bad, everything in between.

Erich: And you can't shirk that responsibility. And I think that that's probably a lot of the other misunderstanding is where does a vendor come in versus where do I come in with that?

Alex: So, yeah, let's chat a bit about autonomous systems. And your post, you draw a line between a compromised assistant giving a bad answer and a compromised autonomous system taking an action. Where does that line sit in payments business, for example? Yeah. The line is action authority.

Alex: And an assistant can recommend payment. An autonomous agent can initiate it, modify, redirect or approve one. Once a system can affect money, movement, customer records, credit decisions or security controls without real time human approval.

Erich: So you're seeing a lot of that within the the financial space where there's automations around the approval process for credit records, and it's just looking at like your credit history and your background before it makes the decision based on whatever criteria that it's got. And usually it's got quite a significant amount of information about the person looking for, say, like a loan or a business, some sort of payment method. So, yeah.

Alex: So what permission would you never grant an agent in payment business?

Erich: Never grant probably decision making around the like a mortgage, because there's so many different variables with that particular decision. Historical background, personal, who they are as a person. And I think it needs more of a human touch. So for me personally, I would hope and pray that they don't have it making mortgage decisions because that is a space where you need a human touch, right? You need to have that person there to look everything over and to be that final arbitrator and a lot of that you are seeing some AIs will make recommendations based on the criteria that they have or the data that they've got.

Erich: So you will see that. But I would hope that they don't do the mortgage. That would be my biggest area that I kind of steer away from, especially when it comes to AI.

Alex: Coming back to practical implementation of governance. We already spoke a bit about it, but I'd like to spend a bit more on this because I looked into this myself being a practitioner in the industry and I have certain arguments about it. So the way I look at this is that that permission written into a system prompt are not access controls. They're more like a suggestion to AI. And as far as I understand, the kind of approach won't survive an audit.

Alex: Am I wrong about that approach or still prompting as an acceptable way for the governance?

Erich: I don't think you're wrong. Prompt instructions or behavioral guidance for AI solution and not necessarily security controls. True access control requires independent enforcement mechanisms such as identity, authentication or authorization, segmentation, least privilege and an auditable policy enforcement. And that's where you're getting to when you talk about an audit. You want to have all of those in place.

Erich: A prompt can influence behavior. They cannot reliably enforce compliance. So auditors evaluate the controls, not the intentions. And really, that's kind of what you're talking about. You're talking about what the outcomes are of implementing those.

Erich: And so an auditor would look at what actually happens and not what you intended.

Alex: So what's interesting? And I'm curious if you're seeing the same thing from your side, what I'm seeing from my side being an industry that most of the people still using prompt as a way to govern AI agents and the LLM implementation.

Erich: Yeah.

Alex: Like, do you see the same patterns?

Erich: So what's happening is there is a growing need for DLP data loss protection. within the AI space. So when you put in a prompt, the query, how you shape the query may contain sensitive information. So say like financial information, and you'll see the organizations will want to put a prompt kind of intermediary between the prompt and the AI solution before it goes to the query, and they're trying to control that area within the that space. There's organizations out there that have got solutions.

Erich: One that I know very well, actually, that will actually record that information or prevent that information from being transmitted to the LLM. And so there's a growing need for just that sort of information. So you're kind of talking about DLP when it comes to submitting queries to a LLM, a large language model, and it's... you can control it. You can try, but it usually ends up taking a technology solution in order to prevent that.

Erich: So they are trying to do governance. A lot of it is end user training that happens that you need to have as part of an implementation of an AI solution. And so that seems to be more of a focus. It's more of a user training instead of a technology. But you also see organizations that have a technology solution implanted or implemented in that space as well.

Erich: So you kind of a one-two punch. If you can't train them, you put in a technology solution. So it depends on what they're what it is and how much control you need to have over it.

Alex: Let's talk about certification. Now there is a certification and organizations being certified. What can a supplier certificate honestly tell you and what can it not? And I'd like to understand if the certifications that everybody is getting is more like a tick box or it's actually important aspect of governing how organization is using AI.

Erich: Yeah. There's lots of certifications out there. There's a growing number. Some of them are dependent on geographical region. So there's the EU AI Act for those folks that are located in the EU.

Erich: So those are requirements. You also have the ISO 42001, which is an AI management system requirements. So we're talking about data protection for a lot of that. Certification, what that means for an organization that's developing or implementing an AI solution. If you're looking at a vendor who has that sort of certification, it means certain things, it means that they have certain things in place based on that standard or those guidelines, it means their engineering team has followed guidance that has been well established and well documented.

Erich: And those are ever evolving. They're making updates to those on a periodic basis. We haven't seen a lot of that. They've just come to the fore probably in the last couple of years or so. We're seeing a push for that.

Erich: And there will be at some point some sort of mandate that organizations that develop AI solutions have to adhere to those guidelines or those standards. So what what it looks like for an organization who goes and procures a solution from a company that has a certification, it means you can trust those organizations. And that kind of falls in line with what literature on the topic states as well, is that there's more trust in organizations that have AI certificates because their process has been evaluated by independent third party. And I think that that's the key aspect there. Right.

Erich: So it's not just that they're saying it, they can prove it, that they're following best practice. They're following how the AI should be developed. And that's an important aspect. That's a level of comfort that organizations can have knowing that their AI solution has passed that or that they've implemented those types of governance controls. So it is it's not just a tick box, but it can be differentiator between an organization that has well thought out solutions versus

Alex: organizations that don't. From your experience, what are the things that an auditor sees that the engineers do not?

Erich: So a lot of the times as an auditor, you see documentation that is they're doing it, you know, they're doing it. You can show that they're doing it, but they don't have it documented somewhere. And so it goes into a documentation rabbit trail where you're trying to find, OK, show me that you're doing this because you can't take their word when you're an auditor and again, I'm putting on my auditor hat here. When you're an auditor, you want to make sure that they have evidence to show that they're actually doing what they're saying they're doing. You can't just take it their word.

Erich: You have to have evidence like you know how they do their their processing, how do they do their procedures, what does that look like? You know, so a lot of a lot of organizations when it comes to the A.I. Standards will fall down when it comes to documentation and evidence that they're actually doing it. It's it's almost an afterthought sometimes. And you really kind of caution, especially if you're looking at how they're they're doing things, you're like, you're doing things really well.

Erich: You just need to document what you've got. And then once the documentation is there, everything is smooth. When you audit an organization, you can tell how well and effective their management team is by how much they have when it comes to documentation. If it's very robust, you can almost be certain that they've got everything in line and there's no no additional follow up. Sometimes you'll find there'll be minor issues and you go, oh, you need to correct this or this is a current best practice.

Erich: So I would suggest this when you get into that space, then it's kind of like, oh, OK, so it's just a minor thing. But you'll usually tell pretty quickly if you look at the documentation. And that's where most organizations will fail.

Alex: Erich, I'd like to spend a bit of time on the research that you're doing. And I think it's worth a separate episode to cover what you're doing. I'd like to tap into the synthetic intelligence. OK. And can you tell what is that and like what's the difference between artificial intelligence?

Erich: Yeah, yeah. So it's it's definitely a new area of focus and it's what we're talking about with the evolution of AI, artificial intelligence is shaping computer thought similar to the human brain and you'll hear the same language used like neurons, neural pathways, even some of the iconography for AI is use of the digital brain because that's kind of what we're doing with artificial intelligence. We're enabling a computer system to think as like a human being. Right. Synthetic intelligence is that next step.

Erich: It's it's thinking not like a human being, but like a computer. So you're providing it all of the content that you would say, like an AI, but you're allowing the AI or the computer system to think on its own to find its own pathways, to find its own way to do things. Think of it like providing its own thought, having it figure it out on its own. You know, they have some AI will go through iterations of training on learning how to do things, and sometimes they're guided training. Sometimes they're non guided training.

Erich: But it really that part of it is left up to the system. And so that's really kind of the change. Instead of providing the guidance for the AI solution, we're kind of letting the AI kind of figure that out on its own. And actually, that's evolving rapidly. The capabilities of synthetic intelligence is becoming even more complex than probably what researchers even thought it could be.

Erich: And that's that's coming out in the literature around the research that I'm doing on that particular space, and it's really an interesting field to see where AI is leading, and it seems to be going in that direction. So it's going from an intelligence that we helped create, develop and model after our own our own thought processes to one where we're kind of hands off and we're allowing it to develop on its own. And I think that that's why you're seeing a lot of kind of hype in the online and in the news lately around AI being it's going to kill us all in a week or whatever. I think that's all hype. I think it's just fear mongering.

Erich: And unfortunately, that does have an impact and people want to talk about it. I think we need to talk about ethics and how AI is used and and leave the development for for those that need to do that. But that's the difference between the synthetic and the artificial intelligence and it's still AI, but it's how it's trained and how it's developed is really the big differences and what you're modeling after.

Alex: Have you seen any practical implementations already or it's still too early and it's more like academia?

Erich: Yeah, yeah, right now it's it's a lot in research and it's still very early in the development side. Now, I can tell you that there are parts of governmental agencies that may be developing synthetic intelligence further along than what is being communicated out to the wider community, either in academia or in other spaces. And so you'll see leaps where one one day it's barely able to do anything and the next day it's flying an aircraft. How did they get from point A to point B? There had to be iterations in between there.

Erich: So I I think that it's being utilized already, but it coming out to the public general knowledge and general use, I don't think is being actively communicated or accurately communicated in a lot of times. So I think the synthetic intelligence is definitely moving forward from just a concept or just academic exercise even to a full on implementation. I think we're still seeing a lot of that. And some of it has to do with compute capacity. We're still lagging very far behind and we need a lot more compute capacity and capability in order to make synthetic intelligence really work.

Alex: Erich, before we finish, I'd like to make sure that listeners and viewers know where they can go and find more information about what you're doing, your research. What's the best place?

Erich: Yeah, so best thing to do to look me up. I'm online, of course, like everybody else. They can look me up on LinkedIn, do Google search and find me. I'd be happy to contact or communicate with folks. You can go to my website, www.erichbarlowllc.com.

Erich: If you're looking for more information about me or the work that I do, that would be a great place to go. Just anywhere online.

Alex: All right. Well, we'll make sure to include those links in the description. Absolutely. Hey, thanks a lot. It's been a great conversation on AI governance and the future of AI.

Alex: Really enjoyed that.

Erich: Thank you. Thank you for having me, Alex. Appreciate it.

Alex: You're welcome. And for the listeners, don't forget to hit the subscribe button, like, and leave a comment or your feedback on the platform where you're listening to this too and see you in the next episodes. Bye bye.