Your Team Doesn't Need to Understand RAG. It Needs These 3 Things

Alex: Prudential in Singapore built an AI assistant for its insurance representatives. 6 of its own experts graded the answers. It passed 41%. 4 months later, it passed 99%. I'll tell you why it scored 41%, because the reason is most of the answer to this question.

Alex: Does your team need to understand RAG, agents and how this model works, or is that the wrong thing to be learning? I'm Alex Khomyakov. I build software for financial firms, and this is Curiosity Code. Here is where I land. Most people in financial firms don't need to know how RAG or an agent works.

Alex: Teach everyone how to use it and where it goes wrong. Teach almost no one how it works inside. What the firm has to know is what it depends on, which model is answering, what it can reach, and how you'd find out the day it changes or gets it wrong. And a few people on every team have to understand the pipeline properly. And they have to be in the room.

Alex: Jeff McMillan led AI at Morgan Stanley. He said that if he had $100, he'd spend $90 of it on education before building any technology. I agree with him and look at what he means by education: what AI is, how to prompt, and how to use it responsibly. How the model works inside isn't on his list. So back to Prudential and why it scored 41.

Alex: The assistant answered questions from more than 2,500 pages of product documents. Representatives ask it through a messaging app. Prudential built it in-house on a bought model. Then 6 experts from product, pricing, operations, and health sat down to grade it. 59% of answers failed.

Alex: Prudential gives 2 reasons. The first is the documents. In places they were too thin. So the answers came back vague or incomplete. The assistant was only as good as what it had been given to read.

Alex: The second reason is the experts. They hadn't agreed what a right answer was. So they sorted the questions into 2 kinds. Some answers must always be exactly correct. Others can be worded freely as long as they are factual.

Alex: Then they graded more than 500 answers the same way. 4 months of that, and it passed 99. Prudential had a technical team, and you need one. But look at what the 4 months of testing found. Thin documents and no shared definition of right.

Alex: Both of those belong to people who know the product. Let me show you how this plays out. I built a small mock-up for this video. A made-up travel insurer, a made-up policy wording, and a real model answering Claude Opus 5.5. A claims handler asks the assistant: the customer just doesn't fancy going any more.

Alex: Can they get their money back? There are 3 ways that answer goes wrong. One. It reads last year's policy wording because nobody took the old one out. The answer is confident, it cites a clause, and it's out of date.

Alex: What catches that is an owner, one named person who answers for what the assistant reads and says. Two. It reads the right wording and the wrong page. In my mock-up, simple word matching saw the word money and landed on the page about lost cash and passports. The assistant said it couldn't find an answer.

Alex: A better search, one that matches meaning as well as the words, found the page that says a change of mind isn't covered and got it right. That search uses a second model from Google. To rank the pages. The answers came from the same model both times. So yes, a few people have to understand search properly, and what catches the wrong page is graded questions.

Alex: My mock-up has 20, each with the answer the claims team would give. Word matching passed 12. The better search passed 14. So the better search still failed 6, and without the graded questions, I wouldn't know which 6. Morgan Stanley keeps 500 questions like that and reruns them every time something changes.

Alex: One of the failures is worth hearing. A customer had far too much to drink and fell off a balcony. Word matching found the page on medical cover and missed the page on alcohol. The assistant said he was covered, with a note to double-check. The better search missed the alcohol page too.

Alex: It just knew to say it wasn't sure. Three. Nothing changes on your side, and the answer changes anyway. Sometimes the vendor tells you. On the 28th of September, OpenAI switched off 4 of its older models.

Alex: Its policy is 6 months' notice for standard models. And as little as 2 weeks for preview ones. And sometimes nobody tells you because nobody knows yet. Last year, Anthropic published a review of 3 bugs in its own system. For weeks, they made some answers worse with no change to the model and no notice.

Alex: Its users noticed before its tests did. In its own words, the evaluations we ran simply didn't capture the degradation users were reporting. So here is what I'd do about it. Keep a register and don't keep it by hand. Pin the version in configuration so a change is a decision someone made and record with every answer which model wrote it.

Alex: Then the register is a report you run. Run your graded questions on a schedule every week, whether or not anyone says something changed. That's the fix Anthropic chose for itself, running its tests continuously on the live system. And give staff a button to flag a wrong answer, with the owner reading the flags. If you bought the whole product, you can't pin anything.

Alex: So put the question in the contract. What are you running and will you tell us before it changes? So that's 3 things: an owner, graded questions, a register. Only one of those failures needed a specialist to fix. All 3 needed someone who knows the policy to notice.

Alex: So who does this work? Mostly people who know the business. In August, a partner at a small mortgage brokerage emailed me. He'd been quoted $15,000 for an off-the-shelf system. He built his own with a coding agent for under $1,000.

Alex: He has no engineering background. He told me consultants don't understand the business. I told him he didn't need us. Guests on this show have seen the same thing from the other end. Stéphane Malrait described a bank pilot with one of the biggest technology companies.

Alex: They had 500 data scientists, he said. What they lacked was expert knowledge of the financial product. And Daniel Flowe of the London Stock Exchange Group warned me about the opposite mistake. An AI governance process run mostly by non-technical people, he said, is in trouble. I think both are right.

Alex: The few who understand the pipeline have to be in the room. And without the people who know the product, they build the wrong thing. So here is what I'd do on Monday. One piece of context first. In October, Singapore's regulator finalised its AI guidelines.

Alex: It asks the board for an adequate understanding of AI, enough to oversee and to challenge. It stops well short of asking the board how the model works. Now, 3 questions for your team. One for each of the 3 things. Which model produced the last AI answer that someone here relied on?

Alex: That's the register. If that model changed tomorrow, how would we find out? That's the graded questions. And who decides what a correct answer is? That's the owner.

Alex: If nobody can answer those, a course on how RAG works won't fix it. The 3 things and the 3 questions are in the show notes. Here is what I can't settle. You need those few people who really understand the pipeline. If you can't judge their work yourself, how do you know you have the right ones?

Alex: Tell me how you'd answer that. If you're watching, leave it in the comments. If you're listening, send me a message on LinkedIn and follow the show wherever you're listening or watching.