Skip to content

Your Team Doesn't Need to Understand RAG. It Needs These 3 Things

· 8:40

Listen on

A solo episode built around one number. Prudential in Singapore built an AI assistant for its insurance representatives, six of its own experts graded the answers, and it passed 41 percent. Four months later it passed 99. Alex Khomyakov, who builds software for financial firms, uses the reason it scored 41 to answer a question his clients keep asking: does the team need to understand retrieval, agents and how the model works, or is that the wrong thing to be learning?

Why did an in house assistant score 41 percent?

For two reasons, and neither was the technology. The assistant answered questions drawn from more than 2,500 pages of product documents, reachable by representatives through a messaging app, built in house on a bought model. Six experts from product, pricing, operations and health sat down to grade it, and 59 percent of the answers failed.

The first reason Prudential gives is the documents. In places they were simply too thin, so answers came back vague or incomplete. The assistant was only ever as good as what it had been given to read.

The second reason is the experts themselves: they had not agreed what a right answer was. The fix is the part worth copying. They sorted questions into two kinds, those whose answers must always be exactly correct and those that can be worded freely provided they are factual, then graded more than 500 answers the same way. Four months of that produced 99 percent.

The conclusion he draws is that Prudential had a technical team, and needed one, and what four months of testing actually found was thin documents and no shared definition of correct. Both of those belong to people who know the product.

What should everyone learn, and what should almost nobody learn?

Everyone should learn how to use it and where it goes wrong. Almost nobody needs to learn how it works inside.

What the firm as a whole has to know is narrower and more specific than a curriculum: what the system depends on, which model is answering, what that model can reach, and how you would find out on the day it changes or gets something wrong.

The exception is real and small. A few people on every team have to understand the pipeline properly, and they have to be in the room.

The authority he cites agrees. Jeff McMillan, who led AI at Morgan Stanley, said that given a hundred dollars he would spend ninety on education before building any technology, and when you look at what he means by education it is what AI is, how to prompt, and how to use it responsibly. How the model works internally is not on the list.

What are the three ways the answer goes wrong?

He built a mock up to show them: an invented travel insurer, invented policy wording, and a real model answering. The question a claims handler asks is whether a customer who simply does not fancy going any more can have their money back.

First, it reads last year's policy wording, because nobody removed the old document. The answer is confident, it cites a clause, and it is out of date. What catches that is an owner: one named person who answers for what the assistant reads and says.

Second, it reads the right wording and the wrong page. In his mock up, simple word matching saw the word money and landed on the page about lost cash and passports, so the assistant said it could not find an answer. A better search, one matching meaning as well as words, found the page stating that a change of mind is not covered and got it right. That improved search used a second model to rank pages, while the answers came from the same model both times, which is why a few people do have to understand search properly.

What catches the wrong page is graded questions, and his numbers make the case. His mock up has twenty questions, each with the answer the claims team would give. Word matching passed twelve. The better search passed fourteen, which means the better search still failed six, and without the graded set he would not know which six. Morgan Stanley keeps 500 questions like that and reruns them every time something changes.

One of his failures is worth hearing in full. A customer had far too much to drink and fell off a balcony. Word matching found the page on medical cover, missed the page on alcohol, and the assistant said he was covered with a note to double check. The better search missed the alcohol page too. The only difference was that it knew to say it was not sure.

Third, nothing changes on your side and the answer changes anyway. Sometimes the vendor tells you: on 28 September OpenAI switched off four of its older models, with a policy of six months' notice for standard models and as little as two weeks for preview ones. Sometimes nobody tells you, because nobody knows yet. Last year Anthropic published a review of three bugs in its own system which for weeks made some answers worse with no change to the model and no notice, and users noticed before its own tests did. In its own words, the evaluations it ran simply did not capture the degradation users were reporting.

What do you actually put in place?

Three things, and a register is the first. Keep it, and do not keep it by hand. Pin the model version in configuration so that a change is a decision somebody made, and record with every answer which model wrote it, at which point the register stops being a document and becomes a report you run.

Second, run the graded questions on a schedule, every week, whether or not anybody has announced a change. That is the fix Anthropic chose for itself, running its tests continuously against the live system.

Third, give staff a button to flag a wrong answer, and have the owner read the flags.

If you bought the whole product rather than building on a model, you cannot pin anything, so the question moves into the contract: what are you running, and will you tell us before it changes.

So: an owner, graded questions, a register. His observation about the three failures is the argument of the whole episode. Only one of them needed a specialist to fix. All three needed somebody who knows the policy to notice.

Who does this work?

Mostly people who know the business, and he has evidence from both directions. In August a partner at a small mortgage brokerage emailed him, having been quoted 15,000 dollars for an off the shelf system, to say he had built his own with a coding agent for under a thousand, with no engineering background, because in his view consultants do not understand the business. Alex told him he did not need them.

Guests on his show have seen the same thing from inside large institutions. Stéphane Malrait described a bank pilot with one of the biggest technology companies, which had 500 data scientists and lacked expert knowledge of the financial product. Daniel Flowe of the London Stock Exchange Group warned him about the opposite failure, that an AI governance process run mostly by non technical people is in trouble.

His reading is that both are right. The few people who understand the pipeline have to be in the room, and without the people who know the product they will build the wrong thing.

The regulatory context he adds supports the split. In October Singapore's regulator finalised its AI guidelines, asking the board for an adequate understanding of AI, enough to oversee and to challenge, and stopping well short of asking the board how the model works.

Which produces three questions to put to a team, one for each of the three things. Which model produced the last AI answer that somebody here relied on, which is the register. If that model changed tomorrow, how would we find out, which is the graded questions. And who decides what a correct answer is, which is the owner. If nobody can answer those, a course on how retrieval works will not fix it.

What he cannot settle is the hiring problem underneath. You need the few people who genuinely understand the pipeline, and if you cannot judge their work yourself, you have no way of knowing whether you have the right ones.

Takeaways

  • An assistant that failed 59 percent of expert grading failed on thin source documents and on experts who had never agreed what a correct answer was.
  • Splitting questions into those that must be exactly right and those that need only be factual is what made grading possible at all.
  • Teach everyone how to use the system and how it fails. Teach almost nobody how it works inside, and keep a few who understand the pipeline in the room.
  • Ninety dollars of every hundred on education, per Morgan Stanley's AI lead, and none of it on model internals.
  • A stale document produces a confident answer with a citation. Only a named owner catches that.
  • Better retrieval improved his mock up from twelve of twenty to fourteen. Without a graded set, nobody would know which six still failed.
  • The best outcome in his worst example was a system that said it was not sure. An assistant that hedges beats one that confirms cover it cannot find.
  • Models are withdrawn on notice periods as short as two weeks, and sometimes quality degrades with no change and no announcement at all.
  • Pin the version, log which model answered, run the graded questions weekly, and give staff a button to flag a bad answer.
  • If you bought the product rather than the model, put the change notification question in the contract.
  • Singapore's regulator asks boards to understand AI well enough to oversee and challenge it, not to explain how it works.

Transcript

Alex: Prudential in Singapore built an AI assistant for its insurance representatives. 6 of its own experts graded the answers. It passed 41%. 4 months later, it passed 99%. I'll tell you why it scored 41%, because the reason is most of the answer to this question.

Alex: Does your team need to understand RAG, agents and how this model works, or is that the wrong thing to be learning? I'm Alex Khomyakov. I build software for financial firms, and this is Curiosity Code. Here is where I land. Most people in financial firms don't need to know how RAG or an agent works.

Alex: Teach everyone how to use it and where it goes wrong. Teach almost no one how it works inside. What the firm has to know is what it depends on, which model is answering, what it can reach, and how you'd find out the day it changes or gets it wrong. And a few people on every team have to understand the pipeline properly. And they have to be in the room.

Alex: Jeff McMillan led AI at Morgan Stanley. He said that if he had $100, he'd spend $90 of it on education before building any technology. I agree with him and look at what he means by education: what AI is, how to prompt, and how to use it responsibly. How the model works inside isn't on his list. So back to Prudential and why it scored 41.

Alex: The assistant answered questions from more than 2,500 pages of product documents. Representatives ask it through a messaging app. Prudential built it in-house on a bought model. Then 6 experts from product, pricing, operations, and health sat down to grade it. 59% of answers failed.

Alex: Prudential gives 2 reasons. The first is the documents. In places they were too thin. So the answers came back vague or incomplete. The assistant was only as good as what it had been given to read.

Alex: The second reason is the experts. They hadn't agreed what a right answer was. So they sorted the questions into 2 kinds. Some answers must always be exactly correct. Others can be worded freely as long as they are factual.

Alex: Then they graded more than 500 answers the same way. 4 months of that, and it passed 99. Prudential had a technical team, and you need one. But look at what the 4 months of testing found. Thin documents and no shared definition of right.

Alex: Both of those belong to people who know the product. Let me show you how this plays out. I built a small mock-up for this video. A made-up travel insurer, a made-up policy wording, and a real model answering Claude Opus 5.5. A claims handler asks the assistant: the customer just doesn't fancy going any more.

Alex: Can they get their money back? There are 3 ways that answer goes wrong. One. It reads last year's policy wording because nobody took the old one out. The answer is confident, it cites a clause, and it's out of date.

Alex: What catches that is an owner, one named person who answers for what the assistant reads and says. Two. It reads the right wording and the wrong page. In my mock-up, simple word matching saw the word money and landed on the page about lost cash and passports. The assistant said it couldn't find an answer.

Alex: A better search, one that matches meaning as well as the words, found the page that says a change of mind isn't covered and got it right. That search uses a second model from Google. To rank the pages. The answers came from the same model both times. So yes, a few people have to understand search properly, and what catches the wrong page is graded questions.

Alex: My mock-up has 20, each with the answer the claims team would give. Word matching passed 12. The better search passed 14. So the better search still failed 6, and without the graded questions, I wouldn't know which 6. Morgan Stanley keeps 500 questions like that and reruns them every time something changes.

Alex: One of the failures is worth hearing. A customer had far too much to drink and fell off a balcony. Word matching found the page on medical cover and missed the page on alcohol. The assistant said he was covered, with a note to double-check. The better search missed the alcohol page too.

Alex: It just knew to say it wasn't sure. Three. Nothing changes on your side, and the answer changes anyway. Sometimes the vendor tells you. On the 28th of September, OpenAI switched off 4 of its older models.

Alex: Its policy is 6 months' notice for standard models. And as little as 2 weeks for preview ones. And sometimes nobody tells you because nobody knows yet. Last year, Anthropic published a review of 3 bugs in its own system. For weeks, they made some answers worse with no change to the model and no notice.

Alex: Its users noticed before its tests did. In its own words, the evaluations we ran simply didn't capture the degradation users were reporting. So here is what I'd do about it. Keep a register and don't keep it by hand. Pin the version in configuration so a change is a decision someone made and record with every answer which model wrote it.

Alex: Then the register is a report you run. Run your graded questions on a schedule every week, whether or not anyone says something changed. That's the fix Anthropic chose for itself, running its tests continuously on the live system. And give staff a button to flag a wrong answer, with the owner reading the flags. If you bought the whole product, you can't pin anything.

Alex: So put the question in the contract. What are you running and will you tell us before it changes? So that's 3 things: an owner, graded questions, a register. Only one of those failures needed a specialist to fix. All 3 needed someone who knows the policy to notice.

Alex: So who does this work? Mostly people who know the business. In August, a partner at a small mortgage brokerage emailed me. He'd been quoted $15,000 for an off-the-shelf system. He built his own with a coding agent for under $1,000.

Alex: He has no engineering background. He told me consultants don't understand the business. I told him he didn't need us. Guests on this show have seen the same thing from the other end. Stéphane Malrait described a bank pilot with one of the biggest technology companies.

Alex: They had 500 data scientists, he said. What they lacked was expert knowledge of the financial product. And Daniel Flowe of the London Stock Exchange Group warned me about the opposite mistake. An AI governance process run mostly by non-technical people, he said, is in trouble. I think both are right.

Alex: The few who understand the pipeline have to be in the room. And without the people who know the product, they build the wrong thing. So here is what I'd do on Monday. One piece of context first. In October, Singapore's regulator finalised its AI guidelines.

Alex: It asks the board for an adequate understanding of AI, enough to oversee and to challenge. It stops well short of asking the board how the model works. Now, 3 questions for your team. One for each of the 3 things. Which model produced the last AI answer that someone here relied on?

Alex: That's the register. If that model changed tomorrow, how would we find out? That's the graded questions. And who decides what a correct answer is? That's the owner.

Alex: If nobody can answer those, a course on how RAG works won't fix it. The 3 things and the 3 questions are in the show notes. Here is what I can't settle. You need those few people who really understand the pipeline. If you can't judge their work yourself, how do you know you have the right ones?

Alex: Tell me how you'd answer that. If you're watching, leave it in the comments. If you're listening, send me a message on LinkedIn and follow the show wherever you're listening or watching.