Jev vs LLMs: What Should Decide Fraud and Credit?
In this episode of The Curiosity Code podcast, host Alex Khomyakov examines whether the systems that decide fraud, credit, and payment routing in finance should run on small specialized models or large language models, using the newly launched $870 million-backed classifier Jev from startup TypeSafe as a case study. He argues the real dividing line is numbers versus text rather than small versus large, contending that money-moving decisions should stay on specialized, explainable models you own, while reserving LLMs only for text judgment calls where a cheap classifier is uncertain. Drawing on insights from industry practitioners Abhishek Taneja and Srishti Andreasen, Alex notes that Jev was beaten on calibration—the very thing it was sold on—in its first independent test.
Transcript
Alex: When you tap your card, a model decides whether the payment is fraud in less than a tenth of a second. A large language model can take several times that just to sort one sentence into a category. So should the systems that decide fraud, credit and payment routing run on small specialized models or on large language models? This autumn investors put $870 million behind one answer. A startup launched a model built for exactly these quick yes or no calls.
Alex: It cannot write a single word. And six days after it launched, independent researchers found a problem with the one thing it was sold on. Hi, I'm Alex, and this is Curiosity Code. No guest today, only me. One thing before I answer.
Alex: My companies build software for financial firms, and one of our products runs on a large language model. So I have no reason to be against them. The startup is called TypeSafe, and the model is called Jev. You give it a piece of text, a question, and a list of possible answers. It gives back a probability for each answer and nothing else.
Alex: Take a support email that says, I was charged twice. Is it a refund request? 90% yes. Your code sets a threshold. Above 90, it goes straight to the refund queue.
Alex: In the middle, a person looks at it. The company says the model is calibrated. When it says 80% it should be right about 80% of the time. The pitch is simple. Most of what software asks a language model is a yes or a no.
Alex: And finance is full of those. Is this payment fraud? Do we approve the loan? Which route does this payment take? Here is where I land.
Alex: The decisions that move money in finance never ran on LLMs, and they shouldn't start. So small against large is the wrong line. Draw it at numbers against text. Transactions go to specialized models you own and can explain. Text judgment calls go to a cheap classifier first, with an LLM for what it isn't sure about.
Alex: And the probability you haven't checked on your own data is only a number. The model sold on being calibrated was beaten on calibration in its first independent test by the models it wants to replace. I'm not the only one who lands near here. I asked a guest a version of this question two weeks ago, Abhishek Taneja runs AI for a major asset manager in Asia after two decades in retail banking. I asked him which kind of model he would trust for a decision about someone's money.
Alex: He said the traditional risk models are still tree-based models. He called them the workhorse of banking and financial services because they are very fast, they are deterministic and they are very stable. He has tested the newer foundation models built for tables of data and he was impressed. But he told me they are not ready for prime time because their explainability is still not there. So he runs them as challengers next to the old model, and the old model stays in production.
Alex: He also gave me the reason it differs across the industry. A regulator does not care if a hedge fund cannot explain its model in English. A bank has to explain its model in English. And one more thing from him that I keep coming back to. A card fraud model finds out within about three months whether it was right.
Alex: That fast feedback is what lets you check a model and retrain it. Another guest, Srishti Andreasen, told me that blunt fraud rules at many institutions flag far more good customers than bad ones. So the people who run these systems already chose small and specialized years ago. Both episodes are linked in the show notes. Start with the numbers side.
Alex: The decisions made on transaction data. Stripe says its fraud system decides in under 100 milliseconds. Mastercard says under 50. Adyen aims for 20. None of them says it runs on a language model in that path and Adyen has said plainly that one would not help.
Alex: The new model, Jev, takes between 70 and 500 milliseconds by its own maker's figures. So it does not fit there either. Now, there is a serious argument on the other side and it's about size. The models starting to beat the old specialists are big. Visa announced one in June.
Alex: It uses the same transformer design as a large language model, but it learns to predict the next transaction, not the next word. Stripe trained one on tens of billions of transactions and says it caught 64% more of one kind of card attack at large businesses. Revolut and Nubank have published similar work. So the frontier in payments is a large model, but it reads transactions, it doesn't read or write text, and bigger did not win anywhere. Revolut's own paper shows its big model doing far worse than its specialist on money laundering.
Alex: So, on the numbers side, the choice is between a specialist you own and a bigger specialist. A language model is not the race. Now the text side. Support messages, alerts an analyst has to read. Documents, notes on a case.
Alex: Here the small model has a real case, and it's cost. One study this year ran the same sorting job both ways. A small model trained for the task cost under $6 per million requests. A leading language model cost $276, and it was slower and less accurate on the job. But the claim that matters for risk is calibration, the idea that you can trust the percentage.
Alex: Here is the problem I mentioned at the start. Six days after Jev launched, two researchers published a test of it on 18 labelling tasks against 19 language models. Jev was about 44 times cheaper than the best language model on each task. It was also less accurate. And when Jev said it was at least 90% sure, it was right about 81% of the time.
Alex: Three large language models simply asked how confident they were, came out better calibrated than the model built for calibration. Several more independent tests since have found problems too. Finance has seen this before. After the pandemic, TransUnion found that borrowers whose credit scores had risen went on to default like borrowers scoring about 25 points lower on cards and personal loans. The scores still put people in the right order.
Alex: They just meant less than they said. One fraud company has tried Jev on real card fraud. In one fraud ring, it caught 27 of 29 fraudulent payments. Simple rules caught 26 and a large language model caught 18. But that is 29 cases with no count of good customers it would have stopped.
Alex: The first study also found something useful. Let the small model answer when it's confident and send the rest to the large one. That matched the large model alone at roughly a quarter to half the cost. So the recap. On numbers, finance already runs on specialists and the language model is too slow.
Alex: On text, small first and large second is cheaper and about as good. And nobody's percentage can be trusted until you have checked it on your own cases. The rules already treat the two halves of this question differently. In Europe, the AI Act puts credit scoring in its high risk category. Those duties now start in December 2027.
Alex: The same line of the law makes an exception for detecting financial fraud. So a fraud model and a lending model sit under different rules, even at the same firm. In the United States, a lender that turns you down still has to give you a specific reason. A model that returns a probability and nothing else does not hand you those reasons. That is the legal version of what my guest said about explaining a model in English and it's why the answer changes with who you are.
Alex: A fund can run a model that a lender cannot. So here is what I would do. First, list the decisions your systems make and mark each one as numbers or text. Leave the numbers ones on models you own, can explain and can retain. Second, for the text ones, put a small cheap model in front, set the threshold by what a mistake costs and send what it's unsure about to a large model or a person.
Alex: Third, before you trust a threshold, take a few hundred of your own cases where you know the answer. Check that 90% means 90%. Do that for the small model and the large one. Those three steps are in the show notes. Here is what I can't settle on.
Alex: In April, US bank regulators replaced their guidance on model risk and they left generative AI out of scope. A model like this one generates no text. But a security firm says it appears to be a language model underneath, and the company hasn't answered. So which rulebook is it under? I don't know, and the guidance does not say.
Alex: I'm talking to people who validate these models for a living in upcoming episodes and I will ask them. Tell me how you would answer it. If you're watching, leave it in the comments. If you're listening, send me a message on LinkedIn. And follow the show wherever you're listening or watching.
Alex: Bye-bye.