Decision Models vs. LLMs
Transcript
Made automatically from the video's sound, so it may contain mistakes. Click a line to play from there.
If you're wiring AI into your software right now, well, you've probably noticed those skyrocketing API bills and latency bottlenecks that just won't seem to budge. Welcome to this explainer. Today we're going to break down the sudden explosion of what the industry is calling decision models.
We'll cut straight through the marketing hype using hard benchmark data. You see, we're right in the middle of a massive shift in how AI is deployed in modern architecture. And to build efficient systems, it's absolutely vital we understand what's actually happening under the hood.
Okay, let's dive into our roadmap. One, what are decision models? Two, decision models versus LLMs. Three, the benchmark reality check.
Four, limitations and shadow AI. And five, the engineering decision checklist. This flow is going to take us seamlessly from the core mechanics all the way to a practical framework you can use with your engineering team today.
So when should you actually use an AI decision model instead of a large language model? Honestly, this is the multi-million pound question right now. We're seeing highly specialized new tools like TypeSafe's GEV, the OpenWaight Model Layer, and the OpenAI decisions API dropping within literally weeks of each other.
It's forcing us to ask where on earth they fit into our existing software stacks. Now, it's incredibly easy to assume they're just the next iteration of the LLMs we already have. You know, maybe just a bit faster or a bit cheaper.
But actually, scratch that, that would be a crucial mistake. They represent a totally different architectural mindset. Let's jump into section one. What are decision models?
The shift from generation to classification. Let's define exactly what we're dealing with here, because as I just mentioned, these are not just smaller, cheaper language models. Here's the formal definition, and this distinction is incredibly important.
A decision model is an AI model that scores a fixed set of possible answers in a single pass. It returns a typed choice, a probability, or a score, rather than generating text token by token.
So, what does that actually mean for us? Well, the crucial point is that it is structurally impossible for this type of model to return a rambling creative sentence, or to hallucinate some random fourth option to a three-option question.
When you use a traditional LLM to say classify a customer support ticket, it generates its answer one single token at a time from its entire vast vocabulary. A decision model rips that creativity right out. You declare the exact parameters up front, and it evaluates the input in one single highly efficient computational pass.
To really go our heads around this, it helps to look at the psychological framework behind the tech. Borrowing from the Nobel Prize-winning psychologist Daniel Kahneman, think of this as system one versus system two thinking.
System one is your gut reaction. It's fast, automated, strictly bounded. System two is deep thinking, slow, deliberate, open-ended. Decision models, they're your software's system one.
They give you rapid, intuitive judgments for routing information. But if you need a fully drafted email, a reasoned plan, or complex conversational context, well, that remains a system two job. You're still going to need an LLM for that.
Moving on to section two. Decision models versus LLMs. The mechanics and economics of routing. Let's look at how they stack up mechanically, because this structural difference completely flips the economics of running AI in production.
Yes, you're reading that right. Zero. Absolutely nothing. Because decision models don't generate conversational prose.
The output tokens for tools like JEV and OpenAI's decisions API cost absolutely zero. You're only paying for the input tokens. Now, if you're in software engineering, you know about hot paths, those critical high-volume processes that run thousands of times an hour.
On a hot path, every single millisecond and every fraction of a penny adds up drastically. When you're making thousands of routing decisions a day, cutting out that text generation cost is literally a game changer for your monthly API bill.
But take a look at the trade-off here. You get massive speed and cost savings, sure, but you completely lose the reasoning trace. Because a decision model evaluates its options in a single forward pass without generating step-by-step text.
There is no chain of thought for you to audit later. You get the final answer, and a confidence score, but you don't get the why. If your compliance team knocks on your door wanting to know exactly why the system rejected a user's application, a decision model simply cannot give you that contextual explanation.
Right, section three. The benchmark reality check. Separating the hype from the hard data. Let's see how this actually translates to real world performance.
We definitely shouldn't just swallow the vendor marketing hype. So let's look at some independent numbers. Independent testing from the University of Technology Sydney confirms that for narrow, repetitive edge service decisions, these models are an absolute triumph.
We're talking about a nearly 81% reduction in cost per correct decision, and a nearly 65% reduction in median latency compared to traditional LLMs. To put that into perspective, at enterprise scale, those fractions of a second and fractions of a penny compound massively.
It saves companies vast amounts of overhead on routine data sorting. But hold on a second. Look closely at this red hat benchmark on adversarial prompt injection. That's basically when malicious users try to trick the AI into breaking its rules or leaking data.
In this scenario, a well-tuned LLM, acting as a judge, specifically the QN 35 billion parameter model, actually beat the decision model in both speed and accuracy. And even more strikingly, older, pre-trained predictive models like Doberta blew them both out of the water on latency, clocking in at a blistering 54.1 milliseconds.
This proves decision models aren't some magic bullet that automatically wins. For highly nuanced tasks requiring deep pattern matching, older specialized models or robust LLMs absolutely still hold the crown. The researchers at UTS also made it very clear where the breaking point lies.
A wide contract, meaning a complex prompt where you're asking the AI to evaluate dozens of variables or extract loads of different fields at once, well that causes the decision model's advantage to completely evaporate. They are spectacular at asking one or two highly targeted questions, but the moment you overload the request, you lose all the speed and cost benefits that made you switch away from an LLM in the first place.
Which brings us to section four, limitations and shadow AI. What goes wrong when you wire them in blindly? Before you enthusiastically go adding one of these to your stack, we really need to explore the structural traps and governance risks.
Because they lack that audit trail we discussed, and because they can still confidently hallucinate a bad score, you should never use these models anywhere that requires fully reproducible, rule-based logic or human-in-the-loop explanations. And I really want to emphasize this third point.
The format is deterministic, but the content is not. What that means is you'll always get a properly formatted score back, say a number between 1 and 10, but the exact same input might get a 7 today and an 8 tomorrow. It's not mathematically perfectly reproducible.
For strict compliance systems, that is a total nightmare. Finally, section 5, the engineering decision checklist, a framework for your architecture. Let's wrap up this explainer with a definitive playbook.
How do you actually decide when to swap out an LLM for a decision model? Run through this checklist with your team. Ask yourselves one, is the answer space strictly bounded? Two, is the core volume extremely high?
Three, is latency on the critical user path? Four, do you just need a probability threshold to automate an action? And five, is the contract narrow, meaning under four fields? If you can confidently answer yes to all five of these, pulling that logic out of an LLM and into a decision model isn't just a cost saving measure, it's a fundamentally sound architectural choice.
But keep this quote in mind. It highlights a critical warning about what the industry's calling shadow AI. Because these API calls are so incredibly cheap and fast, individual developers are quietly wiring them into internal workflows over the weekend.
They're treating them like simple lines of code rather than complex AI models. Suddenly you've got a hidden probability threshold deciding what gets auto approved in your system, with zero documentation, no architecture review, and no team awareness.
That is a massive, massive compliance risk. So I'll leave you with this to ponder. When your system makes its next 10,000 automated decisions, will you have an audit trail to explain why things happened, or will you just have a black box?
Keep that governance question firmly in mind the next time your team debates their AI architecture. Decision models are phenomenal tools when used for the right, narrow jobs, but they demand real engineering discipline.
Thank you so much for joining this explainer.