A simple decision framework for navigating the landscape of LLMs, decision models, and custom trained machine learning models
In late 2022, ChatGPT arrived and a whole class of problems that required months of engineering suddenly became solvable in a matter of days. Many companies ramped up their investments in AI, and buzzwords such as “RAG”, “fine-tuning” and “proprietary models” floated through meeting rooms with a marvelous aura of ambition, confusion, and pressure not to fall behind. ML engineers who had their established ways of cleaning data, watching loss curves, comparing baselines, and tuning hyperparameters drastically shifted their work routines to building agentic workflows by calling APIs.
At the time of writing, in October 2026, there is still a sense of confusion: ML engineers who have successfully deployed ML models over the past 10 years are falling behind quick movers, leadership excitedly jumps from fine-tuning LLMs to RAG solutions to AI agents, and PoCs that never make it into production are being showcased to satisfy shareholder pressure.
The purpose of this article is to clarify confusion in this space with a problem-focused framework. I start with the fundamental assumption that the purpose of implementing a technology is to solve a problem. This isn’t always the case: sometimes an organization starts working with a technology simply to build up competence. However, this article is aimed at decision makers and engineers using technology to address a specific use case. Assuming this use case requires machine intelligence, I provide a simple framework that helps identify what technology to try out first. The emphasis is on simple: in many real-world scenarios, certain constraints don’t allow the optimal choice, or an ML model might just not achieve the desired performance. This is why organizations and engineers will likely end up experimenting with different setups before finding the optimal one.
Fine-tuning LLMs isn’t mentioned in this framework, but it’s not ignored. Fine-tuning LLMs is something I get asked about a lot, probably because it’s state of the art and a demonstration of technical sophistication. I fine-tuned LLMs during my master’s thesis to improve a company sector classification model, and it was a fun challenge to tackle. I don’t explicitly exclude fine-tuning LLMs in this framework as I count them as a subset of training ML models. The skill set needed is comparable to training ML models, the difference being that it requires more sophistication and resources, which is why most ML engineers would probably experiment with other ML architectures first. In some cases, fine-tuning LLMs can also be valuable for a custom chat model – but I would argue that in most cases, LLMs with a RAG setup will achieve satisfying performance at much lower engineering costs. Similarly, AI agents are included in the LLM (with RAG) answers, since they mostly consist of LLMs that run in a harness and are able to call tools.
The decision framework
A. Do you want a chat model, a generic agent, or text output?
If the product is a conversation, an agent that has to handle open-ended tasks, or anything whose output is free-form language, such as summaries, drafts, explanations, or answers to arbitrary questions, use an LLM. This is exactly what they’re built for, and no amount of feature engineering will get a gradient-boosted tree to write a decent email.
If, on the other hand, you’re building a specialized component that maps an input to a bounded output, such as a class, a score, a ranking, or a forecast, keep going. Most of the interesting trade-offs live there.
B. Do you have structured proprietary data?
If you don’t need a chat model or agent, the next thing to check is your data. Public LLMs were pre-trained on the public internet; they’ve never seen your transactions, your sensor logs, or your users’ behavior. The question is what shape your proprietary data is in. 
No structured data: If you have no proprietary data, or only unstructured data such as documents, start with pre-trained models. For simple tasks, or as baselines for complex tasks, start experimenting with decision models, such as TypeSafe AI’s Jev. They are generalizable models that are cheap to deploy and require little engineering effort.
Reach for an LLM when the task needs open-ended answers, tool calling, or higher-level reasoning where the decision model doesn’t succeed. If your knowledge lives in documents such as wikis, contracts, tickets, or manuals, add RAG. It’s cheaper than fine-tuning, easy to keep up to date, and lets you cite sources. An optimal solution can also be a hybrid: let the decision model handle the routine calls and escalate the uncertain ones to an LLM.
Structured proprietary data: Proprietary data, such as labeled examples, time series, structured customer behavior, etc. is what makes a custom model possible. Training works particularly well with large quantities of data, think of around 10,000 input–answer pairs, but depending on the complexity of the problem you may need far fewer or far more. Whether training is worth the effort depends on the next three questions.
A note on decision models: the category is very new. Jev launched in early access in September 2026, and the speed, cost, and accuracy figures so far are TypeSafe’s own. I assume that decision models will soon become better at higher-level reasoning and might be able to call tools, which would allow them to be used more widely.
C. Do you need answers reliably in under ~250 ms?
The word that matters is reliably. Average latency is easy to quote; the 90th percentile is what breaks your checkout flow or your ad auction.
Hosted LLM APIs typically return a first token in a few hundred milliseconds to a couple of seconds, and a complete response takes longer, since output is generated token by token. Reasoning models can take many seconds. Tail latency also moves with provider load, and rate limits can turn a slow response into no response at all. Even decision models don’t comfortably clear the bar: the most prominent example, TypeSafe AI’s Jev, reports responses in the range of roughly 70 to 500 ms.
Self-hosting an LLM on provisioned GPUs gives you more predictable latency, because you control the hardware and the batching. The catch is capacity: it’s fixed, so traffic spikes queue up and latency climbs, and adding GPU capacity takes minutes rather than milliseconds. You end up paying for peak capacity around the clock.
A well-built classical model, on the other hand, often answers in single-digit milliseconds on a CPU, scales horizontally on cheap hardware, and behaves the same at 3 a.m. and on Black Friday.
D. API spend higher than what an ML team would cost?
LLM pricing is per token, so cost grows linearly with traffic. A purpose-built model has a large fixed cost (the team, the training pipeline, monitoring) and a tiny marginal cost per prediction. At some volume the lines cross.
A back-of-the-envelope example: 50 M calls a day at around 500 tokens per call is 25 billion tokens a day. Even at a cheap rate of $0.10 per million tokens, that’s about $2,500 a day, or roughly $0.9 M a year, before retries, longer prompts, or a pricier model. That’s already in the range of a small, dedicated ML team, and the model they build will usually also be faster.
The chart assumes an in-house model costs about $0.9 M a year to build and run. Swap in your own prices and team costs to find your break-even point.
E. Do you need a calibrated model or a fixed false-positive rate?
A model is calibrated when its scores mean what they say: among all cases scored 0.8, about 80% are actually positive. Calibration is what lets you set a threshold for a fixed false-positive rate, price risk, or combine the score with other signals.
LLMs can also provide confidence scores, either by asking the LLM to state a confidence level with its answer or by using decision models, and for many use cases this suffices. However, confidences obtained in this way often don’t meet the calibration standards required for sensitive production systems. You can mitigate some of this with prompt engineering, sampling multiple answers, or post-hoc calibration, but it’s an uphill battle. A classifier trained on your proprietary data can be calibrated with standard techniques, gives the same output for the same input, and can be evaluated with precision-recall curves on your own held-out data.
No to C, D, and E: decision model, then LLM, then ML
If you have proprietary data but latency, volume, and calibration don’t force your hand, start with the quickest path to a working system: a decision model for bounded decisions, and an LLM where the decision model falls short. Use your proprietary data as a test set to benchmark both.
That being said, there are many cases in which you will end up experimenting with ML models anyway. One example is time-series forecasting, where decision models and LLMs aren’t a good fit. Pre-trained time-series models such as Chronos or TimesFM make a quick baseline, but they often still trail models trained on domain-specific data, especially when you have a long history and additional signals such as prices, promotions, or weather. If performance is crucial, you will likely conclude that training a specialized time-series model is worth the effort.
When to break the rules
This framework is deliberately kept simple. Real systems are often hybrids: a cheap classifier or decision model screens everything and escalates the uncertain cases to an LLM, or an LLM labels data that you then use to train a fast, small model. Treat the five questions as a way to start the conversation with your team.
About the author
NDSML Summit
Valentin Buchner is Chief Architect at Chapter Two, the music-royalty platform where he designs the ML systems and autonomous agents that help investors value music rights at scale. He spent his career deploying large language models in high-stakes domains—from private equity to healthcare to creative IP. Before Chapter Two, Valentin engineered LLM systems on EQT’s Motherbrain team, using AI to enrich and query Knowledge Graphs and to map companies to industry sectors. His research spans retrieval-augmented generation, applied NLP, and Fair ML, and was published at top-tier ML conferences and Journals. Valentin’s focus is turning frontier AI research into production systems people can trust – architecting the infrastructure that lets autonomous agents make consequential decisions.
*The views and opinions expressed by the author do not necessarily state or reflect the views or positions of Hyperight.com or any entities they represent.
