Every evaluation of AI running in-house runs into the same objection: if I run an open model on my own infrastructure, I end up with a worse model than whoever is calling the big provider's API. The objection is partly true. The mistake is not in the fact, it is in the comparison. People measure model against model, when what actually decides the question is model against task.
This is the second piece in the series because, once the question of where the data sits is settled, the objection that comes next in most Aptabit evaluations is about quality, and the honest answer starts by admitting that the gap is real.
The difference between the leaderboard and the task
Frontier models are better. They were trained with resources no single organization can assemble on its own, and that distance is exactly what the public benchmarks published by the model makers measure. A vendor who denies it loses the technical person in the room by the third sentence. What the gap is not, necessarily, is decisive for the work your company is going to put into production.
Those tests measure a ceiling of difficulty. The corporate work that AI handles well usually has a low ceiling. Classifying a document, pulling fields out of a form, answering from a closed body of knowledge: all of it has a predictable shape and a right answer that somebody inside the company already knows. On that kind of task, the question is not which model scores higher, it is whether the business unit can tell the two outputs apart on its own material.
That is the practical criterion. If the task has a closed universe, a stable format and somebody able to say whether the answer is right, it is worth measuring whether the gap still matters. If the task is open-ended and depends on knowledge that is not in your own records, the gap shows up as soon as real material enters the test.
Where the model gap really does show
A piece that stopped at the paragraph above would be marketing. There are at least three families of task where the difference between models weighs heavily, and where adjusting the prompt usually does not fix it.
Long, open-ended reasoning. Problems that require holding a chain of decisions together across many steps, with no single answer and no answer key, are where the larger models tend to pull away. Exploratory analysis, multi-stage planning, synthesis of contradictory material. If that is your case, the local model costs you quality, and the cost is visible.
Hard code and rare knowledge. Writing or debugging non-trivial code, and answering on subjects that depend on knowledge that appears rarely, remain frontier model territory. Here the reasonable decision is usually to use the external provider, knowing that the content of that flow leaves your network, which is the subject of the previous piece in this series.
Long-tail languages and domains. Smaller models tend to degrade in languages thinly represented in training and in highly specific technical vocabulary. The term comes out wrong and the error goes unnoticed by anyone outside that field. How far it degrades varies case by case and cannot be read off the size of the model, so this one does not get settled on paper. Only a test on your own material answers it.
There is also a cost that has nothing to do with quality. Running the model in-house trades the per-token invoice for installed compute and for people who know how to operate it. The bill does not disappear, it moves. The question to ask is who operates that capacity and at what cost, before the decision rather than after it.
How to build the evaluation that settles the choice
Regardless of whether the choice leads to Aptabit or somewhere else, these questions tell you whether the evaluation was actually done:
- Did the cases come from my real records, the malformed ones included, or were they picked to demo well?
- Was the standard for a correct answer written down before anyone saw the outputs?
- Did the people grading the outputs know which model produced which?
- Were response time and cost of operation measured alongside quality?
- Will the test be repeated at every change of model or version?
The second is the easiest one to skip, and without it the evaluation tends to confirm whatever someone already wanted confirmed.
Why this piece follows the first
Our site says the platform runs local or cloud models, with no dependency on a single vendor or a single model. That is not a compatibility list. It is the consequence of treating the model as a replaceable part, chosen together with the agent and made explicit flow by flow whenever the destination is an external provider. Treat the model as the center of the product and you tie that product to one vendor's ceiling and to whatever that vendor decides about price, availability and terms of use. Treat it as a part and all you need is for it to be good enough for the task, and replaceable when it stops being good enough.
Going to the data, rather than asking the data to come to us, forces us to admit that not every model fits where the data lives. A later piece in this series deals with where the intelligence runs and what each arrangement charges for that.