We built a public benchmark for typed model decisions, tested a frozen 149M encoder with twenty small classifier heads on it, compared the result with TypeSafe's Jev, and fixed two bugs in the classifier library along the way.
Much of what software asks a language model for is a decision rather than prose: whether a ticket is about billing or a refund, whether a security alert needs a human to look at it, or how urgent something is on a scale the caller has already defined. The usual approach is to have the model write a paragraph and then parse an answer out of it.
TypeSafe ships a model, Jev, built for that job. It takes unstructured state and returns typed decisions with probabilities attached. There are three kinds: a yes/no question returns a probability, a choice returns a distribution over labels, and a score returns a distribution over the levels of a rubric. A single call answers all the questions asked about one piece of state.
We could not find a public benchmark for this kind of output, so we built one and tested the smallest model we could think of on it.
What one case looks like
Each case is a piece of state with questions about it: a customer thread with the account behind it, a security alert with the machine's history, or a vendor invoice with its purchase order and delivery note.
Each case has five questions mixing the three kinds. Every option comes with a written description, since the descriptions are part of the input; a bare list of label names would be a different and easier task.
The gold answers are probability distributions. We sample a teacher model three times per case and average its three distributions instead of taking a majority vote, so the gold is sharp where the teacher is consistent and soft where it disagrees with itself. We score both the label and the distribution, because the probability attached to a typed decision is part of the answer.
The test set covers four workflows, with 400 cases and 2,000 decisions, all held out. Training data comes from a separate generation run with a different seed.
The benchmark we had to throw away first
Our first version of the benchmark was broken, and it took us a while to see it.
It had 5,000 cases but only 105 distinct input strings, each of which appeared in all three splits. Four of its five decisions had been sampled independently of the input, so no model could beat the base rate. None did, and for a short time we read the fact that every model scored about the same as a result.
The replacement checks for that failure explicitly. Each case is built from sampled latent factors, and an audit measures the mutual information between those factors and the gold labels as a fraction of label entropy. The broken version scored near zero on this measure. The published set ranges from 0.119 to 0.850 across its twenty questions, no question collapses onto a single answer, and every state is distinct.
The audit is cheap to run, and we would recommend it to anyone building a synthetic benchmark.
The encoder baseline
The baseline uses Adaptive Classifier, a library that combines a prototype memory of examples with a small neural head, both on top of a frozen sentence encoder. Nothing in it is generative. We trained one classifier per question, twenty in all, on 300 cases each.
Two reference points help in reading the results.
A model that ignores the input and always answers with each question's training frequencies scores 0.470, which is the floor. A model that recovered every case's latent factors perfectly would score 0.704, which we estimate by fitting the factors to the gold directly. That ceiling is optimistic, since the factors contain information the text does not.
A 22M encoder at 22 milliseconds per case reaches 0.587, half of the distance from floor to ceiling. A 149M encoder at 349 milliseconds reaches 0.646, three quarters of that distance.
On the ordinal questions the bigger encoder lands within one rubric level of the teacher 93% of the time.
The larger encoder buys six points of accuracy for sixteen times the latency, and which trade-off is right depends on the application. Both run on a laptop.
We also wanted to know how a model built for this task would do. Once we had API access, on 18 September, we ran Jev on all 400 cases: 2,000 decisions with no errors, at 710 milliseconds per case and 16 cents in total.
Jev scores 0.727. That is above the factor ceiling, eight points above the larger encoder, and 0.008 below the teacher's self-agreement ceiling of 0.735, so on accuracy there is little headroom left on this benchmark for any model.
Jev had never seen these twenty question schemas. The encoders, by contrast, were trained on the same four workflows they are tested on and cannot answer questions outside them. The comparison is between one general model and a set of specialists, not a like-for-like quality ranking.
The distributional comparison came out differently.
| accuracy | KL from gold | |
|---|---|---|
| ModernBERT-base, 149M frozen | 0.646 | 0.223 |
| Jev 1.13.0 | 0.727 | 1.442 |
Jev picks the right label more often, but its KL divergence from the gold distribution is about six times the encoder's. It tends to put nearly all of its probability on one answer. The encoder is right less often, but when the three teacher samples split two to one, its probabilities split in a similar way.
This does not mean Jev is badly calibrated. Its expected calibration error is 0.144, with an overconfidence component of only 0.023, so its confidence mostly reflects how often it is right. The difference comes from how our gold is built: it encodes the spread across three teacher samples, and Jev has no way of knowing about that spread.
Which metric matters depends on how the output is used. If you route on the top label, Jev is eight points better. If you threshold on the probability, rank by it, or pass it to a downstream step that multiplies it, the 149M encoder, a 2024 model, tracks the teacher's uncertainty much more closely (KL 0.223 against 1.442).
One caveat: these scores measure agreement with our teacher, not correctness. Whenever Jev is right and our 4B-class teacher is wrong, Jev is penalised, so its true accuracy advantage is probably larger than 0.727 suggests.
Two bugs in the classifier library
The first run of the larger encoder scored 0.477, barely above the input-blind floor of 0.470. On one five-option question it scored 0.180, below the 0.200 expected by chance. Results that bad usually point to a bug rather than a weak model.
The library reduced every model's token embeddings to a single vector by taking the CLS token, regardless of the model. For most encoders that is the wrong choice. A masked language model does not train CLS as a sentence representation, and the sentence-transformer the library is most often used with, all-MiniLM-L6-v2, states in its config that it was trained for mean pooling.
On ordinary sentence pairs, CLS pooling gives unrelated texts, such as a sentence about a man playing guitar and one about an invoice paid late, a cosine similarity of 0.53. Mean pooling puts the same pair near zero and makes the gap between related and unrelated pairs 2.4 times wider.
The prototype memory does nearest-neighbour search over these vectors. When everything sits in a narrow band of high similarity, the prototypes stop discriminating between classes, which is why our tuning kept pushing their weight toward zero.
We found the second bug while trying to tune around the first. The library exposes prototype_weight and neural_weight to control how much each signal counts. Both were read from the config and stored on the object but never used: two prediction paths each hardcoded their own 0.7 and 0.3, so they could also disagree with each other once a caller changed the config. Several other keys, including epochs and batch_size, had the same problem.
Both bugs are fixed in release 0.2.0. Pooling now defaults to whatever the model's config says it was trained with, and classifiers saved before the change keep their old pooling when loaded, so upgrading does not silently change anyone's predictions.
The pooling fix alone took the larger encoder from 0.477 to 0.565.
Training on the distribution
The gold is a distribution, but the library trains on hard labels, so the soft target was being discarded during training.
A classifier that only accepts labels can still learn a distribution from replicated examples. We entered each case several times and split the copies across labels in proportion to its gold distribution, so a case the teacher called 70/30 became three examples of one label and one of the other. We used largest-remainder apportionment so that rounding does not lose any copies.
Accuracy barely moved, from 0.585 to 0.587. KL divergence from the gold fell by a third, the Brier score by 18%, calibration error by 31%, and error on the ordinal questions by 15%.
The top labels were already mostly right; what improved was the probability attached to them.
The cost is roughly seven times the training time, because the encoder embeds four identical copies of every text and the library does not cache embeddings. Training happens offline, so we accepted that.
What this does not show
The gold comes from a model of roughly 4B capability, sampled three times. A score here measures agreement with that teacher. It does not measure correctness, and the teacher is wrong sometimes: it missed a duplicate invoice whose number literally matched an earlier one.
This sets a ceiling. A fresh teacher sample agrees with gold built from the other samples only 73.5% of the time, so a score much above that would mean predicting the teacher's habits rather than answering the question. A model better than the teacher could score lower, because every case it gets right where the teacher was wrong counts against it.
The states are generated rather than drawn from real support queues or SIEM logs. Both encoders are specialists, trained on the workflows they are scored on and unable to answer questions outside them. A general model such as Jev accepts an arbitrary schema at request time, so comparing the two on accuracy alone favours the specialists.
Conclusion
A frozen 149M encoder with twenty small heads recovers three quarters of the learnable signal in a larger model's typed decisions, in about a third of a second per case on a laptop. The 22M version recovers half of it in 22 milliseconds. Jev is more accurate than either, while the encoder's probabilities track the gold distribution more closely.
For decisions that software makes constantly, often by calling a frontier model, that is a better trade-off than we expected.
The benchmark is public, and so are the library fixes and the reference scores, so new results can be compared against the floor, the ceiling, and the models reported here.