TypeSafe’s Jev took off by returning decisions directly instead of writing answers for software to parse. For the small judgments an application makes all day, a choice and a probability are often more useful than another paragraph of text. As LLMs keep growing, latency becomes harder to ignore, and Jev’s speed made the appeal obvious. We wanted to bring that approach to any LLM, so developers could use the models they already have.
Consider a support ticket: the application needs to decide which team should handle it, whether the customer is threatening to leave, and how urgently someone should respond. The model needs to understand the ticket, but the application only needs the decisions and some indication of how confident they are. Generating JSON works, but it adds token-by-token generation to a job whose possible answers are already known.
How Kev works
Kev converts an LLM into a model that answers those questions directly. You supply the document, questions, and answer options at runtime, and it returns decisions with probabilities. We started with Qwen3 and converted its 1.7B, 4B, and 8B models using the same training hyperparameters. The converted model runs on its own, and changing the questions or options does not require training another classifier.
The conversion trains LoRA adapters and a shared pointer head that scores the options supplied with each question. Training uses both labeled examples and probability distributions collected from the parent, then merges the adapters into the exported model. One detail that matters is how we collect those distributions: LLMs can change their answers when you reorder the options, so we rotate unordered options through every position and combine the results before training. The converted model learns from the corrected targets without having to repeat those rotations at inference.
Speed and accuracy
On our 276-question showcase, the converted 4B model scored 84.1%, compared with 86.6% for its parent generating JSON with reasoning enabled. In a separate timing workload with eight questions about a 512-token document, it took 69 milliseconds against the parent’s 13.75 seconds. The parent took about 200 times as long for a 2.5-point accuracy advantage on the quality panel. Without reasoning, the parent’s accuracy fell to 76.1%, and its latency was still 1.32 seconds.
Decision accuracy and request latency
Avoiding answer generation accounts for part of the speedup, but sharing the document across questions also matters. Kev processes the document once, with a separate attention branch for each question, instead of reading the same document again for every decision. At 1.7B, a request with a 2,048-token document and 32 questions took 116 milliseconds in one forward pass. The parent took 2.47 seconds using direct one-token scoring, which already avoids generating full answers but processes the document separately for each question. That comparison shows a roughly 21-fold latency reduction even against a non-generative baseline.
Latency for 32 questions
Qwen3-1.7B · 2,048-token state · lower is better
Generalization and confidence
We also wanted to know whether conversion preserved the ability to answer questions outside its training tasks. On 300 questions from PAWS, QNLI, and Emotion, the 1.7B conversion trailed its parent’s direct scoring, 70.0% to 74.7%; the 4B tied at 75.7%; and the 8B was slightly ahead, 78.0% to 77.3%. A further 700-question test at 1.7B, covering knowledge questions, sentiment, ratings, and topic classification, scored 77.3% against the parent’s 70.4%. These results support using the model beyond a single trained task, although the smaller model’s loss on the first panel is a reason to check it on the workload you intend to use.
We calibrated the converted models on a separate data partition and measured probability quality against the gold answers. On the matched showcase questions, calibration error was about 0.05–0.06, compared with 0.21–0.36 for the parent’s raw scoring probabilities. Reordering options changed the converted models’ answers in 3.1–4.4% of cases, against 7.5–22.6% for raw parent scoring. Those probabilities give applications a way to send uncertain decisions for review, with thresholds checked against their own data.
Conversion cost and getting started
There is an upfront training cost: collecting teacher targets and training took 49–65 minutes per model on an RTX PRO 6000 Blackwell, plus preparation, calibration, and export. The resulting models keep the full backbone, so conversion does not make the weights substantially smaller. The savings are in inference: at batch size 32, the 4B conversion delivered 126 decisions per second against 30 for JSON generation without reasoning, while using 9.4 GB of peak GPU memory instead of 16.2 GB.
Kev is available on GitHub, with conversion recipes and an editable support-ticket demo. Qwen3 is the first supported backend; the framework is built to add others. Follow the quick start to convert a model and try your own questions through Python, the CLI, or the HTTP API. Pretrained downloads will follow on Hugging Face.
Data and measurement conditions
Accuracy and latency
Accuracy uses the same 276-question showcase for all systems. Timing is a separate 512-token, eight-question workload; values are median request latency.
| Parent | Kev accuracy | Parent JSON, thinking off | Parent JSON, thinking on | Kev latency | JSON latency, thinking off | JSON latency, thinking on |
|---|---|---|---|---|---|---|
| Qwen3-1.7B | 75.7% | 62.0% | 78.3% | 39.2 ms | 838 ms | 11.55 s |
| Qwen3-4B | 84.1% | 76.1% | 86.6% | 68.8 ms | 1,316 ms | 13.75 s |
| Qwen3-8B | 82.6% | 80.8% | 84.4% | 116.6 ms | 1,408 ms | 20.60 s |
Latency for 32 questions
Qwen3-1.7B; 2,048 state tokens and 32 questions. The JSON parent runs without reasoning.
| System | Median latency for 32 questions | Forward passes per request |
|---|---|---|
| Kev, packed | 116 ms | 1 |
| Parent, one-token scoring | 2,467 ms | 32 |
| Parent, cyclic scoring | 12,259 ms | 128 |
| Parent, JSON generation without reasoning | 4,136 ms | 297 on average |
Batch-32 throughput and memory
Eight questions per request and 32 requests per batch. Throughput counts individual decisions. The JSON parent runs without reasoning; peak memory includes resident weights.
| Parent | Kev decisions/s | Parent JSON decisions/s | Kev peak GPU memory | Parent JSON peak GPU memory |
|---|---|---|---|---|
| 1.7B | 329.9 | 67.6 | 4.36 GB | 9.45 GB |
| 4B | 125.7 | 29.7 | 9.43 GB | 16.18 GB |
| 8B | 73.1 | 23.6 | 16.91 GB | 25.45 GB |
Held-out and unseen-source accuracy
The held-out comparison uses the common 2,707 eligible questions. Unseen sources are PAWS, QNLI and Emotion, with 100 questions each. These sources were absent from conversion training.
| Parent | Kev, held-out test | Parent one-token, held-out test | Kev, unseen sources | Parent one-token, unseen sources |
|---|---|---|---|---|
| 1.7B | 75.2% | 62.4% | 70.0% | 74.7% |
| 4B | 81.5% | 75.1% | 75.7% | 75.7% |
| 8B | 82.3% | 75.4% | 78.0% | 77.3% |
Additional unseen sources at 1.7B
Qwen3-1.7B, 100 questions per source. These are sampled panels, not full benchmark scores. Majority is the most common gold label within each source sample. Ten unparsed JSON answers count as wrong. For TweetEval, the JSON parent predicted offensive for 89 of 100 tweets, against 16 gold positives; that source had no parsing failures.
| Source | Majority | Kev | Parent one-token | Parent cyclic | Parent JSON, thinking off |
|---|---|---|---|---|---|
| MMLU | 29% | 50% | 48% | 51% | 51% |
| ARC-Challenge | 32% | 76% | 68% | 76% | 79% |
| SciQ | 28% | 95% | 91% | 91% | 92% |
| TweetEval offensive | 84% | 86% | 78% | 85% | 27% |
| IMDB sentiment | 54% | 95% | 89% | 84% | 88% |
| Amazon stars | 24% | 43% | 30% | 30% | 33% |
| DBpedia-14 | 12% | 96% | 89% | 94% | 96% |
| All 700 | — | 77.3% | 70.4% | 73.0% | 66.6% |
Probability quality
Canonical option order on the same 254 eligible showcase questions. Kev outputs are calibrated; parent one-token outputs are raw. ECE is expected calibration error, NLL is negative log likelihood; lower ECE, NLL and Brier are better.
| Parent | Kev ECE / NLL / Brier | Parent one-token ECE / NLL / Brier |
|---|---|---|
| 1.7B | 0.060 / 0.591 / 0.341 | 0.358 / 4.218 / 0.726 |
| 4B | 0.051 / 0.429 / 0.235 | 0.212 / 2.710 / 0.432 |
| 8B | 0.054 / 0.406 / 0.234 | 0.216 / 3.030 / 0.435 |
Conversion cost
Teacher collection plus training only; dataset preparation, calibration and export add time. One conversion per size, with the same LoRA and pointer-head hyperparameters. The 8B training backbone used BF16; 1.7B and 4B used FP32. The 1.7B GPU was shared for part of training.
| Parent | Teacher collection | Training | Combined |
|---|---|---|---|
| 1.7B | 158 s | 2,793 s | 49 min |
| 4B | 365 s | 3,177 s | 59 min |
| 8B | 624 s | 3,265 s | 65 min |
Option-order sensitivity
Reordering choice options changed Kev’s answers in 4.4%, 4.4% and 3.1% of cases for 1.7B, 4B and 8B respectively. Raw parent scoring changed in 22.6%, 7.9% and 7.5%; cyclic scoring changed in 3.1%, 3.1% and 1.6%.
Corpus and artifacts
The training corpus contains 8,119 records and 10,697 questions. Teacher distributions cover 9,697 questions; 1,000 use gold labels alone. Complete model artifacts are 3.46, 8.06 and 15.16 GB. All backbone layers are retained.
Measurement details
The showcase contains 276 questions from held-out examples of task families represented in training. JSON generation and converted inference support all 276. The direct label-token scorer supports 254 of them, excluding 22 questions whose 77 options exceed its 26-letter label scheme; the probability table uses that same 254-question subset for both systems. The larger held-out comparison similarly uses the common 2,707 questions. The 700-question extended panel is a sample, rather than a full evaluation of the named public benchmarks.
All inference measurements use BF16 on an RTX PRO 6000 Blackwell, with PyTorch 2.11.0, Transformers 5.17.0, SDPA, and TF32 disabled. Ordinary latency profiles use five warm-up requests followed by 30 timed requests; reasoning-enabled JSON uses three warm-ups and ten timed requests. Timing includes request validation, encoding, inference, and output construction with device synchronization. Batch-32 throughput uses the median of five timed batches after one warm-up batch. Peak memory includes resident weights.
The 1.7B and 4B reasoning runs and the 1.7B extended run flagged GPU activity at their ends, so their reported ratios describe those recorded runs. These measurements use the reference Transformers implementation; they do not compare every system’s optimized production serving engine.
ECE uses 15 equal-width bins of top-answer confidence. NLL and Brier use the full predicted distributions. The calibration temperature was fitted on a separate partition, and all reported accuracy and probability calculations use canonical option order.
There was one conversion run per model size. Choice-order variants provide the separate order-sensitivity measurements. The benchmark profiles with one, eight, and 32 questions also change state length, so they do not isolate question count as the only variable.