The TL;DR: Error-cancelled MTP makes a model’s built-in speculative drafter more useful at standard MTP training cost. In our 10M–59M models, drafted-token acceptance improves by 1.1–1.9 percentage points, with the same peak memory and training throughput within 0.4% of standard MTP. We found no next-token loss penalty relative to standard MTP at any tested size.
At 10M, its drafts are comparable to KL distillation. Explicit KL training uses 29–48% more peak memory and has 11–16% lower throughput across the two timed configurations. Error cancellation recovers the KL gradient for the draft head and shared backbone without adding a full-distribution distillation loss.
Read the preprint (PDF) for the full method and experiments. Developed with the help of Marvin, our industry-leading agentic co-scientist.
We improved the built-in drafter in multi-token prediction (MTP) without adding a distillation loss. Error-cancelled MTP increases drafted-token acceptance by 1.1–1.9 percentage points in our 10M–59M models, with training throughput within 0.4% of standard MTP and the same peak memory.
At 10M, its drafts are comparable to those trained with KL distillation. But KL used 29% more peak memory and had 11% lower throughput in our timing. In the larger-vocabulary timing, those overheads rose to 48% and 16%. The useful part of distillation can be recovered through a small change to the backward pass.
More draft tokens accepted
Training throughput within 0.4% of standard MTP, with the same peak memory.
What the draft should learn
Speculative decoding lets a cheap drafter propose tokens for the main model to check. MTP provides that drafter as a small head trained alongside the model. Standard MTP teaches it to predict the future token in the training text, but a useful draft needs to agree with its verifier. Those are different targets: the verifier can be wrong about the text and still reject a draft that was right.
The head’s training gradient separates into two terms: its disagreement with the verifier, and the verifier’s own error. Error-cancelled MTP removes the second term from the head and shared backbone. It leaves the forward computation, loss values and shared output-layer gradient unchanged.
The implementation adds a scaled, detached-minus-attached copy of the head state to the verifier state:
s = s + F.pad(c * (h.detach() - h), (0, 0, k, 0))
That addition is zero in the forward pass. During backpropagation, it sends the verifier-error term into the head with the opposite sign, cancelling it.1 The head and backbone receive the same gradients as a stopped-target KL loss, without explicitly computing that loss over the verifier’s full distribution.
Better drafts without the KL overhead
We trained Llama-style models from scratch on FineWeb-Edu, using one DeepSeek-V3-style MTP module at loss weight 0.3. At 10M–59M, error-cancelled MTP makes drafts 0.8–1.0 percentage points closer to the verifier in total variation, alongside the higher acceptance rates. We found no next-token loss penalty relative to standard MTP at any tested size. At 6M, the draft-quality difference was not resolved.
Training cost, side by side
10M model · 8K vocabulary
60M model · 50K vocabulary
Bars use shared linear scales: 0–400k tokens/s and 0–60 GiB.
The result also carries beyond the training domain. At 10M and above, draft acceptance improves on WikiText-2, PG-19, CNN/DailyMail and GSM8K. Code is noisier, with no clear acceptance gain across seeds. These experiments measure acceptance of one drafted token, not end-to-end generation speedup; the paper reports the complete results and follow-up analyses.
-
For head depth
k,Dheads,T-token windows and MTP loss weightλ,c = (λ/D)(T−1)/(T−k−1). This matches the two losses’ per-position weights so the verifier-error terms cancel. The paper gives the derivation and implementation checks. ↩