Speculative Decoding: How LLMs Write Tokens 3x Faster

Your GPU is not slow. It is bored.

When a large language model generates text one token at a time, the arithmetic is trivial and the memory traffic is brutal. To produce a single token, the hardware must stream every weight in the model from HBM into compute units, do a small amount of math, and throw the weights away. Then it does it again. On a 70B model in FP8, that is roughly 70 GB of reads per token. An H100 has about 3.35 TB/s of memory bandwidth, which caps you somewhere near 50 tokens per second no matter how many FLOPs are sitting idle.

Autoregressive decoding is memory-bandwidth bound, and the compute units are mostly parked. Speculative decoding is the trick that puts them to work — and it is now the default acceleration layer in vLLM, SGLang, and TensorRT-LLM rather than a research curiosity.

The core insight: verification is parallel, generation is not

Here is the asymmetry everything hangs on.

Generating N tokens requires N sequential forward passes. But checking whether a given sequence of N tokens is what the model would have produced requires exactly one forward pass, because the model can score all N positions in parallel. Generation is serial. Verification is embarrassingly parallel.

So: get some tokens cheaply from somewhere else, then verify them all at once with the real model.

That is the entire idea, introduced by Leviathan, Kalman, and Matias in Fast Inference from Transformers via Speculative Decoding (ICML 2023). Two models, a target model p and a small draft model q:

1. Draft phase — q autoregressively generates up to K candidate tokens. It is small, so this is cheap.
2. Verify phase — p scores all K candidates in a single forward pass, producing p(d_j) for every draft token.
3. Accept/reject — walk the draft left to right, accepting each token with probability α = min(1, p(d_j | x, d_<j) / q(d_j | x, d_<j)).
4. On the first rejection, sample a corrected token from an adjusted residual distribution and discard the rest of the draft.

If the draft was right about 5 tokens, you got 5 tokens for the price of roughly one target-model pass plus some small-model overhead.

Why this is lossless, not an approximation

This is the part that trips people up, and it is the reason speculative decoding is safe to turn on in production.

That accept/reject step is rejection sampling, and it is constructed so the output distribution is provably identical to sampling from the target model directly. Not similar. Identical. The original paper's contribution was not just "use a small model" — it was the novel sampling method that guarantees exact decoding from the large model while running the small one in parallel.

Speculative decoding does not trade quality for speed. It trades idle compute for speed. Quantization degrades your model; speculation does not. This is why it is the one inference optimization you can enable without re-running your evals.

The original T5X experiments showed 2X–3X acceleration with identical outputs, on off-the-shelf models, with no retraining and no architecture changes. Everything since has been a fight over one variable: how do you produce better drafts, more cheaply?

The family tree of draft generation

1. A smaller sibling model

The classic setup: serve Llama-3.1-8B as the draft for Llama-3.1-70B. It works, and it requires zero training. It also means loading and running a second full model, which eats VRAM and adds real latency per draft token. The draft model must also share the target's tokenizer — a constraint that quietly rules out most convenient pairings.

2. Medusa: bolt extra heads onto the model you have

Medusa (Cai, Li, Geng, Peng, Lee, Chen, and Dao — yes, the FlashAttention Tri Dao) discards the second model entirely. Instead it attaches additional decoding heads to the frozen backbone, each trained to predict a token further into the future, and verifies multiple candidate continuations at once through tree-based attention.

No separate model, no tokenizer mismatch, and the backbone stays untouched. The paper reports a 2.2×–3.6× speedup across a range of LLMs.

3. EAGLE: draft in feature space, not token space

EAGLE's insight was that a draft head predicting hidden states rather than tokens has a much easier job — the feature sequence is more regular and predictable than the token sequence. It performs autoregression at the feature level, reusing top-layer features from the target model.

Then it hit a wall. As EAGLE-3 (Li, Wei, Zhang, and Zhang; NeurIPS 2025) documents, scaling up draft-model training data gave EAGLE only limited improvement, and the authors traced the ceiling to the feature-prediction constraint itself.

EAGLE-3's fix is a reversal:

• Abandon feature prediction in favor of direct token prediction.
• Replace reliance on top-layer features with multi-layer feature fusion, via a technique the authors call training-time test.

The results: a speedup ratio up to 6.5×, roughly 1.4× better than EAGLE-2, and — the strategically important part — the draft model now benefits proportionally from more training data. In the SGLang framework, EAGLE-3 delivers a 1.38× throughput improvement at batch size 64, which matters because batch 64 is where most speculation stops paying off (more on that below).

4. MTP: train the model to speculate on itself

DeepSeek took a different route. Rather than bolting drafting on after the fact, Multi-Token Prediction is an auxiliary training objective baked into DeepSeek-V3, letting the model predict the next two tokens from the same hidden state.

Per the DeepSeek-V3 technical report, the acceptance rate of the second predicted token runs 85%–90% across generation topics, yielding roughly 1.8× TPS. That acceptance rate is the highest of any approach here, for an obvious reason: the drafter was trained jointly with the model it drafts for. There is no distribution mismatch to overcome.

This is why MTP heads are now shipped as part of the model release for several open-weight frontier models rather than being a serving-time add-on.

5. n-gram and suffix decoding: speculation without a model

The cheapest option is not a neural network at all. n-gram speculation searches the existing prompt and generated text for a matching prefix and proposes whatever followed it last time.

This sounds too crude to work, and for open-ended chat it mostly is. But for summarization, RAG with long retrieved context, code editing, and structured extraction — anywhere the output heavily quotes the input — the hit rate is startling. Zero training, zero extra VRAM, zero draft-model forward passes.

The vLLM docs are blunt about the tradeoff: model-based methods like EAGLE, MTP, draft models, PARD, and MLP give the best latency reduction, while n-gram and suffix decoding provide modest speedups without increasing workload during peak traffic. That second clause is the whole reason n-gram survives in production.

Comparison

Method	Extra training	Extra VRAM	Reported speedup	Best for
Draft model	None	Full second model	~2–3×	Quick wins, existing model families
Medusa	Head training	Small	2.2–3.6×	Self-contained deployments
EAGLE-3	Head training	Small	up to 6.5×	Maximum latency reduction
MTP	Built into pretraining	Minimal	~1.8× (V3)	Models that ship with MTP heads
n-gram / suffix	None	None	Modest	Input-heavy tasks, peak traffic

Speedups are as reported by each method's authors on their own hardware and task mix. They are not directly comparable across rows, and none of them will reproduce on your workload unchanged.

Where speculative decoding stops working

This is the section most tutorials skip, and it is the one that will bite you.

Speculative decoding spends compute to save memory bandwidth. That trade is only profitable while you have spare compute. At batch size 1, you have enormous amounts of it and speculation looks like magic. As batch size climbs, the GPU shifts from memory-bandwidth bound to compute bound — and now those extra draft forward passes and wider verification tensors are competing for the exact resource you ran out of.

The consequence: the relative gain from speculation shrinks monotonically as batch size grows, and past some crossover point it goes negative. You are paying for draft tokens out of a compute budget that verification needed.

The second failure mode is acceptance rate. If your drafter is bad, the target model spends its parallel verification pass evaluating tokens it will throw away. Practitioners commonly report real-world acceptance somewhere in the 0.6–0.8 range for instruction-following work; when it drops well below that, the verification overhead swamps the benefit and you end up slower than plain autoregressive decoding.

Both failure modes are why dynamic speculative decoding exists — adjusting or disabling speculation based on live batch pressure and observed acceptance. If you are running a serving fleet with variable load, this is not optional.

Turning it on in vLLM

The n-gram path takes four lines and needs no draft model:

from vllm import LLM, SamplingParams

llm = LLM(
    model="facebook/opt-6.7b",
    tensor_parallel_size=1,
    speculative_config={
        "method": "ngram",
        "num_speculative_tokens": 5,
        "prompt_lookup_max": 4,
    },
)

outputs = llm.generate(["The future of AI is"], SamplingParams(temperature=0.8, top_p=0.95))

A few rules from the vLLM documentation that will save you an afternoon:

• The model parameter inside speculative_config is required for eagle, eagle3, dflash, medusa, and draft_model. It is auto-resolved for ngram, suffix, and extract_hidden_states.
• For EAGLE-3, you must explicitly set "method": "eagle3". Leaving it as eagle will not silently upgrade you.
• EAGLE-based draft models must run without tensor parallelism — set draft_tensor_parallel_size to 1 in speculative_config. The target model can still use TP normally.
• num_speculative_tokens is the knob to tune first. Higher is not better: longer drafts mean more wasted work per rejection.

vLLM's own engineering blog documents gains of up to 2.8× from speculative decoding in the serving path — a reasonable expectation band for a well-tuned single-stream setup, not a promise for your batch-64 production fleet.

How to choose

Start with n-gram if your workload is summarization, RAG, code editing, or anything where output echoes input. It is free, it cannot cost you VRAM, and it degrades gracefully under load.

Use MTP if your model shipped with MTP heads. The acceptance rate is the best available because the drafter was co-trained; there is nothing to tune.

Invest in EAGLE-3 if you are latency-obsessed, serving at low-to-medium concurrency, and can afford to train a draft head. It is the strongest published option, and unlike its predecessor it keeps improving as you feed it more draft training data.

Reach for a plain draft model only as a stopgap. It is the easiest to stand up and the worst on memory.

And regardless of method: measure acceptance rate in production, not in a benchmark. It is the single number that determines whether speculation is helping you or quietly taxing you.

The Bottom Line

Speculative decoding is the rare optimization with no quality cost — the rejection-sampling step makes the output distribution mathematically identical to the target model's, which is why it went from ICML paper to default serving feature in three years.

But it is not free, and the marketing numbers hide the constraint. Every published speedup figure — 2.2×, 3.6×, 6.5× — is a low-batch number. Speculation buys latency with surplus compute, and a busy production server does not have surplus compute. The teams getting real value are the ones tracking acceptance rate as a first-class metric and letting the serving layer back off speculation when the batch fills up.

Turn it on. Then instrument it.