Speculative Decoding: The Same Distribution, Twice as Fast

Think about how you finish somebody else's sentence. They start "the capital of France is—" and you say "Paris" before they get there. You did not think hard. Most of a sentence is predictable, and only a few words genuinely require thought.

AI models do not get to skip the easy words. They write one word at a time, and each word needs a full pass through the entire model — every one of those billions of numbers consulted, whether the word was "Paris" or "the". A hundred words means a hundred full passes, one after another, each waiting for the last.

That looks like an unavoidable shape. It is not, and the reason is a quirk of the hardware.

1. The quirk

Producing one word and producing five words cost the chip almost exactly the same.

That sounds impossible until you see where the time actually goes. Producing a word means reading every number in the model out of memory. That reading is the slow part. The arithmetic done with those numbers, once they arrive, is comparatively trivial — and doing the arithmetic for five positions instead of one is nearly free, because the numbers were already on their way.

So every step of writing leaves a large amount of the chip's capability unused.

Speculative decoding is how you spend it.

2. The idea

Most words are easy. In the capital of France is Paris, only one word needed a big model. The rest are pinned down by grammar and context so firmly that a much smaller model would have written them identically.

But we pay full price for every word, because we cannot tell in advance which ones are hard.

So flip it. Let a cheap model guess several words ahead, then have the expensive model check them all at once — using the spare capacity that was going to waste anyway.

Yaniv Leviathan, Matan Kalman and Yossi Matias formalised this in Fast Inference from Transformers via Speculative Decoding, an ICML 2023 oral, reporting 2×–3× speedups on T5-XXL with no change to the output and no retraining or redesign. A team at DeepMind published a closely related method independently around the same time.

3. The loop

You need two models that share a vocabulary: a draft model $q$, small and fast, and the target model $p$, the one you actually want answers from. Each round:

1. Guess. Run the small model $\gamma$ times in a row, producing candidate words.
2. Check. Run the big model once over the whole candidate sequence. Because the words already exist, it can evaluate all the positions in parallel, and it tells you what it would have said at each one.
3. Accept or reject. Walk left to right through the candidates, applying the rule below. Keep the accepted run, throw away the rest.
4. Repeat.

If all $\gamma$ guesses survive, you produced $\gamma + 1$ words for the price of one big-model pass. If the first is rejected, you produced one, and wasted the small model's effort. Real text lands in between.

4. The rule that makes it exact

This is the part that makes it a result rather than a hopeful heuristic.

For each candidate word $x$ the small model proposed, accept it with probability

$$\min\!\left(1, \frac{p(x)}{q(x)}\right)$$

In plain terms: if the big model likes this word at least as much as the small one did, keep it, no question. If the big model likes it less, keep it only sometimes — in proportion to how much less.

When you reject, do not simply ask the big model to pick a replacement. That would skew the results, because you would only be replacing in exactly the cases where the small model over-reached. Instead, draw from what is left over:

$$p'(x) = \frac{\max\!\left(0,\, p(x) - q(x)\right)}{\sum_{x'} \max\!\left(0,\, p(x') - q(x')\right)}$$

That is the big model's own preferences with the share the small model already accounted for subtracted out, and the remainder rescaled to add up to one again.

Together, those two steps form a scheme whose output is exactly what you would have got from the big model alone. Not similar. Identical in distribution. Run it with a fixed random seed and you get the same text you would have got without any of this.

That guarantee is what separates this from every other speed-for-quality trade in the field. There is nothing to re-test, no benchmark to re-run, no risk of quiet degradation in the rare cases.

5. What decides the speedup

Two numbers, pulling against each other.

The acceptance rate $\alpha$ — how often the small model's guesses survive. Driven by how well the small model imitates the big one. A draft from the same family, trained on similar text, does well; an unrelated small model does badly.

The draft cost $c$ — the small model's cost relative to the big one. A model ten times smaller gives roughly $c = 0.1$.

The expected number of words per round, when guessing $\gamma$ ahead, is

$$\mathbb{E}[\text{words}] = \frac{1 - \alpha^{\gamma+1}}{1 - \alpha}$$

and the cost of that round is one big pass plus $\gamma$ small ones.

The tension is straightforward. Guessing further ahead raises the ceiling — but the last guess only gets used if every guess before it survived, so its value fades as $\alpha^\gamma$ while its cost is paid every single round. There is a sweet spot. It is usually between 3 and 7.

6. A worked example

Numbers make it concrete. Take a small model one-tenth the cost, so $c = 0.1$, agreeing with the big one 80% of the time, so $\alpha = 0.8$. Guess $\gamma = 4$ ahead.

Expected words per round:

$$\frac{1 - 0.8^{5}}{1 - 0.8} = \frac{1 - 0.328}{0.2} \approx 3.36$$

Cost per round, counting one big pass as 1: one check plus four guesses at 0.1 each, so $1 + 0.4 = 1.4$.

Words per unit of cost: $3.36 / 1.4 = 2.4$. Against a baseline of exactly one word per big pass, that is a 2.4× speedup — right in the range the paper reports.

Now change one thing at a time and the sensitivities appear.

Drop the acceptance rate to 0.6 and you get 1.67×. Drop it to 0.4 and you get 1.18× — most of the benefit is gone. Raise the guessing depth to 8 while keeping $\alpha = 0.8$ and you get 2.3× — very slightly worse than guessing 4 ahead, because those later guesses rarely survive long enough to be used.

Which is why the depth has a sweet spot rather than being "as deep as possible," and why the acceptance rate is the variable worth engineering. A better small model helps twice: it raises acceptance directly, and higher acceptance makes deeper guessing worthwhile.

7. Getting a small model without having one

Needing a compatible small model is a real obstacle, and most of the follow-up work exists to remove it.

Skip your own layers. Run only some of the big model's layers to form the guess. No second model needed, and the vocabulary matches by construction.

Medusa bolts extra prediction heads onto the big model, each trained to guess a word some distance ahead, producing candidates in a single pass with no separate guessing loop.

Copy from the prompt. The cheapest idea and it works better than it has any right to. For jobs where the answer heavily reuses the input — summarising, editing, translating a supplied document, answering from pasted documents — form your guesses by copying phrases from the prompt itself. No model, no training, and a high acceptance rate precisely because the answer really does reuse the input.

Guess a tree, not a line. Propose several alternative continuations arranged as branches and check them all in one pass, raising the odds that some branch survives.

8. What this looks like in production

You do not implement this yourself any more. The major serving systems offer it as a setting, and the interesting work is choosing the settings.

Pick the small model from the same family. The vocabularies must match exactly — the accept/reject rule compares the two models' opinions about the same word, so two models that chop text differently cannot be paired at all.

Measure the acceptance rate on your own traffic. Every serious implementation reports it. It is the single number telling you whether the feature is earning its keep, and it varies enormously by workload. Below about 0.5, the small model costs more than it saves.

Watch memory. Two models are loaded instead of one, and both hold conversation memory. On a tight deployment, the small model can reduce how many requests you can serve at once — and that loss may exceed the speed gain.

Turn it off under heavy load. The whole idea depends on spare capacity while writing. When the chip is already saturated serving many people at once, the small model's work competes with real requests instead of filling idle space. This technique is at its best for one person waiting on one answer — which is, conveniently, the interactive case.

9. Why it matters

Speculative decoding is one of the very few optimisations with nothing on the other side of the scale.

Shrinking numbers trades precision. Training a smaller model trades capability. Caching trades memory. Approximating attention trades exactness. Speculative decoding trades nothing — it spends idle capacity and hands back the identical result.

It exists for the same reason FlashAttention exists: somebody honestly accounted for what the hardware was doing. Writing was never limited by arithmetic. Once you notice that, the question stops being how to make each step cheaper, and becomes how to get more words out of a step you were paying for anyway.

The pattern generalises well past AI. Wherever a system is stuck on one resource and idle on another, the win is usually not efficiency. It is finding speculative work to fill the gap, plus a cheap way to check it.

Sources

• Leviathan, Kalman, Matias — Fast Inference from Transformers via Speculative Decoding, ICML 2023
• Chen, Borgeaud, Irving, Lespiau, Sifre, Jumper — Accelerating Large Language Model Decoding with Speculative Sampling, 2023
• Cai, Li, Geng, Peng, Lee, Chen, Dao — Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads, 2024