Speculative Decoding 2026: How Draft Models Make LLMs 2-3x Faster

LLM generation is sequential - one token at a time - which makes it slow. Speculative decoding breaks that assumption and gets 2-3x speedups. How does it work and when does it actually pay off?

📜 Table of Contents

• The Sequential Bottleneck
• The Trick in Three Steps
• The Numbers That Matter
• Where It Works Best
• Where It Does Not Help
• The Practical Setup (llama.cpp)
• The 2026 Position

The Sequential Bottleneck

Every token depends on the previous one, so generation cannot be parallelized directly. A 100-token answer needs 100 sequential model passes. Speculative decoding sidesteps this with a bet.

The Trick in Three Steps

1. Draft - a small, fast model proposes the next 4-8 tokens in one quick pass.
2. Verify - the big model checks all proposed tokens in a SINGLE parallel pass.
3. Accept or reject - accepted tokens are free (you generated several tokens for the cost of one big-model pass); the first rejected token is corrected and generation continues from there.

Because the big model confirms or rejects in parallel, total sequential steps drop dramatically. When the draft is right most of the time, you win.

The Numbers That Matter

• Acceptance rate: a well-matched draft model gets 60-80% of tokens accepted.
• Speedup: 1.5-3x typical on consumer GPUs; up to 2-3x on some workloads, more with aggressive drafts.
• The draft model must be small (1B or less) or the draft cost eats the savings.

Where It Works Best

• Long generations - the more tokens you generate, the more the savings compound.
• Code and structured output - predictable tokens are accepted more often.
• Local single-user inference - llama.cpp supports speculative decoding with a small draft GGUF; Medusa (35,681 stars) is a popular draft-head approach.
• Batch serving - vLLM (88,691 stars) has native speculative decoding for production.

Where It Does Not Help

• Very short responses (a 20-token answer saves little)
• When the draft model quality is far below the target (acceptance rate collapses)
• Memory-constrained setups: you now load TWO models

The Practical Setup (llama.cpp)

# main model + a 1B draft model
./build/bin/llama-cli -m main-model.gguf --model-draft draft-1b.gguf -p "Write an essay about" -n 200

The 2026 Position

Speculative decoding is now a default feature in llama.cpp, vLLM, and SGLang (31,628 stars) - you enable it with one flag. It is no longer exotic; it is the standard way to spend spare compute on latency.