speculative decoding
**Speculative Decoding** is an **LLM inference acceleration technique that uses a small draft model to propose multiple tokens simultaneously, verified in parallel by the target model** — achieving 2-4x speedup without changing model quality.
**The Core Problem**
- Autoregressive LLM generation is sequential: one token at a time.
- Each forward pass through a 70B+ model takes ~100ms on a GPU.
- The GPU is severely underutilized — most computation is memory-bandwidth bound.
- Solution: Generate multiple tokens per target model forward pass.
**How Speculative Decoding Works**
1. **Draft Phase**: A small model (3B, 7B) generates K candidate tokens autoregressively.
2. **Verify Phase**: The large target model processes all K tokens in ONE forward pass (parallel).
3. **Accept/Reject**: Accept tokens where target model agrees with draft; reject the first disagreement.
4. **Correction**: Sample from the corrected distribution at the first rejection point.
5. **Result**: On average, 3-4 tokens accepted per target model forward pass.
**Why It Works**
- The verify step is nearly free — a forward pass processing K tokens costs only slightly more than 1 token for memory-bound models.
- The small draft model produces correct tokens most of the time for easy/predictable parts of the text.
**Variants**
- **Self-Speculation / MEDUSA**: Train additional "heads" on the target model itself as draft.
- **SpecTr**: Use multiple draft models; choose the best candidates.
- **Prompt Lookup Decoding**: Draft from the input prompt itself (fast, no extra model).
**Typical Speedups**
| Task | Speedup |
|------|---------|
| Code generation | 2.5-4x |
| Mathematical reasoning | 2-3x |
| Open-ended chat | 1.5-2.5x |
Speculative decoding is **a near-free inference speedup** — widely adopted in production LLM serving systems including vLLM, TGI, and Google's production inference.