speculative decoding draft model
**Speculative Decoding** is the **inference acceleration technique that uses a smaller, faster "draft" model to generate multiple candidate tokens which are then verified in parallel by the larger target model — exploiting the observation that verification is much cheaper than generation for autoregressive models, achieving 2-3× inference speedup without any quality degradation because only tokens that the target model would have generated are accepted**.
**Why Speculative Decoding Works**
Autoregressive LLM inference generates one token at a time, each requiring a full forward pass through the model. The bottleneck is memory bandwidth (loading model weights for each token), not compute. A smaller draft model generates K candidate tokens in the time the target model generates 1. The target model then verifies all K candidates in a single forward pass (parallel verification), accepting the longest prefix of correct tokens.
**Algorithm**
1. **Draft Phase**: The draft model generates K tokens autoregressively (fast, small model — e.g., 1B parameters).
2. **Verify Phase**: The target model processes the original context + K draft tokens in a single forward pass, computing the probability distribution at each position.
3. **Accept/Reject**: Starting from the first draft token, accept if the target model's probability for that token meets the acceptance criterion (modified rejection sampling ensures the output distribution exactly matches the target model). Continue accepting until a token is rejected.
4. **Correction**: At the first rejected position, sample a new token from an adjusted distribution. Discard all subsequent draft tokens.
5. **Repeat**: The accepted tokens extend the context. Draft model continues from the new position.
**Acceptance Rate and Speedup**
If the draft model matches the target model well, most tokens are accepted. Typical acceptance rates: 70-90% for well-matched draft/target pairs. Expected tokens per target model forward pass: K×α/(1-α^K) + 1, where α is acceptance rate. At α=0.8, K=5: ~4 tokens per forward pass → ~3-4× speedup.
**Variants**
- **Self-Speculative Decoding**: Use the target model itself as the draft model by skipping layers (layer dropout) or using early exit. No separate draft model needed.
- **Medusa**: Add multiple prediction heads to the target model, each predicting different future token positions simultaneously. Verify all candidates in one forward pass using a tree attention mask. 2-3× speedup with a single model + lightweight heads.
- **EAGLE**: Uses a lightweight auto-regressive head that takes the target model's hidden states as context, generating draft tokens that closely match the target distribution. Higher acceptance rates than Medusa.
- **Lookahead Decoding**: Use n-gram caches from the model's own past generations to propose candidate continuations without a draft model.
**Requirements for Effective Speculation**
- **Draft-Target Alignment**: The draft model must approximate the target model's distribution well. Fine-tuning the draft model on the target model's outputs improves acceptance rate.
- **Latency Budget**: Draft generation + verification must be faster than sequential target generation. If the draft model is too slow or acceptance rate too low, speculation provides no benefit.
- **Batch Size 1 Focus**: Speculative decoding benefits latency (single-request) scenarios most. At high batch sizes, the target model is already compute-bound and speculation provides diminishing returns.
Speculative Decoding is **the algorithmic insight that transformed LLM inference from strictly sequential to partially parallel** — proving that a cheap approximation followed by parallel verification is faster than exact sequential generation, without sacrificing a single bit of output quality.