speculative decoding

Speculative decoding accelerates LLM inference by drafting multiple tokens then verifying in parallel. **Mechanism**: Small "draft" model generates k candidate tokens quickly, large "target" model verifies all k tokens in single forward pass, accept verified prefix and regenerate from first rejection. **Why it works**: Single forward pass through target model processes k tokens in roughly same time as 1 token (attention parallelizes). If draft accepts 70% of tokens on average, effective 2-3x speedup. **Draft model requirements**: Much smaller (10-100x fewer parameters), trained on similar data or distilled from target, fast enough that drafting overhead is minimal. **Variants**: Medusa adds multiple prediction heads to single model, self-speculative uses early exit layers, parallel decoding with candidates from different strategies. **Implementation**: Careful handling of probability distributions during verification, tree-structured speculation for multiple candidates. **Limitations**: Overhead if draft quality poor, memory for draft model, complex implementation. **Best use cases**: Latency-sensitive applications, when draft model available, sequences where patterns are predictable. Used in production by major LLM providers.