speculative decoding
**Speculative Decoding**
**What is Speculative Decoding?**
Speculative decoding uses a smaller, faster "draft" model to generate candidate tokens, then verifies them in parallel with the larger "target" model. This can significantly reduce latency.
**How It Works**
**Standard Autoregressive**
```
Target Model: [token1] → [token2] → [token3] → [token4]
(slow) (slow) (slow) (slow)
Total: 4 sequential forward passes
```
**Speculative Decoding**
```
Draft Model: [t1, t2, t3, t4] (fast, one pass)
↓
Target Model: Verify all 4 in one parallel pass
↓
Accept: [t1, t2, t3] ✓, Reject: [t4] ✗
↓
Resume from [t3] with new speculation
```
**Key Components**
**Draft Model**
- Much smaller than target (e.g., 68M vs 7B)
- Same vocabulary/tokenizer
- Trained on similar data distribution
**Verification**
Target model runs single forward pass over all draft tokens:
- Accept if target agrees with draft
- Reject first disagreement, keep all before it
**Acceptance Rate**
| Factor | Impact on Acceptance |
|--------|---------------------|
| Draft quality | Higher quality → more accepted |
| Task difficulty | Easier tasks → more accepted |
| Draft size | Larger draft → more accurate |
| Speculation length | Longer → lower average acceptance |
Typical acceptance rates: 70-90% for well-matched pairs.
**Implementation in vLLM**
```bash
python -m vllm.entrypoints.openai.api_server
--model meta-llama/Llama-2-70b-chat-hf
--speculative-model meta-llama/Llama-2-7b-chat-hf
--num-speculative-tokens 5
```
**Self-Speculative Decoding**
Use earlier layers of the same model as draft:
- No separate draft model needed
- Slightly lower acceptance rate
- Simpler deployment
**Performance Gains**
| Setup | Speedup |
|-------|---------|
| 7B target + 68M draft | 2-3x |
| 70B target + 7B draft | 2-4x |
| Self-speculative (13B) | 1.5-2x |
**Trade-offs**
| Aspect | Consideration |
|--------|---------------|
| Memory | Need to load draft model too |
| Batching | Less effective with large batches |
| Task dependency | Works best for predictable outputs |
| Draft training | May need custom draft model |
Speculative decoding is most beneficial for latency-sensitive, low-batch scenarios.