Speculative Decoding

What is Speculative Decoding? Speculative decoding uses a smaller, faster "draft" model to generate candidate tokens, then verifies them in parallel with the larger "target" model. This can significantly reduce latency.

How It Works

Standard Autoregressive

Target Model: [token1] → [token2] → [token3] → [token4]
              (slow)     (slow)     (slow)     (slow)
Total: 4 sequential forward passes

Speculative Decoding

Draft Model: [t1, t2, t3, t4]  (fast, one pass)
                ↓
Target Model: Verify all 4 in one parallel pass
                ↓
Accept: [t1, t2, t3] ✓, Reject: [t4] ✗
                ↓
Resume from [t3] with new speculation

Key Components

Draft Model

• Much smaller than target (e.g., 68M vs 7B)
• Same vocabulary/tokenizer
• Trained on similar data distribution

Verification Target model runs single forward pass over all draft tokens:

• Accept if target agrees with draft
• Reject first disagreement, keep all before it

Acceptance Rate

Typical acceptance rates: 70-90% for well-matched pairs.

Implementation in vLLM

python -m vllm.entrypoints.openai.api_server
    --model meta-llama/Llama-2-70b-chat-hf
    --speculative-model meta-llama/Llama-2-7b-chat-hf
    --num-speculative-tokens 5

Self-Speculative Decoding Use earlier layers of the same model as draft:

• No separate draft model needed
• Slightly lower acceptance rate
• Simpler deployment

Performance Gains

Trade-offs

Speculative decoding is most beneficial for latency-sensitive, low-batch scenarios.