multi token prediction

**Multi-Token Prediction** is **the training and inference technique that predicts multiple future tokens simultaneously rather than one token at a time** — enabling parallel decoding that generates 2-4 tokens per forward pass, reducing inference latency by 40-60% while maintaining generation quality, with training benefits including improved sample efficiency and better long-range modeling. **Multi-Token Prediction Training:** - **Multiple Prediction Heads**: add N prediction heads to model; head i predicts token at position t+i given context up to t; typically N=2-8 heads; shared backbone, separate output layers - **Training Objective**: L = Σ(i=1 to N) w_i × CrossEntropy(pred_i, target_{t+i}); weights w_i typically decrease with i (w_1=1.0, w_2=0.5, w_3=0.25); balances near and far predictions - **Auxiliary Task**: multi-token prediction acts as auxiliary task during training; improves representations; better long-range dependencies; 1-3% perplexity improvement even for single-token generation - **Computational Cost**: N× output layers but shared backbone; training cost increase 10-20%; acceptable for inference speedup and quality improvements **Inference with Multi-Token Prediction:** - **Parallel Generation**: at step t, predict tokens t+1, t+2, ..., t+N; verify predictions using standard autoregressive model; accept correct predictions; similar to speculative decoding but self-contained - **Verification**: compute logits for positions t+1 to t+N in single forward pass; check if multi-token predictions match top-k of verified distribution; accept matching tokens - **Acceptance Rate**: typically 40-70% for 2-token prediction, 20-40% for 4-token; depends on task and model quality; higher for repetitive text, lower for creative generation - **Speedup**: expected tokens per step = 1 + α_2 + α_2×α_3 + ... where α_i is acceptance rate for token i; typical speedup 1.5-2.5× for N=4 **Jacobi Decoding:** - **Fixed-Point Iteration**: treat autoregressive generation as fixed-point problem; iterate: x^{(k+1)} = f(x^{(k)}) where f is model prediction; converges to autoregressive solution - **Parallel Updates**: update all positions simultaneously; x_t^{(k+1)} = argmax P(x_t | x_{