position interpolation
**Position Interpolation (PI)** is a **technique for extending the context window of pretrained transformer models beyond their original training length by rescaling position indices to fit within the trained range** — instead of extrapolating to unseen position values (which causes catastrophic performance degradation), PI compresses the new longer sequence positions into the original range (e.g., mapping positions 0-8192 into the 0-4096 range the model was trained on), requiring only a short fine-tuning period to adapt the model to the rescaled positions.
**What Is Position Interpolation?**
- **Definition**: A context extension method (Meta Research, 2023) that modifies the Rotary Position Embedding (RoPE) frequencies by dividing position indices by a scaling factor — so a model trained with max position 4096 can handle 8192 positions by treating position 8192 as position 4096 in the original embedding space.
- **The Extrapolation Problem**: Transformers trained with positions 0-4096 have never seen position 4097 during training — when asked to process longer sequences, the position embeddings produce values outside the trained distribution, causing attention patterns to break down and quality to collapse.
- **Interpolation vs Extrapolation**: Extrapolation asks the model to handle position values it has never seen (guaranteed failure). Interpolation rescales new positions into the trained range — position 8192 becomes position 4096, position 4096 becomes position 2048 — all values the model has seen during training.
- **Scaling Factor**: For extending from length L to length L', the scaling factor is L'/L. Position index i becomes i × (L/L'). For 4K→8K extension: factor = 2, position 8000 → 4000.
**How Position Interpolation Works**
- **Original RoPE**: Position i gets frequency θ_j = i × base^(-2j/d) for each dimension j.
- **PI-Modified RoPE**: Position i gets frequency θ_j = (i / scale) × base^(-2j/d) — dividing by the scale factor compresses all positions into the original range.
- **Fine-Tuning**: After rescaling, a short fine-tuning period (1000-2000 steps on long-context data) adapts the model to the compressed position spacing — the model learns that positions are now more densely packed.
- **Minimal Quality Loss**: PI preserves most of the model's original capabilities — perplexity on short sequences increases slightly due to the denser position spacing, but long-context performance is dramatically better than extrapolation.
**Context Extension Methods Comparison**
| Method | Approach | Fine-Tuning | Quality | Complexity |
|--------|---------|------------|---------|-----------|
| Position Interpolation | Scale positions down | 1K-2K steps | Good | Simple |
| YaRN | Frequency-aware scaling | 400-1K steps | Better | Medium |
| NTK-Aware Scaling | Adjust RoPE base frequency | Minimal | Good | Simple |
| ALiBi | Linear attention bias | None (built-in) | Good | Architecture change |
| LongRoPE | Progressive extension | Multi-stage | Excellent | Complex |
**Position interpolation is the elegant context extension technique that stretches the ruler rather than reading past its end** — by rescaling position indices to fit within the trained range, PI enables pretrained models to handle 2-8× longer sequences with minimal fine-tuning, solving the context length limitation that previously required expensive retraining from scratch.