fire (functional interpolation for relative encoding)

**FIRE (Functional Interpolation for Relative Encoding)** is a positional encoding method for Transformers that represents relative position biases as a continuous, learned function rather than a discrete lookup table, enabling smooth generalization to unseen relative distances and improved length extrapolation. FIRE uses a small MLP to map continuous relative position values to attention biases, interpolating between trained positions and extrapolating beyond the training range. **Why FIRE Matters in AI/ML:** FIRE addresses the **length generalization limitation** of discrete relative position encodings (T5 bias, learned absolute) by representing position as a continuous function that can smoothly interpolate and extrapolate to distances not seen during training. • **Continuous position function** — FIRE learns a function f_θ: ℝ → ℝ (implemented as a small MLP) that maps relative distance |i-j| to an attention bias; unlike T5's discrete bins or learned absolute embeddings, this continuous representation defines a bias for any real-valued distance • **Progressive interpolation** — FIRE applies a learned transformation that normalizes relative positions to [0,1] before feeding to the MLP, enabling the network to gracefully handle positions beyond the training range by extrapolating the learned continuous function • **Smooth distance modeling** — The MLP produces smooth bias curves as a function of distance, naturally capturing the intuition that positions close together should have similar biases; this smoothness acts as an inductive bias that improves generalization • **Per-head functions** — Each attention head has its own position bias function f_θ^h, enabling different heads to specialize in different distance ranges (local vs. global attention patterns), similar to ALiBi's multi-scale slopes but with learned, nonlinear patterns • **Compatibility with other methods** — FIRE can be combined with RoPE or ALiBi: using RoPE for the rotational component and FIRE for additional learned relative biases, providing both the mathematical structure of rotations and the flexibility of learned position functions | Property | FIRE | T5 Relative Bias | ALiBi | RoPE | |----------|------|-----------------|-------|------| | Representation | Continuous MLP | Discrete bins | Fixed linear | Rotation angles | | Extrapolation | Good (smooth) | Poor (bucketed) | Excellent | Moderate | | Parameters | Small MLP per head | n_heads × n_bins | 0 | 0 | | Flexibility | Nonlinear, learned | Learned (bucketed) | Fixed linear | Fixed rotational | | Distance Model | Smooth function | Piecewise constant | Linear | Oscillatory | | Interpolation | Continuous | Bucketed | Continuous | Continuous | **FIRE advances positional encoding by representing relative position biases as continuous, learned functions that naturally interpolate and extrapolate across distances, providing smoother generalization than discrete position buckets and more flexibility than fixed linear biases, representing the evolution toward treating position encoding as a continuous signal processing problem.**