fourier features
**Fourier Features** are a technique for improving the ability of neural networks to learn high-frequency functions by mapping low-dimensional input coordinates through sinusoidal functions before feeding them to the network. The mapping γ(x) = [sin(2π·B·x), cos(2π·B·x)] (where B is a frequency matrix) lifts inputs to a higher-dimensional space where high-frequency patterns become learnable, overcoming the spectral bias of standard neural networks.
**Why Fourier Features Matter in AI/ML:**
Fourier features solved the **spectral bias problem** for coordinate-based neural networks, proving that a simple positional encoding with sinusoidal functions enables standard MLPs to learn signals with arbitrary frequency content—the theoretical foundation for positional encodings in NeRF and Transformers.
• **Spectral bias** — Standard MLPs with ReLU activations are biased toward learning low-frequency functions: they learn smooth, slowly varying functions first and struggle with sharp edges and fine details; Fourier features inject high-frequency basis functions directly into the input
• **Random Fourier Features** — Sampling B from a Gaussian N(0, σ²I) with standard deviation σ controls the frequency range; larger σ enables higher frequencies but can cause training instability; the bandwidth σ is the key hyperparameter controlling the frequency-accuracy tradeoff
• **Deterministic frequency bands** — NeRF-style positional encoding uses fixed, logarithmically spaced frequencies: γ(x) = [sin(2⁰πx), cos(2⁰πx), ..., sin(2^(L-1)πx), cos(2^(L-1)πx)] with L determining the maximum frequency; this deterministic approach avoids the randomness of random Fourier features
• **Neural Tangent Kernel (NTK) theory** — Tancik et al. (2020) proved that Fourier features manipulate the NTK of the network, enabling it to have support at higher frequencies; without Fourier features, the NTK is concentrated at low frequencies, explaining spectral bias
• **Multi-resolution hash encoding** — Instant-NGP extends the concept with learned, multi-resolution hash-based feature grids that provide adaptive spatial frequency encoding, achieving NeRF-quality results in seconds rather than hours
| Encoding Type | Frequencies | Learnable | Training Speed |
|--------------|------------|-----------|----------------|
| No encoding (raw coords) | None | N/A | Fast (but low quality) |
| Sinusoidal (NeRF-style) | Log-spaced, fixed | No | Moderate |
| Random Fourier Features | Gaussian-sampled | No | Moderate |
| Learned Fourier Features | Initialized, then learned | Yes | Moderate |
| Hash Encoding (Instant-NGP) | Multi-resolution grids | Yes | Very fast |
| Gaussian Encoding | Input-dependent bandwidths | Yes | Moderate |
**Fourier features are the theoretical foundation for enabling neural networks to represent high-frequency signals, providing the mathematical bridge (via NTK theory) between input encoding and learnable frequency content that underlies positional encodings in NeRFs, Transformers, and all coordinate-based neural representations.**