infinite-width limit
**The Infinite-Width Limit** is a **theoretical idealization in deep learning where the number of neurons in each hidden layer is taken to infinity — revealing that at this limit, randomly initialized neural networks become Gaussian processes, and gradient descent training becomes kernel regression in the Neural Tangent Kernel space — providing tractable mathematical models of neural network behavior that yield convergence guarantees, generalization bounds, and insights into scaling laws** — while simultaneously highlighting that practical neural networks operate away from this limit, relying on finite-width feature learning that the infinite-width regime cannot capture.
**What Happens at Infinite Width?**
- **Gaussian Process at Initialization**: As hidden layer width n → ∞ (with independent random parameter initialization), by the Central Limit Theorem, the pre-activation distribution at each layer becomes Gaussian — and the function computed by the network becomes a Gaussian Process (GP) with covariance determined by the activation function and architecture.
- **NTK Freezes During Training**: As shown by NTK theory (Jacot et al., 2018), as width → ∞ trained with small learning rates, the Neural Tangent Kernel remains constant throughout training. Training dynamics simplify to linear kernel regression.
- **No Bad Local Minima**: In the infinite-width limit with overparameterization, gradient descent converges to a global minimum — the loss landscape becomes convex in function space.
- **No Feature Learning**: In the kernel regime, the network's internal representations do not change — only the output head weights (effectively) change. The network does not learn progressively better features; it performs fixed-basis function approximation.
**Mathematical Framework**
| Quantity | Finite Width | Infinite Width |
|----------|-------------|----------------|
| **Pre-activations** | Correlated (non-Gaussian) | Independent Gaussians (CLT) |
| **Network at init** | Complex non-GP function | Exact Gaussian Process |
| **Training dynamics** | Nonlinear ODE in weight space | Linear ODE in function space (kernel regression) |
| **Feature representations** | Evolve (feature learning) | Fixed (no representation learning) |
| **Generalization** | Complex, architecture-dependent | RKHS norm regularization (kernel theory) |
**Practical Relevance and Limitations**
**Where the limit helps**:
- **Initialization Design**: Infinite-width analysis motivates proper weight initialization (e.g., He initialization for ReLU, LeCun for tanh) to ensure stable signal propagation and full-rank NTK at training start.
- **Architecture Comparison**: Comparing infinite-width GP/NTK kernels of different architectures provides insight into their inductive biases before training.
- **Neural Scaling Theory**: Infinite-width limit is the starting point for understanding how performance scales with width — corrections at finite width produce scaling law models.
- **Bayesian Deep Learning**: Infinite-width GP correspondence enables exact posterior inference tractable for small datasets.
**Where the limit fails**:
- **Feature Learning**: Real transformer and CNN performance relies on learning increasingly abstract and task-relevant representations — absent at infinite width.
- **Sparse Representations**: Finite-width networks develop sparse features; infinite-width representations are dense Gaussian.
- **Generalization on Large Data**: Kernel methods (infinite-width equivalent) often underperform finite-width networks on large-scale tasks — evidence they lack the inductive biases arising from finite-width training dynamics.
- **Emergent Capabilities**: The emergent capabilities of large language models (in-context learning, chain-of-thought reasoning) have no analog in the infinite-width regime.
**Research Frontiers**
- **Mean-Field Theory**: Studies the 1/n corrections to the infinite-width limit — capturing first-order feature learning effects.
- **Tensor Programs (Greg Yang)**: A unified framework computing the limiting behavior of any architecture as width → ∞, enabling systematic analysis of Transformers, LSTMs, and normalization layers.
- **Maximal Update Parameterization (muP)**: Derived from infinite-width analysis — enables training hyperparameters (learning rate, initialization) to transfer cleanly from small to large width, used in practice for scaling up LLMs efficiently.
The Infinite-Width Limit is **the theoretical microscope for deep learning** — an idealized mathematical lens that, while not accurately describing production neural networks, reveals the structural principles governing convergence, generalization, and architectural inductive biases, grounding practical design decisions in rigorous theory.