The Infinite-Width Limit is a theoretical idealization in deep learning where the number of neurons in each hidden layer is taken to infinity — revealing that at this limit, randomly initialized neural networks become Gaussian processes, and gradient descent training becomes kernel regression in the Neural Tangent Kernel space — providing tractable mathematical models of neural network behavior that yield convergence guarantees, generalization bounds, and insights into scaling laws — while simultaneously highlighting that practical neural networks operate away from this limit, relying on finite-width feature learning that the infinite-width regime cannot capture.

What Happens at Infinite Width?

• Gaussian Process at Initialization: As hidden layer width n → ∞ (with independent random parameter initialization), by the Central Limit Theorem, the pre-activation distribution at each layer becomes Gaussian — and the function computed by the network becomes a Gaussian Process (GP) with covariance determined by the activation function and architecture.
• NTK Freezes During Training: As shown by NTK theory (Jacot et al., 2018), as width → ∞ trained with small learning rates, the Neural Tangent Kernel remains constant throughout training. Training dynamics simplify to linear kernel regression.
• No Bad Local Minima: In the infinite-width limit with overparameterization, gradient descent converges to a global minimum — the loss landscape becomes convex in function space.
• No Feature Learning: In the kernel regime, the network's internal representations do not change — only the output head weights (effectively) change. The network does not learn progressively better features; it performs fixed-basis function approximation.

Mathematical Framework

Practical Relevance and Limitations

Where the limit helps:

• Initialization Design: Infinite-width analysis motivates proper weight initialization (e.g., He initialization for ReLU, LeCun for tanh) to ensure stable signal propagation and full-rank NTK at training start.
• Architecture Comparison: Comparing infinite-width GP/NTK kernels of different architectures provides insight into their inductive biases before training.
• Neural Scaling Theory: Infinite-width limit is the starting point for understanding how performance scales with width — corrections at finite width produce scaling law models.
• Bayesian Deep Learning: Infinite-width GP correspondence enables exact posterior inference tractable for small datasets.

Where the limit fails:

• Feature Learning: Real transformer and CNN performance relies on learning increasingly abstract and task-relevant representations — absent at infinite width.
• Sparse Representations: Finite-width networks develop sparse features; infinite-width representations are dense Gaussian.
• Generalization on Large Data: Kernel methods (infinite-width equivalent) often underperform finite-width networks on large-scale tasks — evidence they lack the inductive biases arising from finite-width training dynamics.
• Emergent Capabilities: The emergent capabilities of large language models (in-context learning, chain-of-thought reasoning) have no analog in the infinite-width regime.

Research Frontiers

• Mean-Field Theory: Studies the 1/n corrections to the infinite-width limit — capturing first-order feature learning effects.
• Tensor Programs (Greg Yang): A unified framework computing the limiting behavior of any architecture as width → ∞, enabling systematic analysis of Transformers, LSTMs, and normalization layers.
• Maximal Update Parameterization (muP): Derived from infinite-width analysis — enables training hyperparameters (learning rate, initialization) to transfer cleanly from small to large width, used in practice for scaling up LLMs efficiently.

The Infinite-Width Limit is the theoretical microscope for deep learning — an idealized mathematical lens that, while not accurately describing production neural networks, reveals the structural principles governing convergence, generalization, and architectural inductive biases, grounding practical design decisions in rigorous theory.