Neural Tangent Kernel (NTK) Theory is a theoretical framework showing that infinitely wide neural networks trained with gradient descent behave exactly as kernel regression in a fixed function space defined by the NTK — where the kernel is fully determined by the network architecture and does not evolve during training — developed by Jacot, Gabriel, and Hongler (2018) as a breakthrough in deep learning theory that provides the first rigorous convergence guarantees for gradient descent on neural networks and a tractable mathematical model of training dynamics, sparking a decade of intensive theoretical research into finite-width corrections, feature learning, and the limits of the kernel regime.

What Is The Neural Tangent Kernel?

• Definition: The NTK K(x, x') at two inputs x and x' is defined as the inner product of the gradient of the network output with respect to its parameters: K(x, x') = ∇_θ f(x, θ) · ∇_θ f(x', θ), where the dot product is over all parameters.
• Infinite Width Limit: As the widths of all hidden layers approach infinity (with appropriate parameter scaling), the NTK K(x, x', θ) converges to a deterministic, architecture-dependent kernel K_∞(x, x') that is constant throughout training.
• Linear Dynamics: Under infinite width, the function f(x, θ_t) evolves linearly in function space: df/dt = -η K_∞(X, x) (f(X, θ_t) - y), where X is the training set and y are the targets.
• Kernel Regression Solution: The solution of this linear ODE is exactly kernel regression with kernel K_∞ — the network converges to the minimum-norm interpolating function in the reproducing kernel Hilbert space (RKHS) of K_∞.

Key Theoretical Results

Architecture-Specific NTKs

• Fully Connected NTK: Can be computed recursively layer by layer — the infinite-width FC NTK is a Gaussian process kernel with architecture-dependent covariance structure.
• Convolutional NTK (CNTK): Derived by Arora et al. (2019) — competitive with finite-width CNNs on CIFAR-10 in the pure kernel regression setting.
• Attention NTK: More complex but derivable — used to analyze the implicit bias of transformer training.

NTK Regime vs. Feature Learning Regime

The most important practical question NTK theory poses:

NTK theory describes networks in the "lazy" regime where weights barely move. Real neural networks operate in the "feature learning" (rich/mean-field) regime — where representation learning occurs. NTK is a theoretical idealization, not the operational regime of practical deep learning.

Impact and Ongoing Research

• Infinite-Width Neural Networks as GPs: At initialization (before training), infinite-width networks are Gaussian Processes — enabling Bayesian inference without MCMC.
• Finite-Width Corrections: Research computing the first-order corrections to NTK theory as width decreases — quantifying how feature learning departs from the kernel regime.
• Signal Propagation: NTK analysis guides weight initialization schemes — ensuring the NTK is full-rank at training start.
• Calibration: GP and NTK regression provides calibrated uncertainty estimates used in Bayesian deep learning.

Neural Tangent Kernel Theory is the first rigorous mathematical framework for understanding neural network optimization — its idealized infinite-width model provides provable convergence guarantees and motivates studying the deviations from kernel behavior that characterize the feature learning responsible for deep learning's practical power.