neural tangent kernel

**Neural Tangent Kernel (NTK)** is a **theoretical framework that describes the training dynamics of infinitely wide neural networks** — showing that in the infinite-width limit, neural networks behave like linear models in a fixed feature space defined by the kernel at initialization. **What Is the NTK?** - **Definition**: $Theta(x, x') = abla_ heta f(x, heta)^T abla_ heta f(x', heta)$ where $f$ is the network output. - **Key Result**: In the infinite-width limit, the NTK is constant during training. - **Implication**: Training dynamics become equivalent to kernel regression with the NTK. - **Paper**: Jacot, Gabriel & Hongler (2018). **Why It Matters** - **Theory**: Provides the first rigorous characterization of when and why neural network training converges. - **Lazy Training**: In the NTK regime, weights barely change from initialization (lazy training). - **Limitation**: Real networks operate in the feature learning regime, not the lazy regime — NTK describes the easier, less interesting case. **NTK** is **the theoretical microscope on neural network training** — revealing the elegant mathematics hidden in the dynamics of gradient descent.