grokking

**Grokking** is a **training phenomenon where a model suddenly generalizes long after memorizing the training data** — the model first achieves perfect training accuracy (memorization), then after many more training steps, test accuracy suddenly jumps from near-random to near-perfect, exhibiting delayed generalization. **Grokking Characteristics** - **Memorization First**: Training loss drops to zero quickly — the model memorizes all training examples. - **Delayed Generalization**: Test accuracy remains at chance for many epochs after memorization. - **Phase Transition**: Generalization appears suddenly — a sharp, discontinuous improvement in test accuracy. - **Weight Decay**: Grokking is strongly influenced by regularization — weight decay encourages the transition from memorization to generalization. **Why It Matters** - **Understanding**: Challenges the assumption that generalization happens gradually alongside training loss reduction. - **Training Duration**: Models may need training far beyond overfitting to achieve generalization — premature stopping can miss grokking. - **Mechanistic**: Research reveals grokking involves learning structured, generalizable algorithms that replace memorized lookup tables. **Grokking** is **generalization after memorization** — the surprising phenomenon where models learn to generalize long after perfectly memorizing their training data.