phase transitions in training

**Phase Transitions in Training** are **sudden, discontinuous changes in model behavior during training** — analogous to physical phase transitions (ice → water), neural networks can undergo abrupt shifts in their learned representations, capabilities, or performance metrics. **Types of Training Phase Transitions** - **Grokking**: Sudden generalization after prolonged memorization. - **Capability Emergence**: Sudden appearance of new capabilities at certain model scales or training durations. - **Loss Spikes**: Sharp, temporary increases in loss followed by rapid improvement to a new, lower plateau. - **Representation Change**: Discontinuous reorganization of internal representations — features suddenly restructure. **Why It Matters** - **Predictability**: Phase transitions make model behavior hard to predict — capabilities appear suddenly. - **Scaling Laws**: Some capabilities emerge only at specific scales — phase transitions define threshold model sizes. - **Safety**: Sudden capability emergence complicates AI safety analysis — capabilities can appear without warning. **Phase Transitions** are **sudden leaps in learning** — discontinuous changes in model behavior that challenge smooth, predictable training assumptions.