superposition hypothesis

**Superposition hypothesis** is the **proposal that neural networks represent many features in shared dimensions by overlapping them rather than allocating one dimension per feature** - it explains how models can encode rich information with limited representational capacity. **What Is Superposition hypothesis?** - **Definition**: Features are packed into the same neurons or directions with partial interference. - **Motivation**: Dense models face pressure to represent more concepts than available clean axes. - **Interpretability Impact**: Explains prevalence of polysemantic units and mixed activations. - **Modeling**: Analyzed through sparse coding and feature dictionary frameworks. **Why Superposition hypothesis Matters** - **Theory Value**: Provides coherent explanation for observed representation entanglement. - **Method Design**: Guides development of feature extraction tools that untangle overlaps. - **Editing Safety**: Highlights risk of naive neuron interventions causing unintended collateral changes. - **Scalability Insight**: Suggests why larger models still exhibit mixed internal features. - **Research Direction**: Motivates sparse feature spaces as interpretability targets. **How It Is Used in Practice** - **Feature Extraction**: Use sparse autoencoders to test whether mixed units decompose into cleaner features. - **Interference Analysis**: Measure behavior overlap when candidate features co-activate. - **Model Comparison**: Evaluate superposition patterns across scales and architectures. Superposition hypothesis is **a key theoretical lens for understanding compressed internal representations** - superposition hypothesis is useful when paired with empirical decomposition and causal behavior testing.