mechanistic interpretability

**Mechanistic Interpretability** is the **research program that aims to reverse-engineer the internal computations of trained neural networks into human-understandable algorithms — identifying the specific circuits (subsets of neurons, attention heads, and their connections) that implement identifiable computational steps like "copy the subject token" or "suppress repeated outputs"**. **Why Mechanistic Interpretability Differs from Feature Attribution** Feature attribution methods (saliency maps, SHAP, LIME) explain which inputs matter for an output but not how the model processes them internally. Mechanistic interpretability digs inside the model to find the algorithms — the specific sequence of attention patterns and MLP transformations that convert input tokens into output logits. **Core Techniques** - **Logit Lens / Tuned Lens**: Applies the model's unembedding matrix to intermediate residual stream states at each layer, revealing what the model "believes" at each processing stage. An early layer might show the raw token identity; middle layers show the emerging semantic interpretation; late layers show the final prediction. - **Activation Patching (Causal Tracing)**: Runs the model on a clean input and a corrupted input simultaneously. At each layer, the clean activation for specific components is patched into the corrupted run. If patching a particular attention head restores the correct output, that head is causally responsible for that computation. - **Circuit Discovery**: Identifies minimal subnetworks (circuits) that are necessary and sufficient for a specific behavior. The "Indirect Object Identification" circuit in GPT-2 Small was reverse-engineered to show exactly how 26 attention heads collaborate across layers to perform the task "When Mary and John went to the store, John gave the bag to → Mary." **What Has Been Found** - **Induction Heads**: Pairs of attention heads (one in an early layer, one later) that implement in-context copying — the fundamental mechanism behind in-context learning in transformers. - **Superposition**: Networks represent more features than they have neurons by encoding features as nearly-orthogonal directions in activation space, making individual neuron interpretation misleading. - **Privileged Basis**: Some neurons do correspond to interpretable features, but most meaningful computation occurs in linear combinations of neurons (directions in activation space). **Limitations** Mechanistic interpretability has only fully reverse-engineered tiny models (1-2 layer transformers) or specific narrow circuits in larger models. Scaling to frontier models with hundreds of billions of parameters and emergent capabilities remains an open and potentially intractable challenge. Mechanistic Interpretability is **the deepest level of understanding we can pursue for neural networks** — seeking not just what they do or which inputs matter, but the exact algorithms they learned and why those algorithms sometimes fail in dangerous ways.