mechanistic

**Mechanistic Interpretability** is the **branch of AI safety and interpretability research that reverse-engineers neural networks by identifying the specific algorithms, circuits, and features that implement model behaviors** — pursuing complete, faithful understanding of how transformers compute rather than post-hoc approximations or correlational probes. **What Is Mechanistic Interpretability?** - **Definition**: The systematic effort to identify and understand the actual computational mechanisms inside neural networks — the specific neurons, attention heads, circuits, and features that causally produce observed model behaviors. - **Analogy**: Mechanistic interpretability is to neural networks what neuroscience is to brains — or more precisely, what reverse engineering is to compiled software. The goal is to reconstruct human-readable pseudocode from the network's weights. - **Key Methods**: Feature visualization, attention pattern analysis, activation patching (causal tracing), probing, sparse autoencoders, and circuit analysis. - **Primary Focus**: Transformer language models — particularly GPT-2, GPT-4, and Claude — where the architecture is well-understood and the stakes of understanding alignment are highest. **Why Mechanistic Interpretability Matters** - **AI Safety**: If we can understand what computations a model performs, we can verify whether it has learned deceptive behaviors, dangerous knowledge, or misaligned goals — rather than hoping alignment training worked correctly. - **Debugging**: Identify why models fail on specific inputs by tracing the computation that produced the failure — enabling targeted fixes rather than blind retraining. - **Alignment Verification**: Confirm that safety training actually removed harmful behaviors rather than merely suppressing them — mechanistic verification vs. behavioral testing. - **Scientific Understanding**: Build a true scientific theory of how neural networks learn and represent knowledge — foundational for the field of AI. - **Capability Prediction**: Understand what new behaviors emerge from scale before deploying larger models. **Core Concepts** **Features**: - The basic units of representation — directions in activation space that correspond to specific concepts. - A "banana feature" fires when the model processes banana-related text. - Features may be monosemantic (one concept per neuron) or polysemantic (multiple concepts per neuron — superposition). **Circuits**: - Subgraphs of the network (specific neurons + attention heads + weights) that implement a specific algorithm. - Discovered by following information flow from input to output for specific behaviors. - Example: The "indirect object identification" circuit in GPT-2 that identifies the indirect object in sentences like "John gave Mary the book." **Attention Heads**: - Transformers consist of attention heads that route information between token positions. - Heads have identifiable functions: some copy information, some attend to previous similar tokens (induction heads), some identify syntactic structure. **Key Discoveries in Mechanistic Interpretability** **Induction Heads (Anthropic, 2022)**: - Specific attention head pairs that implement in-context learning — searching for the pattern [A][B]...[A] and predicting [B]. - Formed during a sudden phase transition in training, coinciding with the emergence of in-context learning ability. - Suggests in-context learning has a specific, identifiable mechanical implementation in transformer weights. **Indirect Object Identification Circuit (Redwood Research / Anthropic)**: - Analyzed GPT-2's circuit for completing "John gave Mary the book; Mary gave [John]." - Identified 26 specific attention heads with specific roles: subject inhibition, name mover, backup name mover, S-inhibition, induction. - Complete causal account of a specific linguistic capability. **Grokking and Modular Arithmetic**: - Models trained on modular arithmetic exhibit "grokking" — sudden generalization after overfitting. - Mechanistic analysis revealed the model learned a specific Fourier frequency algorithm for modular arithmetic. **Superposition and Sparse Autoencoders**: - Models represent more features than dimensions by encoding features as nearly-orthogonal directions that overlap. - Sparse autoencoders decompose these overlapping representations into interpretable monosemantic features. **The Circuits Approach** **Step 1 — Identify a behavior**: "The model correctly identifies the indirect object in double-object constructions." **Step 2 — Activation Patching**: Systematically corrupt then restore activations at different network components to identify which are causally necessary. **Step 3 — Component Attribution**: Determine which attention heads, MLPs, and residual connections contribute most to the behavior. **Step 4 — Weight Inspection**: Directly inspect what those components compute from their weight matrices. **Step 5 — Reverse Engineering**: Formalize the discovered algorithm in pseudocode and verify it generalizes. **Research Organizations** - **Anthropic Interpretability Team**: Core circuit analysis work; Claude interpretability; sparse autoencoder research. - **Redwood Research**: IOI circuit; causal scrubbing for faithfulness testing. - **EleutherAI**: Open-source models + interpretability tooling (TransformerLens). - **Chris Olah (Google Brain / Anthropic)**: Pioneered feature visualization and circuit discovery. Mechanistic interpretability is **the scientific program to make AI systems as understandable as the circuits inside a computer** — as researchers scale circuit analysis from toy models to frontier AI, mechanistic interpretability promises to transform AI alignment from a behavioral art into an engineering discipline with formal verification of safety-critical properties.