Explainability

Explainability is a model's ability to have its decisions understood by a human — not just "it predicted X," but why it predicted X, in terms a person can actually reason about and trust or challenge. The field split early into two different goals that get conflated constantly: interpretability (the model's internal mechanism is simple enough to understand directly, like a short decision tree) versus explainability (the model stays a complex black box, but a separate method produces a human-readable explanation of its specific output after the fact). That distinction matters because the two approaches trade off against each other in opposite directions. At a high level, where explainability fits relative to a model's accuracy looks like this:

The accuracy-versus-interpretability tradeoff is real but not absolute, which is a frequently overstated point worth getting precisely right. A linear regression's coefficients are directly readable — "a $1,000 income increase predicts a 0.3% higher approval rate" — with zero translation step, but it can't capture nonlinear interactions a gradient-boosted tree or neural network can. For a long time this created a near-universal assumption that higher accuracy necessarily meant less interpretability; post-hoc explanation methods (SHAP, LIME, attention visualization, saliency maps) exist specifically to break that assumption by keeping the complex, accurate model and building a separate explanation layer on top of it, rather than sacrificing accuracy for a simpler model upfront.

Explainability became a legal requirement, not just a trust-building nicety, once regulation started demanding it directly. The EU's GDPR includes language widely interpreted as a "right to explanation" for automated decisions with significant effects (loan denial, hiring rejection); the EU AI Act goes further, requiring documented transparency and human oversight specifically for "high-risk" AI systems in areas like employment, credit scoring, and law enforcement. This regulatory pressure is the direct reason post-hoc explanation methods moved from academic research curiosities into production MLOps pipelines — a bank denying a loan based on a model's score now frequently needs to produce a SHAP-style breakdown of which features drove that specific decision, not because engineers wanted to build it, but because a regulator or a lawsuit can require it.

Mechanistic interpretability is the frontier research direction trying to solve a harder, different problem: not approximating why a model made one decision, but actually reverse-engineering what algorithm the model's internal weights implement. Post-hoc methods like SHAP produce a plausible-sounding explanation without ever confirming it reflects the model's true internal computation; mechanistic interpretability instead looks for actual "circuits" inside a trained network — specific groups of neurons and attention heads that implement an identifiable sub-algorithm, like a circuit found in small transformers that performs modular addition, or features found in larger models that appear to represent abstract concepts directly. This work is still mostly confined to smaller models and specific capabilities, because large production models' internal computation remains, in a very literal sense, not yet understood even by the people who trained them.

Read explainability as two genuinely different promises that both get called "explaining the model": interpretability promises the mechanism itself is simple enough to trust directly, while post-hoc explanation promises only a plausible story about a complex mechanism that was never actually simplified — and the gap between those two promises is exactly what mechanistic interpretability research is trying to close, one verified circuit at a time, for models whose true internal logic otherwise remains opaque even to their own creators.