Interpretability studies how model inputs, representations, computations, training data, and parameters produce predictions or behavior. It supports debugging, scientific understanding, safety investigation, accountability, and human decision support, but explanations must be evaluated for fidelity rather than persuasive appearance. Interpretability is often distinguished from explainability, though usage varies: mechanistic work reverse-engineers internal algorithms and circuits, while post-hoc methods attribute or summarize behavior without fully describing computation. A production definition states the base model and revision, tokenizer and vocabulary, context and output limits, numerical precision, data provenance, objective, trainable state, inference runtime, tool or retrieval boundary, evaluation population, latency and cost target, failure policy, and reproducibility artifacts. Similar labels can hide materially different implementations, so exact interfaces and assumptions belong in the contract. An interpretation claim states model and checkpoint, target output, method, layer or feature, causal versus correlational status, baseline, data, aggregation, human audience, uncertainty, and known limitations.

Architecture, representation, and operating mechanism. Input attribution includes gradients, integrated gradients, SHAP-like values and occlusion; concept methods test human-defined directions; probing predicts properties from representations; feature visualization seeks preferred inputs; mechanistic methods use activation patching, ablation, path analysis, sparse autoencoders, and circuit hypotheses. A rigorous workflow observes a phenomenon, forms a hypothesis, localizes candidate components, intervenes causally, predicts effects on held-out inputs, measures specificity and completeness, attempts falsification, and documents residual unexplained behavior. Global versus local, intrinsic versus post-hoc, model-specific versus model-agnostic, feature versus example attribution, representation probing, counterfactual explanation, training-data attribution, and mechanistic circuit analysis answer different questions. The complete stack includes input normalization, tokenization, embeddings, Transformer blocks, attention and KV state, output decoding, adapters or post-training weights, retrieval and tools where used, orchestration, policy controls, telemetry, and artifact storage. Data, control, and trust boundaries should remain visible instead of being collapsed into a single model call. Evaluation keeps task quality beside factuality, calibration, robustness, safety, subgroup behavior, context utilization, throughput, time to first token, inter-token latency, tail latency, memory, bandwidth, accelerator utilization, energy, and cost. Controlled comparisons hold prompts, sampling, data, model, hardware, concurrency, and judge protocol fixed and report uncertainty across repeated runs.

Implementation, serving infrastructure, and failure modes. Choose baselines explicitly, control for probe capacity, compare random and label-shuffled controls, avoid cherry-picked examples, run ablations and causal interventions, account for superposition and polysemantic units, version hooks against model code, and preserve exact prompts and activations. Large-model analysis captures or edits enormous activations and gradients, requiring HBM, storage, distributed hooks, compression, selective caching, and reproducible inference. Sparse autoencoder training adds substantial accelerator cost beyond the target model. Attention is presented as explanation, probes extract information the model does not use, saliency is visually plausible but unstable, ablations leave distribution, features are assigned anthropomorphic labels, circuit claims do not generalize, or explanations leak sensitive training data. Implementation starts with a small explicit reference, typed schemas, deterministic fixtures, versioned prompts and templates, and traceable input-output examples. Production adds batching, streaming, mixed precision, compilation, caching, parallelism, retries, fallbacks, rate limits, redaction, isolation, and observability without changing semantics silently. Accelerators execute dense and sparse tensor kernels while HBM stores weights, activations, adapters, and KV state; CPUs tokenize and orchestrate; host memory, storage, PCIe, scale-up fabric, and scale-out networks move artifacts and requests. Batch, sequence length, vocabulary, precision, cache locality, communication, and power determine delivered rather than peak behavior. Typical failures include data leakage, template mismatch, tokenizer drift, train-serving skew, stale caches, unsupported operators, precision loss, memory fragmentation, prompt injection, malformed structured output, tool side effects, runaway loops, evaluation contamination, hidden retries, and average metrics that conceal catastrophic tails. A fluent answer is not evidence of correctness.

Evaluation, security, and lifecycle controls. Use sanity checks, randomization, invariance and stability tests, causal patching/ablation, held-out prompts, adversarial counterexamples, inter-rater studies, predictive circuit tests, completeness estimates, and comparison against simpler behavioral baselines. Fidelity, completeness, sufficiency/necessity, stability, localization, sparsity, human usefulness, calibration, runtime, activation storage, reproducibility, and downstream debugging or safety benefit matter. Interpretations can overstate certainty or expose sensitive examples. Communicate limits, separate evidence from narrative, control activation/data access, audit high-impact explanations, and never substitute explanation for outcome testing and recourse. Verification combines unit and property tests, reference parity, adversarial and edge-case prompts, schema validation, deterministic replay, offline benchmark suites, human review, safety red teaming, privacy and security tests, load and fault injection, long-context checks, shadow traffic, canary rollout, and rollback drills. Every result links to the exact model, data, tokenizer, configuration, code, and runtime. Collection, filtering, training or tuning, evaluation, registration, deployment, monitoring, incident response, refresh, rollback, retention, deletion, and retirement form one lifecycle. Model cards, data and prompt lineage, approvals, exceptions, dependencies, licenses, checkpoints, adapter versions, tool permissions, and evaluation evidence remain auditable. Owners define intended and prohibited use, access and tenant isolation, data minimization, consent or lawful basis, secret handling, human confirmation for consequential actions, rate and spend limits, abuse monitoring, appeal and escalation, retention, and incident responsibility. External model or framework behavior is treated as an untrusted dependency with pinned versions and compensating controls.

<svg viewBox="0 0 760 470" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system,Segoe UI,Roboto,sans-serif">
  <rect x="0" y="0" width="760" height="470" fill="#0d1117"/>
  <text x="380" y="28" fill="#e6edf3" font-size="21" font-weight="700" text-anchor="middle">Interpretability — Understanding Neural Networks</text>
  <text x="380" y="48" fill="#8b98a5" font-size="12" text-anchor="middle">mechanistic interpretability: reverse-engineer what circuits, features, and algorithms live inside trained models</text>

  <!-- === TOP: Methods spectrum === -->
  <rect x="25" y="62" width="710" height="105" rx="6" fill="#080d14" stroke="#233043" stroke-width="1.2"/>
  <text x="380" y="80" fill="#e6edf3" font-size="11" text-anchor="middle" font-weight="600">Interpretability Methods (local → global)</text>

  <rect x="40" y="94" width="125" height="58" rx="4" fill="#0f1a2a" stroke="#60a5fa" stroke-width="0.9"/>
  <text x="102" y="112" fill="#93c5fd" font-size="9" text-anchor="middle" font-weight="600">Input Attribution</text>
  <text x="102" y="126" fill="#8b98a5" font-size="7.5" text-anchor="middle">which input tokens matter?</text>
  <text x="102" y="140" fill="#6b7684" font-size="7" text-anchor="middle">gradients, SHAP, attention</text>
  <text x="102" y="150" fill="#6b7684" font-size="6.5" text-anchor="middle">integrated gradients</text>

  <rect x="180" y="94" width="125" height="58" rx="4" fill="#14261f" stroke="#34d399" stroke-width="0.9"/>
  <text x="242" y="112" fill="#6ee7b7" font-size="9" text-anchor="middle" font-weight="600">Probing</text>
  <text x="242" y="126" fill="#8b98a5" font-size="7.5" text-anchor="middle">what do activations encode?</text>
  <text x="242" y="140" fill="#6b7684" font-size="7" text-anchor="middle">linear probes on hidden states</text>
  <text x="242" y="150" fill="#6b7684" font-size="6.5" text-anchor="middle">concept presence detectors</text>

  <rect x="320" y="94" width="125" height="58" rx="4" fill="#2a1a0a" stroke="#f59e0b" stroke-width="0.9"/>
  <text x="382" y="112" fill="#fbbf24" font-size="9" text-anchor="middle" font-weight="600">Activation Patching</text>
  <text x="382" y="126" fill="#8b98a5" font-size="7.5" text-anchor="middle">which components cause X?</text>
  <text x="382" y="140" fill="#6b7684" font-size="7" text-anchor="middle">swap activations, measure Δ</text>
  <text x="382" y="150" fill="#6b7684" font-size="6.5" text-anchor="middle">causal intervention</text>

  <rect x="460" y="94" width="125" height="58" rx="4" fill="#1a1520" stroke="#a78bfa" stroke-width="0.9"/>
  <text x="522" y="112" fill="#c4b5fd" font-size="9" text-anchor="middle" font-weight="600">Sparse Autoencoders</text>
  <text x="522" y="126" fill="#8b98a5" font-size="7.5" text-anchor="middle">what features are encoded?</text>
  <text x="522" y="140" fill="#6b7684" font-size="7" text-anchor="middle">decompose into mono-semantic</text>
  <text x="522" y="150" fill="#6b7684" font-size="6.5" text-anchor="middle">features (SAE dictionaries)</text>

  <rect x="600" y="94" width="115" height="58" rx="4" fill="#1a0f0f" stroke="#f87171" stroke-width="0.9"/>
  <text x="657" y="112" fill="#f87171" font-size="9" text-anchor="middle" font-weight="600">Circuit Analysis</text>
  <text x="657" y="126" fill="#8b98a5" font-size="7.5" text-anchor="middle">which paths compute X?</text>
  <text x="657" y="140" fill="#6b7684" font-size="7" text-anchor="middle">trace end-to-end circuits</text>
  <text x="657" y="150" fill="#6b7684" font-size="6.5" text-anchor="middle">induction heads, IOI</text>

  <!-- === MIDDLE LEFT: Mechanistic interp === -->
  <rect x="25" y="175" width="350" height="125" rx="6" fill="#0b1220" stroke="#233043" stroke-width="1"/>
  <text x="200" y="193" fill="#e6edf3" font-size="10" text-anchor="middle" font-weight="600">Mechanistic Interpretability (Anthropic/DeepMind)</text>

  <text x="45" y="215" fill="#8b98a5" font-size="8.5">Goal: reverse-engineer the "source code" of the network</text>
  <text x="45" y="235" fill="#34d399" font-size="8.5" font-weight="600">Sparse autoencoders (SAEs)</text>
  <text x="45" y="251" fill="#8b98a5" font-size="8.5">train sparse dictionary on residual stream activations</text>
  <text x="45" y="267" fill="#8b98a5" font-size="8.5">find interpretable features (Golden Gate Bridge, code bugs)</text>
  <text x="45" y="285" fill="#60a5fa" font-size="8.5" font-weight="600">Superposition hypothesis</text>
  <text x="45" y="299" fill="#8b98a5" font-size="8.5">models store more features than dimensions (compressed)</text>

  <!-- === MIDDLE RIGHT: Why it matters === -->
  <rect x="390" y="175" width="345" height="125" rx="6" fill="#0b1220" stroke="#233043" stroke-width="1"/>
  <text x="562" y="193" fill="#e6edf3" font-size="10" text-anchor="middle" font-weight="600">Why Interpretability Matters</text>

  <text x="410" y="215" fill="#f87171" font-size="8.5" font-weight="600">AI safety:</text>
  <text x="470" y="215" fill="#8b98a5" font-size="8.5">detect deceptive alignment, hidden goals</text>
  <text x="410" y="233" fill="#fbbf24" font-size="8.5" font-weight="600">Debugging:</text>
  <text x="480" y="233" fill="#8b98a5" font-size="8.5">find why model hallucinates on topic X</text>
  <text x="410" y="251" fill="#60a5fa" font-size="8.5" font-weight="600">Steering:</text>
  <text x="470" y="251" fill="#8b98a5" font-size="8.5">activation engineering (add/remove features)</text>
  <text x="410" y="269" fill="#34d399" font-size="8.5" font-weight="600">Trust:</text>
  <text x="455" y="269" fill="#8b98a5" font-size="8.5">explain decisions in high-stakes domains</text>
  <text x="410" y="287" fill="#c4b5fd" font-size="8.5" font-weight="600">Science:</text>
  <text x="465" y="287" fill="#8b98a5" font-size="8.5">understand what learning algorithms discover</text>

  <!-- === BOTTOM: Key discoveries === -->
  <rect x="25" y="310" width="710" height="92" rx="5" fill="#0b1220" stroke="#233043" stroke-width="1"/>
  <text x="380" y="328" fill="#e6edf3" font-size="10" text-anchor="middle" font-weight="600">Key Discoveries</text>

  <text x="130" y="352" fill="#c4b5fd" font-size="9" text-anchor="middle" font-weight="600">Induction heads</text>
  <text x="130" y="366" fill="#8b98a5" font-size="8" text-anchor="middle">copy patterns (in-context)</text>
  <text x="130" y="378" fill="#6b7684" font-size="7.5" text-anchor="middle">Olsson et al. 2022</text>

  <text x="300" y="352" fill="#fbbf24" font-size="9" text-anchor="middle" font-weight="600">Polysemantic neurons</text>
  <text x="300" y="366" fill="#8b98a5" font-size="8" text-anchor="middle">one neuron = many concepts</text>
  <text x="300" y="378" fill="#6b7684" font-size="7.5" text-anchor="middle">→ need SAEs to decompose</text>

  <text x="480" y="352" fill="#34d399" font-size="9" text-anchor="middle" font-weight="600">Features as directions</text>
  <text x="480" y="366" fill="#8b98a5" font-size="8" text-anchor="middle">concepts = linear dirs in space</text>
  <text x="480" y="378" fill="#6b7684" font-size="7.5" text-anchor="middle">linear representation hypothesis</text>

  <text x="650" y="352" fill="#60a5fa" font-size="9" text-anchor="middle" font-weight="600">Scaling monosemanticity</text>
  <text x="650" y="366" fill="#8b98a5" font-size="8" text-anchor="middle">Anthropic: SAEs on Claude 3</text>
  <text x="650" y="378" fill="#6b7684" font-size="7.5" text-anchor="middle">millions of features found</text>

  <text x="380" y="396" fill="#6b7684" font-size="8" text-anchor="middle">Tools: TransformerLens, SAE-Vis, Neuronpedia, CircuitsVis, Baukit</text>

  <!-- Key insight -->
  <rect x="25" y="410" width="710" height="22" rx="3" fill="#0b1220" stroke="#233043" stroke-width="0.8"/>
  <text x="380" y="425" fill="#fbbf24" font-size="9" text-anchor="middle">We're at "early microscopy" stage: SAEs are the first tool that lets us see individual features inside frontier LLMs.</text>

  <text x="380" y="460" fill="#6b7684" font-size="11" text-anchor="middle">Interpretability is the path to AI safety: you can't align what you don't understand.</text>
</svg>

Selection and practical application. Use SHAP-like methods for tabular feature attribution with assumptions stated, saliency/Grad-CAM for vision debugging, probes for encoded information with controls, causal methods for mechanisms, and counterfactuals for actionable local questions. Model debugging, bias and safety analysis, scientific discovery, failure triage, feature auditing, compliance support, circuit research, data attribution, and human decision support use interpretability. Interpretability spans data, model internals, inference hooks, causal experiments, visualization, evaluators, domain experts, governance, and deployment monitoring. The useful optimization boundary is the end-to-end application: user interface, model, tokenizer, context builder, cache, adapter, retriever, tools, runtime, accelerator, scheduler, network, policy, monitoring, and human workflow. Improving one component can move the bottleneck or weaken correctness, safety, isolation, and recoverability elsewhere. A production definition states the base model and revision, tokenizer and vocabulary, context and output limits, numerical precision, data provenance, objective, trainable state, inference runtime, tool or retrieval boundary, evaluation population, latency and cost target, failure policy, and reproducibility artifacts. Similar labels can hide materially different implementations, so exact interfaces and assumptions belong in the contract. Evaluation keeps task quality beside factuality, calibration, robustness, safety, subgroup behavior, context utilization, throughput, time to first token, inter-token latency, tail latency, memory, bandwidth, accelerator utilization, energy, and cost. Controlled comparisons hold prompts, sampling, data, model, hardware, concurrency, and judge protocol fixed and report uncertainty across repeated runs. CFS connects this topic to semiconductor architecture, implementation, verification, manufacturing, packaging, test, and deployed AI-system tradeoffs across the platform.