Model distillation trains a smaller student to reproduce useful behavior or internal representations of a larger teacher. It converts expensive model capability into a lower-latency, lower-memory, lower-energy artifact for deployment, while exposing a controlled trade between capacity and fidelity. Soft teacher probabilities contain relative class or token preferences sometimes called dark knowledge; temperature can reveal this structure more clearly than one-hot labels. A production definition states the base model and revision, tokenizer and vocabulary, context and output limits, numerical precision, data provenance, objective, trainable state, inference runtime, tool or retrieval boundary, evaluation population, latency and cost target, failure policy, and reproducibility artifacts. Similar labels can hide materially different implementations, so exact interfaces and assumptions belong in the contract. A distillation specification names teacher and student, data source, prompt distribution, teacher sampling, target layers, temperature, loss weights, tokenizer compatibility, generated-data filtering, safety constraints, and deployment objective.

Architecture, representation, and operating mechanism. Logit distillation minimizes divergence between teacher and student output distributions, feature distillation aligns hidden states through projections, attention distillation matches maps or relations, and sequence-level distillation trains on teacher-generated outputs. Progressive methods use intermediate teachers or stages. The teacher produces logits, features, rationales, or synthetic responses; targets are cached or generated online; the student optimizes a mixture of teacher imitation and ground-truth or preference loss; evaluation compares capability, calibration, safety, and efficiency. Response-based, feature-based, relation/attention-based, self-distillation, multi-teacher, task-specific, data-free, online, offline, and progressive distillation differ in required access and compute. Distillation can accompany quantization and pruning. The complete stack includes input normalization, tokenization, embeddings, Transformer blocks, attention and KV state, output decoding, adapters or post-training weights, retrieval and tools where used, orchestration, policy controls, telemetry, and artifact storage. Data, control, and trust boundaries should remain visible instead of being collapsed into a single model call. Evaluation keeps task quality beside factuality, calibration, robustness, safety, subgroup behavior, context utilization, throughput, time to first token, inter-token latency, tail latency, memory, bandwidth, accelerator utilization, energy, and cost. Controlled comparisons hold prompts, sampling, data, model, hardware, concurrency, and judge protocol fixed and report uncertainty across repeated runs.

Implementation, serving infrastructure, and failure modes. Use stable KL/log-softmax computations, align vocabularies or map distributions, mask padding, manage teacher temperature, prevent answer leakage into evaluation, filter low-quality synthetic data, balance hard and soft losses, and preserve exact teacher/checkpoint provenance. Offline teacher inference may dominate cost and storage; cached logits for large vocabularies are enormous, so top-k targets, lower precision, on-the-fly generation, and feature selection trade bandwidth for recomputation. Student training uses conventional accelerators. The student inherits teacher errors and bias, overfits narrow synthetic prompts, loses tail knowledge or calibration, tokenizer mismatch corrupts targets, exposed rationales are treated as faithful reasoning, or efficiency gains vanish in unsupported kernels. Implementation starts with a small explicit reference, typed schemas, deterministic fixtures, versioned prompts and templates, and traceable input-output examples. Production adds batching, streaming, mixed precision, compilation, caching, parallelism, retries, fallbacks, rate limits, redaction, isolation, and observability without changing semantics silently. Accelerators execute dense and sparse tensor kernels while HBM stores weights, activations, adapters, and KV state; CPUs tokenize and orchestrate; host memory, storage, PCIe, scale-up fabric, and scale-out networks move artifacts and requests. Batch, sequence length, vocabulary, precision, cache locality, communication, and power determine delivered rather than peak behavior. Typical failures include data leakage, template mismatch, tokenizer drift, train-serving skew, stale caches, unsupported operators, precision loss, memory fragmentation, prompt injection, malformed structured output, tool side effects, runaway loops, evaluation contamination, hidden retries, and average metrics that conceal catastrophic tails. A fluent answer is not evidence of correctness.

Evaluation, security, and lifecycle controls. Measure held-out task quality, teacher-student divergence, calibration, robustness, safety, subgroup behavior, memorization, adversarial transfer, latency, throughput, memory, energy, and performance under distribution shift. Parameter and byte reduction, compression ratio, teacher query cost, quality retained, KL divergence, calibration, tokens/s, latency, memory, energy, and cost to target matter. Teacher outputs are derived data with license, privacy, provenance, retention, and safety implications. Document what knowledge was transferred, filtering decisions, inherited limitations, and evaluation boundaries. Verification combines unit and property tests, reference parity, adversarial and edge-case prompts, schema validation, deterministic replay, offline benchmark suites, human review, safety red teaming, privacy and security tests, load and fault injection, long-context checks, shadow traffic, canary rollout, and rollback drills. Every result links to the exact model, data, tokenizer, configuration, code, and runtime. Collection, filtering, training or tuning, evaluation, registration, deployment, monitoring, incident response, refresh, rollback, retention, deletion, and retirement form one lifecycle. Model cards, data and prompt lineage, approvals, exceptions, dependencies, licenses, checkpoints, adapter versions, tool permissions, and evaluation evidence remain auditable. Owners define intended and prohibited use, access and tenant isolation, data minimization, consent or lawful basis, secret handling, human confirmation for consequential actions, rate and spend limits, abuse monitoring, appeal and escalation, retention, and incident responsibility. External model or framework behavior is treated as an untrusted dependency with pinned versions and compensating controls.

<svg viewBox="0 0 760 470" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system,Segoe UI,Roboto,sans-serif">
  <rect x="0" y="0" width="760" height="470" fill="#0d1117"/>
  <text x="380" y="28" fill="#e6edf3" font-size="21" font-weight="700" text-anchor="middle">Knowledge Distillation — Teacher to Student</text>
  <text x="380" y="48" fill="#8b98a5" font-size="12" text-anchor="middle">train a small student to mimic the soft predictions of a large teacher — compress without retraining from scratch</text>

  <!-- === TOP: Core distillation flow === -->
  <rect x="25" y="62" width="710" height="145" rx="6" fill="#080d14" stroke="#233043" stroke-width="1.2"/>
  <text x="380" y="80" fill="#e6edf3" font-size="11" text-anchor="middle" font-weight="600">Distillation Training Loop</text>

  <!-- Input -->
  <rect x="40" y="100" width="55" height="30" rx="4" fill="#14261f" stroke="#34d399" stroke-width="0.9"/>
  <text x="67" y="118" fill="#6ee7b7" font-size="8.5" text-anchor="middle">Input x</text>

  <!-- Arrow to both models -->
  <path d="M98,110 L140,95" fill="none" stroke="#8b98a5" stroke-width="0.8"/>
  <path d="M98,120 L140,145" fill="none" stroke="#8b98a5" stroke-width="0.8"/>

  <!-- Teacher model (large) -->
  <rect x="145" y="80" width="110" height="38" rx="5" fill="#1a1520" stroke="#a78bfa" stroke-width="1.2"/>
  <text x="200" y="98" fill="#c4b5fd" font-size="9.5" text-anchor="middle" font-weight="600">Teacher (70B)</text>
  <text x="200" y="112" fill="#6b7684" font-size="7.5" text-anchor="middle">frozen weights</text>

  <!-- Student model (small) -->
  <rect x="145" y="132" width="110" height="38" rx="5" fill="#0f1a2a" stroke="#60a5fa" stroke-width="1.2"/>
  <text x="200" y="150" fill="#93c5fd" font-size="9.5" text-anchor="middle" font-weight="600">Student (7B)</text>
  <text x="200" y="164" fill="#6b7684" font-size="7.5" text-anchor="middle">trainable</text>

  <!-- Teacher output: soft logits -->
  <path d="M258,99 L295,99" fill="none" stroke="#a78bfa" stroke-width="0.8"/>
  <rect x="298" y="86" width="85" height="28" rx="3" fill="#1a1520" stroke="#a78bfa" stroke-width="0.8"/>
  <text x="340" y="100" fill="#c4b5fd" font-size="8" text-anchor="middle">soft logits (T=τ)</text>
  <text x="340" y="110" fill="#6b7684" font-size="7" text-anchor="middle">dark knowledge</text>

  <!-- Student output: soft logits -->
  <path d="M258,151 L295,151" fill="none" stroke="#60a5fa" stroke-width="0.8"/>
  <rect x="298" y="138" width="85" height="28" rx="3" fill="#0f1a2a" stroke="#60a5fa" stroke-width="0.8"/>
  <text x="340" y="153" fill="#93c5fd" font-size="8" text-anchor="middle">student logits (T=τ)</text>

  <!-- KL divergence loss -->
  <path d="M385,100 L420,120" fill="none" stroke="#a78bfa" stroke-width="0.7"/>
  <path d="M385,151 L420,130" fill="none" stroke="#60a5fa" stroke-width="0.7"/>
  <rect x="423" y="110" width="75" height="28" rx="4" fill="#2a1a0a" stroke="#f59e0b" stroke-width="1"/>
  <text x="460" y="124" fill="#fbbf24" font-size="8.5" text-anchor="middle">KL Div Loss</text>
  <text x="460" y="135" fill="#a1701a" font-size="7" text-anchor="middle">soft targets</text>

  <!-- Hard label loss -->
  <rect x="423" y="148" width="75" height="24" rx="3" fill="#14261f" stroke="#34d399" stroke-width="0.8"/>
  <text x="460" y="164" fill="#6ee7b7" font-size="8" text-anchor="middle">CE (hard labels)</text>

  <!-- Combined loss -->
  <path d="M500,124 L530,130" fill="none" stroke="#f59e0b" stroke-width="0.7"/>
  <path d="M500,160 L530,140" fill="none" stroke="#34d399" stroke-width="0.7"/>
  <rect x="535" y="122" width="80" height="30" rx="4" fill="#0b1220" stroke="#f87171" stroke-width="0.9"/>
  <text x="575" y="136" fill="#f87171" font-size="8" text-anchor="middle">L = α·KL + (1-α)·CE</text>
  <text x="575" y="148" fill="#6b7684" font-size="7" text-anchor="middle">α ≈ 0.5-0.9</text>

  <!-- Gradient to student -->
  <path d="M575,155 C575,180 200,185 200,172" fill="none" stroke="#f87171" stroke-width="0.8" stroke-dasharray="3,2"/>
  <text x="400" y="192" fill="#f87171" font-size="8" text-anchor="middle">∇ updates student only</text>

  <!-- === MIDDLE LEFT: Temperature scaling === -->
  <rect x="25" y="215" width="230" height="120" rx="6" fill="#0b1220" stroke="#233043" stroke-width="1"/>
  <text x="140" y="233" fill="#e6edf3" font-size="10" text-anchor="middle" font-weight="600">Temperature τ (Softmax Smoothing)</text>

  <text x="45" y="254" fill="#fbbf24" font-size="9">τ = 1 (sharp): [0.95, 0.03, 0.02]</text>
  <text x="45" y="272" fill="#c4b5fd" font-size="9">τ = 4 (soft): [0.45, 0.30, 0.25]</text>
  <text x="45" y="292" fill="#8b98a5" font-size="8.5">Higher τ → smoother distribution</text>
  <text x="45" y="306" fill="#8b98a5" font-size="8.5">→ reveals inter-class relationships</text>
  <text x="45" y="320" fill="#6b7684" font-size="8.5">the "dark knowledge" Hinton et al. 2015</text>

  <!-- === MIDDLE RIGHT: Distillation types === -->
  <rect x="270" y="215" width="465" height="120" rx="6" fill="#0b1220" stroke="#233043" stroke-width="1"/>
  <text x="502" y="233" fill="#e6edf3" font-size="10" text-anchor="middle" font-weight="600">Distillation Methods</text>

  <text x="290" y="254" fill="#60a5fa" font-size="9" font-weight="600">Response-based</text>
  <text x="290" y="268" fill="#8b98a5" font-size="8.5">match final logits (classic KD)</text>

  <text x="290" y="288" fill="#34d399" font-size="9" font-weight="600">Feature-based</text>
  <text x="290" y="302" fill="#8b98a5" font-size="8.5">match intermediate representations</text>

  <text x="290" y="322" fill="#c4b5fd" font-size="9" font-weight="600">Relation-based</text>
  <text x="290" y="332" fill="#8b98a5" font-size="8.5">match pairwise / structural relations</text>

  <text x="520" y="254" fill="#fbbf24" font-size="9" font-weight="600">Online distillation</text>
  <text x="520" y="268" fill="#8b98a5" font-size="8.5">teacher and student train together</text>

  <text x="520" y="288" fill="#f87171" font-size="9" font-weight="600">Self-distillation</text>
  <text x="520" y="302" fill="#8b98a5" font-size="8.5">same architecture, deeper → shallower</text>

  <text x="520" y="322" fill="#8b98a5" font-size="9" font-weight="600">Multi-teacher</text>
  <text x="520" y="332" fill="#8b98a5" font-size="8.5">ensemble of teachers → one student</text>

  <!-- === BOTTOM: Real examples === -->
  <rect x="25" y="345" width="710" height="70" rx="5" fill="#0b1220" stroke="#233043" stroke-width="1"/>
  <text x="380" y="363" fill="#e6edf3" font-size="10" text-anchor="middle" font-weight="600">Production Distillation Examples</text>

  <text x="105" y="385" fill="#c4b5fd" font-size="9.5" text-anchor="middle" font-weight="600">DistilBERT</text>
  <text x="105" y="399" fill="#8b98a5" font-size="8" text-anchor="middle">BERT-base → 60% size</text>
  <text x="105" y="411" fill="#6b7684" font-size="7.5" text-anchor="middle">97% perf, 2× faster</text>

  <text x="255" y="385" fill="#fbbf24" font-size="9.5" text-anchor="middle" font-weight="600">Gemma/Phi</text>
  <text x="255" y="399" fill="#8b98a5" font-size="8" text-anchor="middle">large → 2B/7B models</text>
  <text x="255" y="411" fill="#6b7684" font-size="7.5" text-anchor="middle">synthetic data distill</text>

  <text x="405" y="385" fill="#60a5fa" font-size="9.5" text-anchor="middle" font-weight="600">TinyLlama</text>
  <text x="405" y="399" fill="#8b98a5" font-size="8" text-anchor="middle">Llama arch, 1.1B</text>
  <text x="405" y="411" fill="#6b7684" font-size="7.5" text-anchor="middle">3T tokens + KD</text>

  <text x="555" y="385" fill="#34d399" font-size="9.5" text-anchor="middle" font-weight="600">Whisper</text>
  <text x="555" y="399" fill="#8b98a5" font-size="8" text-anchor="middle">large → small/tiny</text>
  <text x="555" y="411" fill="#6b7684" font-size="7.5" text-anchor="middle">edge-deployable ASR</text>

  <text x="680" y="385" fill="#f87171" font-size="9.5" text-anchor="middle" font-weight="600">DINO</text>
  <text x="680" y="399" fill="#8b98a5" font-size="8" text-anchor="middle">self-distill ViT</text>
  <text x="680" y="411" fill="#6b7684" font-size="7.5" text-anchor="middle">no labels needed</text>

  <!-- Key insight -->
  <rect x="25" y="423" width="710" height="18" rx="3" fill="#0b1220" stroke="#233043" stroke-width="0.8"/>
  <text x="380" y="436" fill="#fbbf24" font-size="9" text-anchor="middle">Soft targets carry more gradient information per sample than one-hot labels — the student learns faster with fewer examples.</text>

  <text x="380" y="462" fill="#6b7684" font-size="11" text-anchor="middle">Distillation trades teacher compute at training time for permanent inference speedup — the dominant LLM compression path.</text>
</svg>

Selection and practical application. Use logit or sequence distillation when only outputs are available, feature/attention methods with white-box access, progressive teachers for a large capacity gap, and task-specific distillation for narrow edge deployment. Compact language models, mobile vision and speech, recommender ranking, edge assistants, search, safety classifiers, and specialized enterprise models use distillation. Distillation interacts with data generation, tokenizer, alignment, quantization, pruning, compiler kernels, deployment hardware, monitoring, and teacher update cadence. The useful optimization boundary is the end-to-end application: user interface, model, tokenizer, context builder, cache, adapter, retriever, tools, runtime, accelerator, scheduler, network, policy, monitoring, and human workflow. Improving one component can move the bottleneck or weaken correctness, safety, isolation, and recoverability elsewhere. A production definition states the base model and revision, tokenizer and vocabulary, context and output limits, numerical precision, data provenance, objective, trainable state, inference runtime, tool or retrieval boundary, evaluation population, latency and cost target, failure policy, and reproducibility artifacts. Similar labels can hide materially different implementations, so exact interfaces and assumptions belong in the contract. Evaluation keeps task quality beside factuality, calibration, robustness, safety, subgroup behavior, context utilization, throughput, time to first token, inter-token latency, tail latency, memory, bandwidth, accelerator utilization, energy, and cost. Controlled comparisons hold prompts, sampling, data, model, hardware, concurrency, and judge protocol fixed and report uncertainty across repeated runs. CFS connects this topic to semiconductor architecture, implementation, verification, manufacturing, packaging, test, and deployed AI-system tradeoffs across the platform.