<svg viewBox="0 0 760 470" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system,Segoe UI,Roboto,sans-serif">
  <rect x="0" y="0" width="760" height="470" fill="#0d1117"/>
  <text x="380" y="28" fill="#e6edf3" font-size="21" font-weight="700" text-anchor="middle">Transformer Architecture — The Foundation of Modern AI</text>
  <text x="380" y="48" fill="#8b98a5" font-size="12" text-anchor="middle">attention is all you need: self-attention + feed-forward + residuals + layer norm, stacked N times</text>

  <!-- === TOP: One transformer block === -->
  <rect x="25" y="62" width="420" height="195" rx="6" fill="#080d14" stroke="#233043" stroke-width="1.2"/>
  <text x="235" y="82" fill="#e6edf3" font-size="11" text-anchor="middle" font-weight="600">One Transformer Block (decoder-only, GPT-style)</text>

  <!-- Input -->
  <rect x="195" y="92" width="80" height="22" rx="3" fill="#14261f" stroke="#34d399" stroke-width="0.8"/>
  <text x="235" y="106" fill="#6ee7b7" font-size="8.5" text-anchor="middle">Input (x)</text>

  <!-- Residual arrow (left side) -->
  <path d="M195,103 L50,103 L50,245 L195,245" fill="none" stroke="#34d399" stroke-width="0.7" stroke-dasharray="3,2"/>
  <text x="55" y="170" fill="#34d399" font-size="7" transform="rotate(-90,55,170)">residual</text>

  <!-- Arrow down -->
  <path d="M235,116 L235,128" fill="none" stroke="#8b98a5" stroke-width="0.7"/>

  <!-- Layer Norm 1 -->
  <rect x="195" y="130" width="80" height="20" rx="2" fill="#0b1220" stroke="#233043" stroke-width="0.7"/>
  <text x="235" y="143" fill="#8b98a5" font-size="8" text-anchor="middle">RMSNorm</text>

  <path d="M235,152 L235,162" fill="none" stroke="#8b98a5" stroke-width="0.7"/>

  <!-- Multi-head attention -->
  <rect x="170" y="164" width="130" height="32" rx="4" fill="#1a1520" stroke="#a78bfa" stroke-width="1"/>
  <text x="235" y="180" fill="#c4b5fd" font-size="9" text-anchor="middle" font-weight="600">Multi-Head Attention</text>
  <text x="235" y="192" fill="#6b7684" font-size="7" text-anchor="middle">Q·K^T/√d → softmax → ×V</text>

  <!-- + residual -->
  <path d="M235,198 L235,206" fill="none" stroke="#8b98a5" stroke-width="0.7"/>
  <circle cx="235" cy="210" r="6" fill="none" stroke="#34d399" stroke-width="0.8"/>
  <text x="235" y="213" fill="#34d399" font-size="8" text-anchor="middle">+</text>

  <path d="M235,218 L235,226" fill="none" stroke="#8b98a5" stroke-width="0.7"/>

  <!-- Layer Norm 2 -->
  <rect x="195" y="228" width="80" height="20" rx="2" fill="#0b1220" stroke="#233043" stroke-width="0.7"/>
  <text x="235" y="241" fill="#8b98a5" font-size="8" text-anchor="middle">RMSNorm</text>

  <!-- Second residual arrow -->
  <path d="M195,238 L145,238 L145,286 L195,286" fill="none" stroke="#34d399" stroke-width="0.7" stroke-dasharray="3,2"/>

  <!-- FFN / MLP -->
  <rect x="320" y="228" width="110" height="32" rx="4" fill="#2a1a0a" stroke="#f59e0b" stroke-width="1"/>
  <text x="375" y="244" fill="#fbbf24" font-size="9" text-anchor="middle" font-weight="600">FFN (SwiGLU)</text>
  <text x="375" y="256" fill="#6b7684" font-size="7" text-anchor="middle">up → gate × act → down</text>
  <path d="M277,238 L317,244" fill="none" stroke="#8b98a5" stroke-width="0.7"/>

  <!-- Output -->
  <rect x="195" y="275" width="80" height="22" rx="3" fill="#14261f" stroke="#34d399" stroke-width="0.8"/>
  <text x="235" y="289" fill="#6ee7b7" font-size="8.5" text-anchor="middle">Output (x')</text>

  <!-- Repeat annotation -->
  <text x="235" y="310" fill="#6b7684" font-size="8" text-anchor="middle">× N layers (N = 32 for 7B, 80 for 70B, 126 for 405B)</text>

  <!-- === TOP RIGHT: Key components === -->
  <rect x="460" y="62" width="275" height="195" rx="6" fill="#0b1220" stroke="#233043" stroke-width="1"/>
  <text x="597" y="82" fill="#e6edf3" font-size="10" text-anchor="middle" font-weight="600">Key Components</text>

  <text x="480" y="102" fill="#c4b5fd" font-size="8.5" font-weight="600">Attention (QKV)</text>
  <text x="480" y="116" fill="#8b98a5" font-size="8">global context, O(n²) in seq length</text>

  <text x="480" y="136" fill="#fbbf24" font-size="8.5" font-weight="600">FFN / MLP (2/3 of params)</text>
  <text x="480" y="150" fill="#8b98a5" font-size="8">knowledge storage, per-token transform</text>

  <text x="480" y="170" fill="#34d399" font-size="8.5" font-weight="600">Residual connections</text>
  <text x="480" y="184" fill="#8b98a5" font-size="8">gradient highway, enables depth</text>

  <text x="480" y="204" fill="#60a5fa" font-size="8.5" font-weight="600">Positional encoding</text>
  <text x="480" y="218" fill="#8b98a5" font-size="8">RoPE (relative, extrapolatable)</text>

  <text x="480" y="238" fill="#f87171" font-size="8.5" font-weight="600">Causal mask</text>
  <text x="480" y="252" fill="#8b98a5" font-size="8">autoregressive: can't see future tokens</text>

  <!-- === BOTTOM LEFT: Model families === -->
  <rect x="25" y="325" width="350" height="80" rx="5" fill="#0b1220" stroke="#233043" stroke-width="1"/>
  <text x="200" y="343" fill="#e6edf3" font-size="10" text-anchor="middle" font-weight="600">Transformer Variants</text>

  <text x="45" y="363" fill="#60a5fa" font-size="8.5" font-weight="600">Encoder-only:</text>
  <text x="135" y="363" fill="#8b98a5" font-size="8.5">BERT (bidirectional, classification)</text>
  <text x="45" y="379" fill="#34d399" font-size="8.5" font-weight="600">Decoder-only:</text>
  <text x="140" y="379" fill="#8b98a5" font-size="8.5">GPT, Llama, Claude (generation)</text>
  <text x="45" y="395" fill="#fbbf24" font-size="8.5" font-weight="600">Encoder-decoder:</text>
  <text x="155" y="395" fill="#8b98a5" font-size="8.5">T5, BART (seq2seq, translation)</text>

  <!-- === BOTTOM RIGHT: Scale === -->
  <rect x="390" y="325" width="345" height="80" rx="5" fill="#0b1220" stroke="#233043" stroke-width="1"/>
  <text x="562" y="343" fill="#e6edf3" font-size="10" text-anchor="middle" font-weight="600">Scale Progression</text>

  <text x="410" y="363" fill="#8b98a5" font-size="8.5">GPT-1 (2018): 117M params, 12 layers</text>
  <text x="410" y="379" fill="#8b98a5" font-size="8.5">GPT-3 (2020): 175B params, 96 layers</text>
  <text x="410" y="395" fill="#fbbf24" font-size="8.5">Llama-3 (2024): 405B params, 126 layers</text>

  <!-- Key insight -->
  <rect x="25" y="413" width="710" height="22" rx="3" fill="#0b1220" stroke="#233043" stroke-width="0.8"/>
  <text x="380" y="428" fill="#fbbf24" font-size="9" text-anchor="middle">The transformer is just attention + MLP + skip connections — its power comes from scale, data, and the training objective.</text>

  <text x="380" y="460" fill="#6b7684" font-size="11" text-anchor="middle">Every frontier model — GPT-4, Claude, Gemini, Llama — is a transformer. The architecture won.</text>
</svg>
```**Transformer** is the neural-network architecture introduced in the 2017 paper *Attention Is All You Need*, and it is the foundation of virtually every modern large language model, image generator, and speech system. Its breakthrough was replacing the sequential, step-by-step processing of earlier recurrent networks with a mechanism — self-attention — that looks at an entire sequence at once and lets every element directly consult every other. That single change made it possible to train on far more data, in parallel, than anything before it. The diagram shows the repeating block that gets stacked to build the whole model.\n\n```svg\n<svg viewBox="0 0 760 500" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system,Segoe UI,Roboto,sans-serif">\n  <rect x="0" y="0" width="760" height="500" fill="#0d1117"/>\n  <text x="380" y="32" fill="#e6edf3" font-size="21" font-weight="700" text-anchor="middle">The Transformer Block</text>\n  <text x="380" y="54" fill="#8b98a5" font-size="12.5" text-anchor="middle">attention mixes tokens, the feed-forward network thinks per token — stacked N times</text>\n  <rect x="125.0" y="430" width="250" height="40" rx="5" fill="#134e3a" stroke="#34d399" stroke-width="1.3"/><text x="250" y="454" fill="#6ee7b7" font-size="12.5" font-weight="700" text-anchor="middle">Input embeddings + positional encoding</text>\n  <path d="M250 430 L250 412 M245 420 L250 412 L255 420" stroke="#5b6b7f" stroke-width="1.3" fill="none"/>\n  <rect x="125.0" y="372" width="250" height="44" rx="5" fill="#3b2f0a" stroke="#fbbf24" stroke-width="1.3"/><text x="250" y="392" fill="#fcd34d" font-size="12.5" font-weight="700" text-anchor="middle">Multi-head self-attention</text><text x="250" y="407" fill="#9aa4b2" font-size="12" text-anchor="middle">every token attends to every other</text>\n  <path d="M111.0 424 L111.0 358 L121.0 358" fill="none" stroke="#64748b" stroke-width="1.1" stroke-dasharray="4 3"/>\n  <path d="M250 372 L250 358 M245 366 L250 358 L255 366" stroke="#5b6b7f" stroke-width="1.3" fill="none"/>\n  <rect x="125.0" y="330" width="250" height="26" rx="5" fill="#101a28" stroke="#60a5fa" stroke-width="1.3"/><text x="250" y="347" fill="#93c5fd" font-size="13" font-weight="700" text-anchor="middle">Add &amp; Norm</text>\n  <path d="M250 330 L250 312 M245 320 L250 312 L255 320" stroke="#5b6b7f" stroke-width="1.3" fill="none"/>\n  <rect x="125.0" y="272" width="250" height="44" rx="5" fill="#1c1633" stroke="#a78bfa" stroke-width="1.3"/><text x="250" y="292" fill="#c4b5fd" font-size="12.5" font-weight="700" text-anchor="middle">Feed-forward network</text><text x="250" y="307" fill="#9aa4b2" font-size="12" text-anchor="middle">expand → nonlinearity → project</text>\n  <path d="M111.0 324 L111.0 258 L121.0 258" fill="none" stroke="#64748b" stroke-width="1.1" stroke-dasharray="4 3"/>\n  <path d="M250 272 L250 258 M245 266 L250 258 L255 266" stroke="#5b6b7f" stroke-width="1.3" fill="none"/>\n  <rect x="125.0" y="230" width="250" height="26" rx="5" fill="#101a28" stroke="#60a5fa" stroke-width="1.3"/><text x="250" y="247" fill="#93c5fd" font-size="13" font-weight="700" text-anchor="middle">Add &amp; Norm</text>\n  <path d="M250 230 L250 212 M245 220 L250 212 L255 220" stroke="#5b6b7f" stroke-width="1.3" fill="none"/>\n  <rect x="125.0" y="190" width="250" height="30" rx="5" fill="#1e2530" stroke="#cbd5e1" stroke-width="1.3"/><text x="250" y="209" fill="#e6edf3" font-size="13" font-weight="700" text-anchor="middle">to next block / output</text>\n  <rect x="101.0" y="182" width="290" height="288" rx="9" fill="none" stroke="#3f5169" stroke-width="1.2" stroke-dasharray="5 4"/>\n  <text x="405.0" y="330" fill="#8b98a5" font-size="13" transform="rotate(90 409.0 330)">× N identical layers</text>\n  <text x="95.0" y="350" fill="#64748b" font-size="12" transform="rotate(-90 95.0 350)">residual (skip) connections</text>\n  <line x1="500" y1="90" x2="500" y2="400" stroke="#233043" stroke-width="1"/>\n  <text x="530" y="120" fill="#e6edf3" font-size="13" font-weight="700">Inside one attention head</text>\n  <text x="530" y="146" fill="#fcd34d" font-size="13" font-weight="700">Query · Key · Value</text>\n  <text x="530" y="161" fill="#8b98a5" font-size="12">each token makes 3 vectors</text>\n  <text x="530" y="188" fill="#6ee7b7" font-size="13" font-weight="700">scores = Q·Kᵀ / √d</text>\n  <text x="530" y="203" fill="#8b98a5" font-size="12">how much to attend</text>\n  <text x="530" y="230" fill="#93c5fd" font-size="13" font-weight="700">softmax → weights</text>\n  <text x="530" y="245" fill="#8b98a5" font-size="12">sum to 1 across tokens</text>\n  <text x="530" y="272" fill="#c4b5fd" font-size="13" font-weight="700">output = Σ weight · V</text>\n  <text x="530" y="287" fill="#8b98a5" font-size="12">context-mixed vector</text>\n  <text x="530" y="326" fill="#f87171" font-size="13" font-weight="700">No recurrence</text>\n  <text x="530" y="341" fill="#8b98a5" font-size="12">all tokens processed</text>\n  <text x="530" y="355" fill="#8b98a5" font-size="12">in parallel — the reason</text>\n  <text x="530" y="369" fill="#8b98a5" font-size="12">transformers scale</text>\n  <text x="380" y="486" fill="#6b7684" font-size="13" text-anchor="middle">Attention answers &#8220;which other tokens matter here?&#8221;; the feed-forward layer transforms each token given that context.</text>\n</svg>\n```\n\n**Self-attention is the core idea.** For every token, the model produces three vectors — a query, a key, and a value. It compares each token's query against all the keys to decide how much attention to pay to every other token, normalizes those scores with a softmax, and returns a weighted blend of the values. The result is a new representation of each token that has absorbed exactly the context it needs, whether the relevant word is one position away or a thousand.\n\n**Multi-head attention looks in several ways at once.** Rather than a single attention computation, the block runs several in parallel — different "heads" that can specialize, one tracking syntax, another coreference, another local phrasing. Their outputs are concatenated and projected back together, giving the model multiple relationship types per layer.\n\n**The feed-forward network processes each token alone.** After attention has mixed information across positions, a small two-layer network is applied independently to every token: expand to a wider dimension, apply a nonlinearity, project back. This is where much of the model's raw capacity and stored knowledge lives. Attention decides *what to combine*; the feed-forward layer decides *what to do with it*.\n\n**Residual connections and normalization make depth trainable.** Each sub-layer's output is added back to its input (a residual, or skip, connection) and normalized. This keeps gradients flowing cleanly through dozens or hundreds of stacked layers, which is what lets Transformers go deep without the signal degrading.\n\n**Parallelism is the reason it won.** Because there is no recurrence, all positions in a sequence are processed simultaneously during training — a perfect match for the wide, parallel arithmetic of GPUs and TPUs. Recurrent networks had to march through a sequence one step at a time; the Transformer turned language modeling into big matrix multiplications, and that is exactly what modern accelerators do fastest.\n\n| Piece | What it does | Question it answers |\n|---|---|---|\n| Query / Key / Value | per-token vectors for attention | what am I looking for, offering, carrying |\n| Attention scores | Q·Kᵀ scaled, then softmax | which tokens matter to me |\n| Multi-head | parallel attention subspaces | what relationships exist at once |\n| Feed-forward | per-token transformation | what to make of the mixed context |\n| Residual + norm | add input back, normalize | how to stay trainable when deep |\n\nRead a Transformer through an *all-at-once attention* lens rather than a *sequence-processing* lens: earlier models understood a sentence by walking through it word by word, carrying a running memory, while the Transformer lays the whole sequence out and lets every token pull directly from every other in a single parallel step. That shift is why it trains efficiently at massive scale, why context length is such a central design axis, and why "attention" — not recurrence or convolution — became the organizing principle of modern AI.\n