<svg viewBox="0 0 760 470" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system,Segoe UI,Roboto,sans-serif">
  <rect x="0" y="0" width="760" height="470" fill="#0d1117"/>
  <text x="380" y="28" fill="#e6edf3" font-size="21" font-weight="700" text-anchor="middle">Backpropagation — Gradients Flow Backward</text>
  <text x="380" y="48" fill="#8b98a5" font-size="12" text-anchor="middle">the chain rule applied to a computational graph: compute dL/dW for every weight, layer by layer</text>

  <!-- === COMPUTATIONAL GRAPH (forward + backward) === -->
  <text x="380" y="74" fill="#e6edf3" font-size="12" font-weight="600" text-anchor="middle">Forward Pass (left→right) then Backward Pass (right→left)</text>

  <!-- Input x -->
  <circle cx="60" cy="155" r="18" fill="#0f1a2a" stroke="#60a5fa" stroke-width="1.3"/>
  <text x="60" y="159" fill="#93c5fd" font-size="11" text-anchor="middle">x</text>
  <text x="60" y="185" fill="#6b7684" font-size="8" text-anchor="middle">input</text>

  <!-- W1 multiply -->
  <path d="M78,155 L118,155" fill="none" stroke="#34d399" stroke-width="1.5"/>
  <polygon points="116,152 122,155 116,158" fill="#34d399"/>
  <text x="100" y="148" fill="#34d399" font-size="8" text-anchor="middle">W₁</text>

  <!-- Node: z1 = W1*x -->
  <circle cx="145" cy="155" r="18" fill="#14261f" stroke="#34d399" stroke-width="1.3"/>
  <text x="145" y="152" fill="#6ee7b7" font-size="9" text-anchor="middle">×</text>
  <text x="145" y="163" fill="#8b98a5" font-size="7" text-anchor="middle">z₁</text>

  <!-- Activation -->
  <path d="M163,155 L193,155" fill="none" stroke="#a78bfa" stroke-width="1.5"/>
  <polygon points="191,152 197,155 191,158" fill="#a78bfa"/>

  <circle cx="220" cy="155" r="18" fill="#1c1633" stroke="#a78bfa" stroke-width="1.3"/>
  <text x="220" y="152" fill="#c4b5fd" font-size="9" text-anchor="middle">σ</text>
  <text x="220" y="163" fill="#8b98a5" font-size="7" text-anchor="middle">a₁</text>
  <text x="220" y="185" fill="#6b7684" font-size="7.5" text-anchor="middle">ReLU/sigmoid</text>

  <!-- W2 multiply -->
  <path d="M238,155 L278,155" fill="none" stroke="#34d399" stroke-width="1.5"/>
  <polygon points="276,152 282,155 276,158" fill="#34d399"/>
  <text x="258" y="148" fill="#34d399" font-size="8" text-anchor="middle">W₂</text>

  <!-- Node: z2 = W2*a1 -->
  <circle cx="305" cy="155" r="18" fill="#14261f" stroke="#34d399" stroke-width="1.3"/>
  <text x="305" y="152" fill="#6ee7b7" font-size="9" text-anchor="middle">×</text>
  <text x="305" y="163" fill="#8b98a5" font-size="7" text-anchor="middle">z₂</text>

  <!-- Activation 2 -->
  <path d="M323,155 L353,155" fill="none" stroke="#a78bfa" stroke-width="1.5"/>
  <polygon points="351,152 357,155 351,158" fill="#a78bfa"/>

  <circle cx="380" cy="155" r="18" fill="#1c1633" stroke="#a78bfa" stroke-width="1.3"/>
  <text x="380" y="152" fill="#c4b5fd" font-size="9" text-anchor="middle">σ</text>
  <text x="380" y="163" fill="#8b98a5" font-size="7" text-anchor="middle">a₂</text>

  <!-- To loss -->
  <path d="M398,155 L438,155" fill="none" stroke="#f59e0b" stroke-width="1.5"/>
  <polygon points="436,152 442,155 436,158" fill="#f59e0b"/>

  <!-- Loss node -->
  <circle cx="465" cy="155" r="22" fill="#2a1a0a" stroke="#f59e0b" stroke-width="1.5"/>
  <text x="465" y="151" fill="#fbbf24" font-size="10" text-anchor="middle" font-weight="600">L</text>
  <text x="465" y="164" fill="#a1701a" font-size="7" text-anchor="middle">loss</text>
  <text x="465" y="185" fill="#fbbf24" font-size="8" text-anchor="middle">scalar</text>

  <!-- === BACKWARD PASS (red arrows flowing left) === -->
  <!-- dL/dL = 1 -->
  <text x="500" y="145" fill="#f87171" font-size="8">dL/dL=1</text>

  <!-- Backward arrow from L to a2 -->
  <path d="M443,140 L400,130" fill="none" stroke="#f87171" stroke-width="1.8" stroke-dasharray="4,2"/>
  <polygon points="402,127 396,130 400,134" fill="#f87171"/>
  <text x="420" y="125" fill="#f87171" font-size="7.5" text-anchor="middle">dL/da₂</text>

  <!-- Backward from a2 to z2 -->
  <path d="M362,140 L325,130" fill="none" stroke="#f87171" stroke-width="1.8" stroke-dasharray="4,2"/>
  <polygon points="327,127 321,130 325,134" fill="#f87171"/>
  <text x="343" y="123" fill="#f87171" font-size="7.5" text-anchor="middle">× σ'</text>

  <!-- Backward from z2 to a1 -->
  <path d="M287,140 L240,130" fill="none" stroke="#f87171" stroke-width="1.8" stroke-dasharray="4,2"/>
  <polygon points="242,127 236,130 240,134" fill="#f87171"/>
  <text x="263" y="123" fill="#f87171" font-size="7.5" text-anchor="middle">× W₂ᵀ</text>

  <!-- Backward from a1 to z1 -->
  <path d="M202,140 L165,130" fill="none" stroke="#f87171" stroke-width="1.8" stroke-dasharray="4,2"/>
  <polygon points="167,127 161,130 165,134" fill="#f87171"/>
  <text x="183" y="123" fill="#f87171" font-size="7.5" text-anchor="middle">× σ'</text>

  <!-- Gradient to W2 (branch down from z2) -->
  <path d="M305,173 L305,205" fill="none" stroke="#f87171" stroke-width="1.2" stroke-dasharray="3,2"/>
  <polygon points="302,203 305,209 308,203" fill="#f87171"/>
  <rect x="280" y="210" width="50" height="20" rx="3" fill="#2a1a1a" stroke="#f87171" stroke-width="0.8"/>
  <text x="305" y="223" fill="#fca5a5" font-size="8" text-anchor="middle">dL/dW₂</text>

  <!-- Gradient to W1 (branch down from z1) -->
  <path d="M145,173 L145,205" fill="none" stroke="#f87171" stroke-width="1.2" stroke-dasharray="3,2"/>
  <polygon points="142,203 145,209 148,203" fill="#f87171"/>
  <rect x="120" y="210" width="50" height="20" rx="3" fill="#2a1a1a" stroke="#f87171" stroke-width="0.8"/>
  <text x="145" y="223" fill="#fca5a5" font-size="8" text-anchor="middle">dL/dW₁</text>

  <!-- Legend -->
  <text x="60" y="235" fill="#34d399" font-size="8.5">→ forward (compute output)</text>
  <text x="60" y="250" fill="#f87171" font-size="8.5">⇠ backward (compute gradients)</text>

  <!-- === CHAIN RULE EQUATION === -->
  <rect x="30" y="260" width="450" height="50" rx="5" fill="#080d14" stroke="#233043" stroke-width="1"/>
  <text x="255" y="278" fill="#e6edf3" font-size="10" text-anchor="middle" font-weight="600">The Chain Rule (one weight)</text>
  <text x="255" y="298" fill="#fca5a5" font-size="10" text-anchor="middle">dL/dW₁ = dL/da₂ · da₂/dz₂ · dz₂/da₁ · da₁/dz₁ · dz₁/dW₁</text>

  <!-- === RIGHT SIDE: Why it works + what can go wrong === -->
  <rect x="500" y="84" width="230" height="145" rx="6" fill="#0b1220" stroke="#233043" stroke-width="1"/>
  <text x="615" y="102" fill="#e6edf3" font-size="10" text-anchor="middle" font-weight="600">Key Properties</text>

  <text x="515" y="122" fill="#34d399" font-size="9">✓ Exact gradients (not approx)</text>
  <text x="515" y="138" fill="#34d399" font-size="9">✓ O(N) compute (same as forward)</text>
  <text x="515" y="154" fill="#34d399" font-size="9">✓ Embarrassingly parallelizable</text>
  <text x="515" y="174" fill="#f87171" font-size="9">✗ Vanishing gradient (deep nets)</text>
  <text x="515" y="190" fill="#f87171" font-size="9">✗ Exploding gradient (RNNs)</text>
  <text x="515" y="210" fill="#8b98a5" font-size="8.5">Fix: ResNet skips, LayerNorm,</text>
  <text x="515" y="222" fill="#8b98a5" font-size="8.5">gradient clipping, careful init</text>

  <!-- === BOTTOM: Autodiff + implementation === -->
  <rect x="30" y="320" width="700" height="95" rx="6" fill="#0b1220" stroke="#233043" stroke-width="1"/>
  <text x="380" y="340" fill="#e6edf3" font-size="11" text-anchor="middle" font-weight="600">Automatic Differentiation (how frameworks do it)</text>

  <text x="50" y="362" fill="#8b98a5" font-size="9.5">1. Forward pass: build computational graph (tape), compute output</text>
  <text x="50" y="378" fill="#8b98a5" font-size="9.5">2. Backward pass: walk graph in reverse, multiply local gradients (chain rule)</text>
  <text x="50" y="394" fill="#8b98a5" font-size="9.5">3. Accumulate: each weight gets dL/dW → pass to optimizer (Adam, SGD)</text>
  <text x="50" y="410" fill="#6b7684" font-size="9">PyTorch: loss.backward() triggers the entire backward graph. Autograd records ops on tensors with requires_grad=True.</text>

  <!-- Scale numbers -->
  <text x="380" y="435" fill="#e6edf3" font-size="9.5" text-anchor="middle" font-weight="600">GPT-4 training: backprop through 96 transformer layers, 1.8T parameters, 13T tokens — the chain rule at industrial scale</text>

  <text x="380" y="460" fill="#6b7684" font-size="11" text-anchor="middle">Backprop is the only reason deep learning works — one algorithm that lets gradients reach every weight in the network.</text>
</svg>
```ckpropagation** is the algorithm that lets a neural network learn. After the network makes a prediction and we measure how wrong it was, backpropagation efficiently computes how much each of the millions or billions of weights contributed to that error — the gradient — so an optimizer can nudge every weight in the direction that reduces the loss. It is, at heart, the chain rule from calculus applied systematically across a computation graph, and it is what makes training deep networks tractable at all. The diagram shows the two passes: forward to get the error, backward to distribute the blame.\n\n```svg\n<svg viewBox="0 0 760 490" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system,Segoe UI,Roboto,sans-serif">\n  <rect x="0" y="0" width="760" height="490" fill="#0d1117"/>\n  <text x="380" y="30" fill="#e6edf3" font-size="21" font-weight="700" text-anchor="middle">Backpropagation — One Forward, One Backward</text>\n  <text x="380" y="52" fill="#8b98a5" font-size="12.5" text-anchor="middle">run the network forward to get the error, then push that error back to blame every weight</text>\n  <circle cx="120" cy="250" r="24" fill="#0f1c17" stroke="#a78bfa" stroke-width="1.8"/>\n  <text x="120" y="255" fill="#a78bfa" font-size="14" font-weight="700" text-anchor="middle">x</text>\n  <circle cx="250" cy="250" r="24" fill="#0f1c17" stroke="#34d399" stroke-width="1.8"/>\n  <text x="250" y="255" fill="#34d399" font-size="14" font-weight="700" text-anchor="middle">h1</text>\n  <circle cx="380" cy="250" r="24" fill="#0f1c17" stroke="#34d399" stroke-width="1.8"/>\n  <text x="380" y="255" fill="#34d399" font-size="14" font-weight="700" text-anchor="middle">h2</text>\n  <circle cx="510" cy="250" r="24" fill="#0f1c17" stroke="#f87171" stroke-width="1.8"/>\n  <text x="510" y="255" fill="#f87171" font-size="14" font-weight="700" text-anchor="middle">ŷ</text>\n  <line x1="144" y1="250" x2="226" y2="250" stroke="#3f5169" stroke-width="1.4"/>\n  <text x="185" y="242" fill="#8b98a5" font-size="13" text-anchor="middle">W1</text>\n  <line x1="274" y1="250" x2="356" y2="250" stroke="#3f5169" stroke-width="1.4"/>\n  <text x="315" y="242" fill="#8b98a5" font-size="13" text-anchor="middle">W2</text>\n  <line x1="404" y1="250" x2="486" y2="250" stroke="#3f5169" stroke-width="1.4"/>\n  <text x="445" y="242" fill="#8b98a5" font-size="13" text-anchor="middle">W3</text>\n  <rect x="590" y="230" width="72" height="40" rx="6" fill="#2a1420" stroke="#f87171" stroke-width="1.6"/>\n  <text x="626" y="248" fill="#fca5a5" font-size="12" font-weight="700" text-anchor="middle">Loss</text>\n  <text x="626" y="263" fill="#8b98a5" font-size="12" text-anchor="middle">L(ŷ,y)</text>\n  <line x1="534" y1="250" x2="590" y2="250" stroke="#3f5169" stroke-width="1.4"/>\n  <path d="M110 170 L560 170" stroke="#34d399" stroke-width="2" fill="none"/>\n  <path d="M552 165 L560 170 L552 175" stroke="#34d399" stroke-width="2" fill="none"/>\n  <text x="330" y="160" fill="#6ee7b7" font-size="13" font-weight="700" text-anchor="middle">FORWARD PASS — compute prediction &amp; loss</text>\n  <path d="M640 330 L110 330" stroke="#fbbf24" stroke-width="2" fill="none"/>\n  <path d="M118 325 L110 330 L118 335" stroke="#fbbf24" stroke-width="2" fill="none"/>\n  <text x="360" y="352" fill="#fcd34d" font-size="13" font-weight="700" text-anchor="middle">BACKWARD PASS — chain rule sends gradients back</text>\n  \n  \n  \n  <text x="185" y="308" fill="#fcd34d" font-size="13" text-anchor="middle">∂L/∂W1</text>\n  <text x="315" y="308" fill="#fcd34d" font-size="13" text-anchor="middle">∂L/∂W2</text>\n  <text x="445" y="308" fill="#fcd34d" font-size="13" text-anchor="middle">∂L/∂W3</text>\n  <rect x="90" y="400" width="580" height="66" rx="8" fill="#0b1220" stroke="#233043" stroke-width="1.2"/>\n  <text x="110" y="424" fill="#cbd5e1" font-size="12" font-weight="700">Chain rule, reused layer by layer:</text>\n  <text x="110" y="446" fill="#93c5fd" font-size="13" font-family="ui-monospace,monospace">∂L/∂W1 = (∂L/∂ŷ) · (∂ŷ/∂h2) · (∂h2/∂h1) · (∂h1/∂W1)</text>\n  <text x="655" y="446" fill="#6b7684" font-size="12" text-anchor="end"></text>\n  <text x="380" y="482" fill="#6b7684" font-size="13" text-anchor="middle">Each weight&#8217;s gradient is a product of local derivatives already computed one layer downstream — nothing is calculated twice.</text>\n</svg>\n```\n\n**The forward pass computes the prediction and the loss.** Input data flows layer by layer through the network — each layer multiplying by its weights and applying a nonlinearity — until it produces an output. That output is compared to the correct answer with a loss function, giving a single number that measures how wrong the network currently is. Along the way, each layer stores the intermediate values it computed, because the backward pass will need them.\n\n**The backward pass applies the chain rule in reverse.** Starting from the loss, backpropagation works backward through the layers, computing at each step how the loss changes with respect to that layer's inputs and weights. The key efficiency is reuse: the gradient at layer *k* is built directly from the gradient already computed at layer *k+1*, multiplied by a local derivative. Nothing is recomputed, which is why a full gradient over billions of parameters costs only about twice a forward pass.\n\n**Gradients are just directions for improvement.** The gradient with respect to a weight answers one question — if I increase this weight slightly, does the loss go up or down, and how fast? Backpropagation produces that answer for every weight at once. It does not change anything itself; it only measures. The actual learning step is handed to an optimizer such as SGD or Adam.\n\n**The vanishing-gradient problem shaped modern architectures.** When gradients are repeatedly multiplied through many layers, they can shrink toward zero (or blow up), stalling learning in the earliest layers. Much of deep-learning design — ReLU activations, residual/skip connections, careful normalization and initialization — exists specifically to keep gradients healthy as they propagate back through great depth.\n\n**It requires stored activations, which is why training is memory-hungry.** Because the backward pass needs the intermediate values from the forward pass, they must be kept in memory until used. This is a major reason training a model costs far more memory than running it, and it motivates techniques like gradient (activation) checkpointing, which trade recomputation for reduced memory.\n\n| Step | Direction | Produces | Cost |\n|---|---|---|---|\n| Forward pass | input → output | prediction + loss | one pass |\n| Backward pass | loss → inputs | gradient for every weight | about one pass |\n| Optimizer step | — | updated weights | cheap |\n| Repeat | over many batches | a trained model | the whole training run |\n\nRead backpropagation through a *credit-assignment* lens rather than a *magic-learning* lens: the entire algorithm is a bookkeeping method for answering "how much did each weight contribute to this mistake?" without redoing work, by caching local derivatives on the way in and multiplying them together on the way out. Every scaling and stability trick in deep learning — residual connections, normalization, mixed precision, activation checkpointing — is ultimately about keeping that backward flow of credit accurate, fast, and affordable.\n