micrograd

**micrograd** is a **tiny autograd engine created by Andrej Karpathy that implements backpropagation and a dynamic computation graph in under 100 lines of Python** — demonstrating that the core mechanism behind PyTorch, TensorFlow, and all modern deep learning frameworks (automatic differentiation via reverse-mode accumulation on a directed acyclic graph) can be understood by reading a single file, making it the most influential educational resource for demystifying how neural networks actually learn. **What Is micrograd?** - **Definition**: A minimal automatic differentiation engine that implements scalar-valued backpropagation — each `Value` object tracks its data, gradient, the operation that created it, and its parent nodes, forming a computation graph that `backward()` traverses in reverse topological order to compute gradients via the chain rule. - **Creator**: Andrej Karpathy — former Director of AI at Tesla, founding member of OpenAI, and Stanford CS231n instructor. micrograd accompanies his legendary "Neural Networks: Zero to Hero" YouTube lecture series. - **Educational Purpose**: micrograd exists to teach, not to compete — it proves that PyTorch is "not magic" by showing that the entire autograd mechanism (the engine that computes gradients for training neural networks) fits in 100 lines of readable Python. - **Scalar Operations**: Unlike PyTorch (which operates on tensors/matrices), micrograd operates on individual scalar values — making every gradient computation explicit and traceable at the single-number level. **Core Implementation** The entire engine is built around a `Value` class: - **data**: The scalar value (a single float). - **grad**: The gradient of the loss with respect to this value (accumulated during backward pass). - **_backward**: A closure that computes the local gradient contribution. - **_prev**: Set of parent Value nodes in the computation graph. - **backward()**: Topological sort of the graph, then call `_backward()` on each node in reverse order — this is backpropagation. **Supported Operations**: Addition, multiplication, power, ReLU, negation, subtraction, division — enough to build multi-layer perceptrons and train them with gradient descent. **Why micrograd Matters** - **Demystifies Deep Learning**: Reading micrograd's 100 lines teaches you that neural network training is just: (1) build a math expression graph, (2) compute the output (forward pass), (3) walk the graph backward computing derivatives (backward pass), (4) nudge each parameter in the direction that reduces the loss. - **"Software 2.0" Foundation**: Karpathy uses micrograd to teach that neural networks are mathematical expressions optimized via gradient descent — the foundation of his "Software 2.0" thesis that neural networks are a new programming paradigm. - **Gateway to PyTorch**: After understanding micrograd, PyTorch's `autograd` module becomes transparent — it's the same algorithm operating on tensors instead of scalars, with GPU acceleration and thousands of optimized operations. - **Millions of Learners**: The accompanying YouTube video has millions of views — micrograd has taught more people how backpropagation works than any textbook. **micrograd is the 100-line Python program that demystified deep learning for millions of developers** — proving that the autograd engine at the heart of every modern ML framework is simply reverse-mode differentiation on a computation graph, making neural network training conceptually accessible to anyone who can read basic Python.