tinygrad
**tinygrad** is a **minimalist deep learning framework created by George Hotz (geohot) that implements a complete neural network training and inference system in under 3,000 lines of code** — built on the philosophy that "complex software is buggy software," tinygrad uses lazy evaluation to build computation graphs that compile to raw C, CUDA, Metal, or OpenCL shaders, serving as both a production-capable framework and the single best codebase for understanding how deep learning frameworks work under the hood.
**What Is tinygrad?**
- **Definition**: A tiny but fully functional deep learning framework that implements tensor operations, automatic differentiation, optimizers, and hardware compilation in a deliberately minimal codebase — proving that the core of PyTorch can be expressed in thousands, not millions, of lines of code.
- **Creator**: George Hotz (geohot) — known for being the first person to jailbreak the iPhone and for founding comma.ai (self-driving car company). tinygrad is used in comma.ai's production self-driving stack.
- **Lazy Evaluation**: tinygrad doesn't execute operations immediately — it builds a computation graph (AST) and only materializes results when data is explicitly requested, enabling the compiler to fuse operations and optimize memory access patterns before execution.
- **Multi-Backend Compilation**: The lazy computation graph compiles to raw C (CPU), CUDA (NVIDIA), Metal (Apple), OpenCL, Vulkan, and even WebGPU — the same model code runs on any hardware through backend-specific code generation.
**Key Features**
- **Under 3,000 Lines**: The core framework (tensor operations, autograd, optimizers, compilation) fits in ~3,000 lines of Python — readable in a single sitting, making it the best educational resource for understanding deep learning internals.
- **Production Use**: Despite its size, tinygrad runs comma.ai's production neural networks for self-driving — proving that minimal code can be production-grade.
- **Kernel Fusion**: The lazy evaluation engine automatically fuses element-wise operations into single GPU kernels — reducing kernel launch overhead and memory bandwidth usage.
- **Custom Accelerator Support**: tinygrad's compilation approach makes it straightforward to add new hardware backends — the community has added support for AMD GPUs, Intel GPUs, and custom accelerators.
**tinygrad vs Alternatives**
| Feature | tinygrad | PyTorch | JAX | micrograd |
|---------|---------|---------|-----|----------|
| Lines of code | ~3,000 | ~3,000,000 | ~500,000 | ~100 |
| Training | Yes | Yes | Yes | Yes (scalar only) |
| GPU support | CUDA, Metal, OpenCL, Vulkan | CUDA, ROCm, MPS | CUDA, TPU | No |
| Lazy evaluation | Yes | No (eager) | Yes | No |
| Production use | Yes (comma.ai) | Yes (everywhere) | Yes (Google) | No (educational) |
| Educational value | Excellent | Low (too complex) | Medium | Excellent (basics) |
**tinygrad is the proof that deep learning frameworks don't need millions of lines of code** — implementing a complete, production-capable training and inference system in under 3,000 lines that compiles to any GPU backend, serving as both a practical framework for comma.ai's self-driving cars and the best educational resource for understanding how PyTorch works under the hood.