nanoGPT
**nanoGPT** is a **minimal, readable implementation of GPT-2/GPT-3 training and inference created by Andrej Karpathy in two files of clean PyTorch code** — designed to be the simplest possible codebase that can reproduce GPT-2 (124M parameters) training on a single GPU, enabling thousands of engineers to understand transformer language models by stepping through the training loop line-by-line in a debugger rather than navigating Hugging Face's deep abstraction layers.
**What Is nanoGPT?**
- **Definition**: A ~300-line PyTorch implementation of the GPT architecture (decoder-only transformer with causal attention) that includes both training (`train.py`) and inference (`sample.py`) — the entire GPT-2 architecture, training loop, and text generation in two readable files.
- **Creator**: Andrej Karpathy — created nanoGPT as part of his mission to make deep learning fundamentally understandable, following micrograd (autograd in 100 lines) with a complete language model in 300 lines.
- **Reproducible GPT-2**: nanoGPT can reproduce OpenAI's GPT-2 (124M) training results on OpenWebText — training on a single A100 GPU in ~4 days, achieving comparable perplexity to the original model.
- **Simplicity Over Generality**: Unlike Hugging Face Transformers (which handles 100+ architectures with deep abstraction trees), nanoGPT implements exactly one architecture (GPT) with zero abstraction — every line of code maps directly to a concept in the "Attention Is All You Need" paper.
**What nanoGPT Teaches**
- **Transformer Architecture**: The complete GPT block — multi-head causal self-attention, layer normalization, feed-forward network with GELU activation, residual connections — all visible in a single `Block` class.
- **Training Loop**: Data loading, forward pass, loss computation (cross-entropy), backward pass, gradient clipping, optimizer step (AdamW), learning rate scheduling — the complete training recipe in one function.
- **Text Generation**: Autoregressive sampling with temperature, top-k — the `generate()` method shows exactly how language models produce text token by token.
- **Scaling**: The same code trains a 124M GPT-2 or a larger model by changing config parameters — demonstrating that model scaling is just changing dimensions, not changing architecture.
**nanoGPT vs Alternatives**
| Feature | nanoGPT | HF Transformers | Megatron-LM | GPT-NeoX |
|---------|---------|----------------|-------------|----------|
| Lines of code | ~300 | ~300,000 | ~50,000 | ~30,000 |
| Architectures | GPT only | 100+ | GPT only | GPT only |
| Purpose | Education | Production | Large-scale training | Large-scale training |
| Readability | Excellent | Complex | Complex | Complex |
| Multi-GPU | Basic DDP | Full | Full (3D parallelism) | Full |
| Can reproduce GPT-2 | Yes | Yes | Yes | Yes |
**nanoGPT is the repository that taught a generation of engineers how transformer language models actually work** — by implementing GPT-2 training and inference in 300 lines of transparent PyTorch code, Karpathy created the definitive educational resource that makes the architecture behind ChatGPT, Claude, and every modern LLM fundamentally understandable.