bitter lesson

The Bitter Lesson is Rich Sutton's influential 2019 essay arguing that the biggest lesson from 70 years of AI research is that general methods leveraging computation are ultimately the most effective, consistently outperforming approaches that attempt to build in human knowledge. Historical evidence: (1) chess (Deep Blue's search beat handcrafted evaluation), (2) speech recognition (statistical/neural methods beat phonetic rules), (3) computer vision (deep learning beat hand-engineered features like SIFT/HOG), (4) Go (AlphaGo/AlphaZero's search + learning beat expert heuristics), (5) NLP (transformers + scale beat linguistic rules). Core argument: researchers repeatedly invest effort in encoding human knowledge into systems, and these approaches show initial gains but are eventually surpassed by simpler methods that scale with compute. The "bitter" part: researchers' intellectual contributions (clever features, domain knowledge) become irrelevant as compute grows. Implications for modern AI: scaling laws validate this—larger models with more data consistently outperform smaller, more cleverly designed ones (GPT series, Chinchilla). Counterarguments: compute efficiency matters (not just raw scale), domain knowledge helps with data efficiency, and safety/alignment may require structured approaches. The lesson has shaped the "scale is all you need" philosophy driving large language model development.