ulmfit

**ULMFiT** (Universal Language Model Fine-Tuning) is a **pioneering transfer learning method for NLP** — demonstrating that pre-trained language models can be effectively fine-tuned for text classification with very few labeled examples, using techniques like discriminative fine-tuning, progressive unfreezing, and slanted triangular learning rates. **What Is ULMFiT?** - **Three Stages**: 1. **LM Pre-Training**: Pre-train an AWD-LSTM language model on a large corpus (Wikitext-103). 2. **LM Fine-Tuning**: Fine-tune the LM on the target domain text (unsupervised). 3. **Classifier Fine-Tuning**: Add a classifier head and fine-tune using labeled data with progressive unfreezing + discriminative LR. - **Paper**: Howard & Ruder (2018). **Why It Matters** - **NLP Transfer Pioneer**: Demonstrated that ImageNet-style transfer learning works for NLP — before BERT and GPT. - **Key Techniques**: Introduced discriminative fine-tuning, progressive unfreezing, and STLR — now widely adopted. - **Impact**: Directly inspired the pre-train/fine-tune paradigm that dominates modern NLP (BERT, GPT, T5). **ULMFiT** is **the grandfather of modern NLP transfer learning** — the paper that proved pre-trained language models could be fine-tuned for any text task with minimal data.