ULMFiT (Universal Language Model Fine-Tuning) is a pioneering transfer learning method for NLP — demonstrating that pre-trained language models can be effectively fine-tuned for text classification with very few labeled examples, using techniques like discriminative fine-tuning, progressive unfreezing, and slanted triangular learning rates.

What Is ULMFiT?

• Three Stages:

1. LM Pre-Training: Pre-train an AWD-LSTM language model on a large corpus (Wikitext-103). 2. LM Fine-Tuning: Fine-tune the LM on the target domain text (unsupervised). 3. Classifier Fine-Tuning: Add a classifier head and fine-tune using labeled data with progressive unfreezing + discriminative LR.

• Paper: Howard & Ruder (2018).

Why It Matters

• NLP Transfer Pioneer: Demonstrated that ImageNet-style transfer learning works for NLP — before BERT and GPT.
• Key Techniques: Introduced discriminative fine-tuning, progressive unfreezing, and STLR — now widely adopted.
• Impact: Directly inspired the pre-train/fine-tune paradigm that dominates modern NLP (BERT, GPT, T5).

ULMFiT is the grandfather of modern NLP transfer learning — the paper that proved pre-trained language models could be fine-tuned for any text task with minimal data.