slanted triangular learning rates

**Slanted Triangular Learning Rates (STLR)** is a **learning rate schedule introduced in ULMFiT** — that quickly increases the learning rate early in training (warm-up) and then linearly decays it for the remainder, creating a skewed triangular shape that balances fast convergence with careful fine-tuning. **How Does STLR Work?** - **Shape**: Sharp rise to peak (5-10% of training), then gradual linear decay (90-95%). - **Intuition**: High LR early to quickly adapt to the new task's loss landscape. Low LR later for fine-grained optimization. - **Parameters**: Peak LR, cut fraction (fraction of iterations spent warming up), and ratio (LR at start vs. peak). **Why It Matters** - **Fast Convergence**: The warm-up phase helps escape the pre-trained loss basin quickly. - **Stability**: The long decay phase prevents overshooting and allows careful fine-tuning. - **Widely Adopted**: The warm-up + decay paradigm (now often called "linear warmup + cosine/linear decay") is standard in transformer training. **STLR** is **the ramp-up-then-slow-down schedule** — a simple but effective learning rate policy that became the blueprint for modern training schedules.