task-specific pre-training

**Task-Specific Pre-training** is an **intermediate step between general pre-training and fine-tuning, where the model is further pre-trained on valid data using objectives closely related to the final target task** — bridging the gap between the generic MLM objective and the specific downstream application. **Mechanism** - **Phase 1**: General Pre-training (Wiki + Books, MLM). - **Phase 2 (Task-Specific)**: Continue training on domain data using designated objectives (e.g., Gap Sentence Generation for Summarization). - **Phase 3**: Fine-tuning on labeled data. **Why It Matters** - **Alignment**: Standard MLM is not aligned with generation or retrieval. Task-specific pre-training aligns the internal representations. - **Performance**: Consistently improves performance, especially when labeled data is scarce. - **Domain**: Often combined with Domain-Adaptive Pre-training (DAPT). **Task-Specific Pre-training** is **specialized drills** — practicing the specific mechanics of the final game (reordering, summarizing) before the actual match.