domain-adaptive pre-training

**Domain-Adaptive Pre-training (DAPT)** is the **process of taking a general-purpose pre-trained model (like BERT) and continuing to pre-train it on a large corpus of unlabeled text from a specific domain (e.g., biomedical, legal, financial)** — adapting the model's vocabulary and statistical understanding to the target domain before fine-tuning. **Process (Don't Stop Pre-training)** - **Source**: Start with RoBERTa (trained on CommonCrawl). - **Target**: Continue training MLM on all available Biomedical papers (PubMed). - **Result**: "BioRoBERTa" — better at medical jargon and scientific reasoning. - **Fine-tune**: Finally, fine-tune on the specific medical task (e.g., diagnosis prediction). **Why It Matters** - **Vocabulary Shift**: "Virus" means something different in biology vs. computer security. DAPT updates context. - **Performance**: Significant gains on in-domain tasks compared to generic models. - **Cost**: Much cheaper than pre-training from scratch on domain data. **Domain-Adaptive Pre-training** is **specializing the expert** — sending a generalist model to law school or med school to learn the specific language of a field.