data-constrained regime

**Data-constrained regime** is the **training regime where model performance is primarily limited by insufficient effective data rather than compute or model size** - it indicates that adding high-quality tokens may yield better returns than increasing parameters. **What Is Data-constrained regime?** - **Definition**: Model capacity and compute are available, but data coverage or novelty becomes bottleneck. - **Symptoms**: Loss improvements stall unless new diverse data is introduced. - **Quality Dependence**: Low-diversity or duplicated corpora can trigger data constraints earlier. - **Implication**: Scaling model size alone may not improve capability substantially. **Why Data-constrained regime Matters** - **Strategy**: Guides investment toward data acquisition, cleaning, and curation. - **Efficiency**: Prevents overspending on parameters with limited data support. - **Capability Growth**: High-quality data expansion can unlock stalled performance. - **Safety**: Better data quality can reduce harmful behavior learned from noisy sources. - **Roadmap**: Helps prioritize corpus engineering as a first-class scaling lever. **How It Is Used in Practice** - **Data Audit**: Quantify diversity, duplication, and domain coverage gaps. - **Corpus Expansion**: Add targeted high-value data aligned to capability objectives. - **Ablation**: Test gains from new data slices before large retraining commitments. Data-constrained regime is **a key bottleneck mode in mature model training pipelines** - data-constrained regime detection should trigger immediate focus on corpus quality and coverage rather than blind parameter scaling.