training data quality vs quantity
**Training data quality vs quantity** is the **tradeoff between adding more tokens and improving corpus quality to maximize model learning efficiency** - balancing these factors is critical for effective scaling and reliable behavior.
**What Is Training data quality vs quantity?**
- **Definition**: Quantity increases coverage while quality determines signal-to-noise of learned patterns.
- **Quality Dimensions**: Includes correctness, diversity, deduplication, domain relevance, and toxicity control.
- **Failure Modes**: High volume of low-quality data can dilute useful gradients and amplify harmful artifacts.
- **Optimization**: Best outcomes usually require both sufficient scale and high curation quality.
**Why Training data quality vs quantity Matters**
- **Capability**: High-quality data can unlock larger gains than raw token growth alone.
- **Safety**: Quality filtering reduces harmful behavior and undesirable memorization.
- **Compute ROI**: Better data quality improves effectiveness of each training token.
- **Generalization**: Cleaner diverse corpora support more robust downstream performance.
- **Strategy**: Informs whether to invest in data curation pipeline versus corpus expansion.
**How It Is Used in Practice**
- **Ablation Studies**: Compare quality-improved subsets against larger unfiltered baselines.
- **Pipeline Metrics**: Track deduplication, toxicity, and domain-balance indicators continuously.
- **Adaptive Sampling**: Increase weighting of high-value domains aligned with capability goals.
Training data quality vs quantity is **a central optimization tradeoff in modern large-model training** - training data quality vs quantity should be managed as a joint optimization problem, not a single-axis scaling decision.