data mix

Data mix balances training data across domains like web text books code and papers with proportions affecting model capabilities. Optimal mixing is empirically determined through ablation studies. More code improves reasoning and structured thinking. More books improve long-form coherence and writing quality. More web data improves factual knowledge and diversity. Scientific papers improve technical reasoning. The mix is typically specified as percentages: 60 percent web 20 percent books 15 percent code 5 percent papers. Upsampling high-quality sources and downsampling low-quality sources improves outcomes. Dynamic mixing adjusts proportions during training. Curriculum learning starts with easier domains. Data mix affects downstream task performance: code-heavy mixes excel at programming while book-heavy mixes excel at creative writing. Documenting data mix enables reproducibility and analysis. Challenges include determining optimal proportions handling domain imbalance and ensuring diversity. Data mix is a key hyperparameter for pretraining often as important as model architecture. Careful mixing produces well-rounded models with broad capabilities.