corpus

Training corpus is the text collection used for pretraining language models typically including web crawls books code and papers. Corpus composition critically affects model capabilities. Common sources include Common Crawl web scrapes Books3 literature corpus GitHub code repositories arXiv scientific papers Wikipedia encyclopedic knowledge and curated datasets. Quality and diversity matter more than raw size. Preprocessing includes deduplication removing near-duplicates filtering removing low-quality content toxicity filtering and format normalization. Data mix proportions affect capabilities: more code improves reasoning more books improve coherence more web data improves factual knowledge. Multilingual corpora enable cross-lingual transfer. Corpus curation involves balancing domains languages and quality levels. Challenges include copyright concerns toxic content and bias. Modern models train on trillions of tokens from diverse sources. Corpus documentation enables reproducibility and analysis. The Pile and RedPajama are open training corpora. Corpus quality is often more important than size for model performance. Careful curation produces better models than indiscriminate web scraping.