llm pretraining data
**LLM Pretraining Data Curation** is the **systematic process of collecting, filtering, deduplicating, and mixing text corpora to create the training dataset for large language models** — with research consistently showing that data quality and mixture composition are as important as model architecture and scale, where a well-curated 1T token dataset can outperform a poorly curated 5T token dataset on downstream benchmarks.
**Scale of Modern LLM Training Data**
- GPT-3 (2020): ~300B tokens
- LLaMA 1 (2023): 1.4T tokens
- LLaMA 2 (2023): 2T tokens
- Llama 3 (2024): 15T tokens
- Gemini Ultra (2024): ~100T tokens
- Chinchilla law: Optimal tokens ≈ 20× parameters (for compute-optimal training)
**Data Sources**
| Source | Examples | Content Type |
|--------|---------|-------------|
| Web crawl | Common Crawl, CC-Net | Broad internet text |
| Curated web | OpenWebText, C4, ROOTS | Filtered web |
| Books | Books3, PG-19, BookCorpus | Long-form narrative |
| Code | GitHub, Stack Exchange | Source code |
| Academic | ArXiv, PubMed, S2ORC | Scientific papers |
| Encyclopedia | Wikipedia, Wikidata | Factual knowledge |
| Conversations | Reddit, HN, Stack Overflow | Dialog, Q&A |
**Common Crawl Processing Pipeline**
1. **Language identification**: Keep only target language(s). Tool: FastText LangDetect.
2. **Quality filtering**:
- Perplexity filtering: Train small KenLM on Wikipedia → remove low-quality text (too high or too low perplexity).
- Heuristic filters: Minimum length (200 tokens), fraction of alphabetic characters > 0.7, word repetition rate < 0.2.
- Blocklist: Remove URLs from spam/adult content lists.
3. **Deduplication**:
- Exact: Remove documents with identical SHA256 hash.
- Near-duplicate: MinHash + LSH → remove documents with > 80% Jaccard similarity.
- N-gram bloom filter: Remove documents sharing many 13-gram spans.
4. **PII removal**: Remove phone numbers, emails, SSNs via regex.
**Data Mixing and Proportions**
- Final mixture combines sources at specific proportions:
- Llama 3: ~50% general web, ~30% code, ~10% books, ~10% multilingual
- Falcon-180B: 80% web, 6% books, 6% code, 3% academic
- Up-weighting quality: Books, Wikipedia up-weighted 5–10× vs raw web crawl.
- Code weight: Higher code proportion → better reasoning, not just coding (see Llama 3).
**Data Quality Models (DSIR, MATES)**
- DSIR (Data Selection via Importance Resampling): Score documents by importance relative to target distribution → sample proportional to importance.
- MATES: Use small proxy model to score document quality → select high-scoring documents.
- FineWeb: Hugging Face's quality-filtered Common Crawl (15T tokens); aggressive quality filtering → FineWeb-Edu focuses on educational content.
**Contamination and Benchmark Leakage**
- Problem: Test benchmarks may appear in training data → inflated benchmark scores.
- Detection: N-gram overlap between training data and benchmark questions.
- Mitigation: Remove benchmark splits from training data; evaluate on new, held-out benchmarks.
- Time-based split: Evaluate on data after a cutoff date not in training.
LLM pretraining data curation is **the hidden engineering that separates excellent from mediocre language models** — Llama 3's remarkable quality despite being a relatively standard architecture compared to its contemporaries is attributed largely to superior data curation using quality classifiers and balanced domain mixing, confirming that in the era of large language models, the dataset IS the model in many respects, and that investments in data quality compound through the entire training process into measurably better downstream capabilities.