dataset
**Training Data for LLMs**
**Pretraining Datasets**
Large language models are pretrained on massive text corpora—often trillions of tokens from diverse sources.
**Common Pretraining Sources**
| Source | Content | Scale |
|--------|---------|-------|
| Common Crawl | Web pages | Petabytes |
| The Pile | Curated diverse text | 825 GB |
| Wikipedia | Encyclopedia articles | ~20 GB |
| Books3 | Books | ~100 GB |
| GitHub | Source code | ~150 GB |
| ArXiv | Scientific papers | ~90 GB |
| Stack Exchange | Q&A | ~60 GB |
**Data Processing Pipeline**
1. **Crawling**: Collect raw text from sources
2. **Deduplication**: Remove duplicate documents
3. **Filtering**: Remove low-quality, toxic, or harmful content
4. **Language detection**: Filter by language if needed
5. **Tokenization**: Convert to token sequences
6. **Shuffling**: Randomize for training
**Fine-Tuning Datasets**
**By Task Type**
| Task | Datasets | Size |
|------|----------|------|
| Instruction | Alpaca, Dolly, OpenAssistant | 15K-200K |
| Code | CodeAlpaca, StarCoder data | 20K-1M |
| Math | GSM8K, MATH | 8K-12K |
| Dialogue | ShareGPT, UltraChat | 50K-1M |
| Safety | Anthropic HH-RLHF | 160K |
**Data Quality Principles**
**Quality > Quantity**
Research shows that smaller, high-quality datasets often outperform larger noisy ones:
- Phi-1: 1.3B model trained on 6B tokens of textbook-quality data
- LIMA: 1K carefully curated examples for instruction tuning
**Key Quality Factors**
- **Accuracy**: Factually correct information
- **Diversity**: Wide coverage of topics and styles
- **Consistency**: Uniform formatting and quality standards
- **Recency**: Up-to-date information when relevant
- **Safety**: No harmful, biased, or toxic content
**Legal Considerations**
- Respect copyright and licensing
- Consider opt-out mechanisms for data subjects
- Document data provenance for compliance