Training Data for LLMs

Pretraining Datasets Large language models are pretrained on massive text corpora—often trillions of tokens from diverse sources.

Common Pretraining Sources

Data Processing Pipeline 1. Crawling: Collect raw text from sources 2. Deduplication: Remove duplicate documents 3. Filtering: Remove low-quality, toxic, or harmful content 4. Language detection: Filter by language if needed 5. Tokenization: Convert to token sequences 6. Shuffling: Randomize for training

Fine-Tuning Datasets

By Task Type

Data Quality Principles

Quality > Quantity Research shows that smaller, high-quality datasets often outperform larger noisy ones:

• Phi-1: 1.3B model trained on 6B tokens of textbook-quality data
• LIMA: 1K carefully curated examples for instruction tuning

Key Quality Factors

• Accuracy: Factually correct information
• Diversity: Wide coverage of topics and styles
• Consistency: Uniform formatting and quality standards
• Recency: Up-to-date information when relevant
• Safety: No harmful, biased, or toxic content

Legal Considerations

• Respect copyright and licensing
• Consider opt-out mechanisms for data subjects
• Document data provenance for compliance