Data Curation
Data curation is the work of selecting, cleaning, filtering, deduplicating, balancing, and sometimes rewriting or generating the examples a model is trained on, so that the training set is better than the raw pile it was drawn from. For large language models the raw pile is usually a web crawl, and curation decides which fraction of it is worth spending compute on. The guiding bet is that, beyond a certain point, improving what a model reads yields more than simply making it read more.
Quality over quantity became a headline claim with "Textbooks Are All You Need" (Gunasekar et al., Microsoft Research, 2023), which introduced phi-1, a model of about 1.3 billion parameters for Python code. It was trained on a few billion tokens of web code filtered for "textbook quality" using a classifier, plus about a billion tokens of synthetic textbook-style text and exercises generated with GPT-3.5, and it reached roughly 50 percent pass@1 on the HumanEval coding benchmark, a result competitive with models many times larger and trained on far more data. The paper's message was that clear, self-contained, instructive examples teach more per token than the noisy mix found in typical scrapes. Later phi models extended the recipe to general text, though critics noted the heavy reliance on synthetic data from a stronger model, and the question of how well benchmark scores generalize.
Curation is a pipeline of many small decisions, not one filter. Typical stages include language identification, removal of boilerplate such as navigation menus, heuristic quality rules (minimum length, symbol-to-word ratios, repetition checks), model-based quality scoring, deduplication at document or paragraph level, and removal of personal data or unsafe material. Lee et al. (2022, "Deduplicating Training Data Makes Language Models Better") showed that exact and near-duplicate text inflates memorization and wastes compute, and that removing it improves perplexity and reduces verbatim regurgitation. Each stage trades recall for precision, so aggressive filtering can also strip out dialects, niche topics, or minority viewpoints that the heuristics happen to rate as low quality.
FineWeb (Penedo et al., Hugging Face, 2024) is a well-documented open example of web-scale curation done with ablations. It is a dataset of about 15 trillion tokens built from 96 Common Crawl snapshots, and the authors released the pipeline and trained small models on candidate filtering choices to see which actually helped. A derived subset, FineWeb-Edu, kept pages scored as educational by a classifier trained on annotations from a Llama 3 model, and models trained on it did notably better on knowledge and reasoning benchmarks such as MMLU and ARC. This ablation-driven method, measuring each filter by the downstream performance of a small trained model, is now a standard way to decide between curation recipes, and earlier datasets such as C4 (Raffel et al., 2020) and the Pile (Gao et al., 2020) used the same spirit with fewer experiments.
Curation applies beyond pretraining, and the same logic works at smaller scales. LIMA (Zhou et al., 2023) fine-tuned a model on only about a thousand carefully chosen prompt and response pairs and found it surprisingly capable, supporting the "superficial alignment" argument that most knowledge comes from pretraining and a small clean set can teach style and format. Instruction-tuning, preference data, and reward-model datasets all benefit from the same careful selection. In each case, curation also touches benchmark-contamination: removing test examples from training data is itself a curation step, and web-crawl sources such as Common Crawl are the raw material most pipelines start from.
The common misconception is that quality always beats quantity, or that a filter can be applied once and forgotten. The scaling-law literature, including the Chinchilla paper (Hoffmann et al., 2022), shows that data quantity matters substantially, and filtering that discards too much leaves a model undertrained or repeated over a small corpus. What counts as quality is also relative to the target use: text ideal for coding or math may be poor for dialogue. The sound view is that curation shifts the efficiency frontier, so the honest question is not "more or better" but how much capability each token of compute buys, measured on benchmarks that were themselves not leaked into the data.
Read data curation as editing before teaching: a model learns whatever its training set contains, so deciding what to keep, what to drop, and what to rewrite is a design decision about the model's knowledge and blind spots as much as a cleaning chore.