filtering

Data filtering removes low-quality examples from training datasets using heuristics, rule-based systems, or trained classifiers, which is essential for LLM pretraining where indiscriminate web crawl data introduces noise, toxic content, and duplicates that degrade model quality. Filtering approaches: heuristic filters (minimum text length, language detection, character distribution, punctuation ratios), quality classifiers (trained on high-quality sources like Wikipedia to score web text), deduplication (exact and near-duplicate removal using MinHash or suffix arrays), and content filters (removing toxic, adult, or illegal content using trained classifiers). Common heuristics: exclude pages with too few words, high symbol ratios, or non-target languages; remove boilerplate (headers, footers, navigation); and filter by compression ratio (too compressible suggests repetitive text). Quality classifier training: label Wikipedia, books, academic papers as "high quality"; label random web text as "low quality"; train classifier and filter to high-quality scores. Trade-offs: aggressive filtering reduces noise but may remove legitimate domain diversity; light filtering retains more coverage but includes more noise. Data quality has emerged as critical for LLM training—filtering decisions significantly impact model capabilities and safety properties. Clean data reduces training compute needed for equivalent capability.