deduplication

Deduplication removes repeated or near-duplicate text from training corpora, improving data quality, training efficiency, and model generalization by preventing memorization and overrepresentation of duplicated content. Why important: web crawl data contains massive duplication (mirror sites, boilerplate, copied content); training on duplicates wastes compute, biases models toward repeated content, and increases memorization risks (privacy, copyright). Exact deduplication: hash-based matching (MD5, SHA)—fast but misses near-duplicates. Near-duplicate detection: MinHash/LSH (approximate similarity via hashing), n-gram overlap (Jaccard similarity of text shingles), and embedding similarity (semantic duplicates). Common approaches: document-level (remove entire duplicate documents), paragraph-level (remove repeated paragraphs), and substring-level (remove repeated phrases/boilerplate). Fuzzy matching: allow small variations (whitespace, formatting, minor edits). Scale considerations: web-scale requires efficient algorithms—exact comparison is O(n²); MinHash enables sublinear scaling. ThePile, C4, and other curated corpora use deduplication as essential preprocessing. Impact: deduplicated training shows improved perplexity and downstream performance compared to raw data. Deduplication is often combined with other filtering (quality, language, toxicity) for comprehensive data curation.