Data deduplication is the process of identifying and removing duplicate or near-duplicate examples from a dataset. It is a critical data quality step for training language models, as duplicate data can waste compute, bias the model toward overrepresented content, and inflate evaluation metrics through train-test leakage.

Why Deduplication Matters

• Training Efficiency: Duplicate examples waste training compute on content the model has already seen.
• Memorization Risk: High duplication rates increase the chance of the model memorizing and regurgitating specific training examples verbatim.
• Evaluation Contamination: If duplicates exist across train and test splits, evaluation metrics are inflated.
• Distribution Skew: Overrepresented content biases the model toward certain topics, styles, or sources.

Deduplication Methods

• Exact Deduplication: Hash each example (using MD5, SHA-256) and remove exact matches. Fast and simple.
• URL Deduplication: For web data, deduplicate based on source URL before processing content.
• MinHash + LSH: MinHash creates compact signatures of document content, and Locality-Sensitive Hashing (LSH) efficiently groups similar documents. The standard approach for large-scale near-duplicate detection.
• Suffix Array: Build a suffix array over the concatenated corpus to find shared substrings. Used by the Llama and GPT training pipelines.
• Embedding-Based: Compute embeddings of each document and cluster by similarity. More expensive but catches semantic duplicates.

Scale Considerations

• Web-scale datasets like Common Crawl contain 30–50% duplicate content that must be removed.
• Efficient deduplication at trillion-token scale requires distributed, O(N) algorithms — exact comparison (O(N²)) is infeasible.

Best Practice: Apply deduplication at multiple granularities — document level, paragraph level, and even sentence level for critical datasets. The RefinedWeb dataset demonstrated that aggressive deduplication significantly improves downstream model performance.