data deduplication

**Data deduplication** is the process of identifying and removing **duplicate or near-duplicate** examples from a dataset. It is a critical data quality step for training language models, as duplicate data can waste compute, bias the model toward overrepresented content, and inflate evaluation metrics through train-test leakage. **Why Deduplication Matters** - **Training Efficiency**: Duplicate examples waste training compute on content the model has already seen. - **Memorization Risk**: High duplication rates increase the chance of the model **memorizing** and regurgitating specific training examples verbatim. - **Evaluation Contamination**: If duplicates exist across train and test splits, evaluation metrics are inflated. - **Distribution Skew**: Overrepresented content biases the model toward certain topics, styles, or sources. **Deduplication Methods** - **Exact Deduplication**: Hash each example (using **MD5, SHA-256**) and remove exact matches. Fast and simple. - **URL Deduplication**: For web data, deduplicate based on source URL before processing content. - **MinHash + LSH**: **MinHash** creates compact signatures of document content, and **Locality-Sensitive Hashing (LSH)** efficiently groups similar documents. The standard approach for large-scale near-duplicate detection. - **Suffix Array**: Build a suffix array over the concatenated corpus to find shared substrings. Used by the **Llama** and **GPT** training pipelines. - **Embedding-Based**: Compute embeddings of each document and cluster by similarity. More expensive but catches semantic duplicates. **Scale Considerations** - Web-scale datasets like **Common Crawl** contain **30–50% duplicate content** that must be removed. - Efficient deduplication at trillion-token scale requires distributed, O(N) algorithms — exact comparison (O(N²)) is infeasible. **Best Practice**: Apply deduplication at **multiple granularities** — document level, paragraph level, and even sentence level for critical datasets. The **RefinedWeb** dataset demonstrated that aggressive deduplication significantly improves downstream model performance.