near-duplicate detection

**Near-duplicate detection** is the **identification of highly similar but not identical text samples in large datasets** - it is essential for controlling hidden redundancy in web-scale corpora. **What Is Near-duplicate detection?** - **Definition**: Targets paraphrased, lightly edited, or templated content repetitions. - **Difficulty**: Near duplicates require approximate similarity methods beyond exact hashing. - **Impact**: Unchecked near duplicates can bias training toward narrow repeated patterns. - **Methods**: Typically uses shingling, MinHash signatures, and locality-sensitive retrieval. **Why Near-duplicate detection Matters** - **Data Efficiency**: Reducing near duplicates increases information diversity per token. - **Memorization Control**: Lowers repeated exposure to identical factual or sensitive content. - **Benchmark Integrity**: Helps prevent accidental overlap between train and evaluation material. - **Quality**: Improves representation of broader linguistic and topical variation. - **Pipeline Accuracy**: Near-duplicate handling is crucial in multi-source data ingestion. **How It Is Used in Practice** - **Similarity Tuning**: Set domain-specific thresholds for what counts as near duplicate. - **Incremental Indexing**: Run detection continuously as new corpora are ingested. - **Validation**: Check false-positive and false-negative rates with human-reviewed samples. Near-duplicate detection is **a key deduplication stage for high-quality large-language-model corpora** - near-duplicate detection should balance aggressive redundancy removal with preservation of legitimate variation.