data filtering

**Data filtering** is the process of systematically removing **low-quality, irrelevant, harmful, or redundant** examples from a training dataset to improve model performance. In the era of large-scale web-scraped data, filtering has become one of the most impactful steps in the ML pipeline — the quality of training data often matters more than its quantity. **Common Filtering Criteria** - **Language Detection**: Remove text in unintended languages using tools like **fastText** language identification. - **Quality Scoring**: Use heuristics or classifiers to score text quality — remove content that is too short, too repetitive, mostly URLs/boilerplate, or poorly formatted. - **Toxicity Filtering**: Remove text containing hate speech, explicit content, or violence using classifiers like **Perspective API**. - **Deduplication**: Remove exact and near-duplicate content (see data deduplication). - **Perplexity Filtering**: Remove text with very high or very low perplexity as measured by a reference language model — extreme perplexity often indicates garbage or trivial content. - **Domain Filtering**: Select or exclude specific domains (e.g., keep educational content, remove social media spam). **Impact on Model Quality** - The **Llama** training pipeline applies extensive filtering to Common Crawl data, keeping only **~5%** of raw web text. - **Phi** models from Microsoft demonstrated that a small, highly filtered dataset can train models competitive with those trained on much larger, less filtered data. - **DCLM (DataComp for Language Models)** showed that better data filtering algorithms consistently lead to better model performance. **Best Practices** - **Multiple Passes**: Apply filtering in stages — cheap heuristic filters first, expensive classifier-based filters later. - **Sample Inspection**: Manually inspect random samples of filtered-in and filtered-out data to verify filter quality. - **Filter Logging**: Track why each example was removed to enable analysis and adjustment. Data filtering is increasingly recognized as one of the **highest-ROI** activities in ML development — clean data reduces training time, improves performance, and reduces harmful outputs.