Data Cleaning
Data cleaning is the process of detecting and correcting errors, inconsistencies, and gaps in a dataset before it's used for analysis or modeling — fixing malformed entries, standardizing inconsistent formats, removing duplicates, and deciding how to handle missing values. It is routinely cited by practitioners as the single largest time cost in a data project, with the often-repeated "80% of a data scientist's time is spent cleaning data" figure circulating widely across industry surveys and commentary through the 2010s and 2020s as a rough, widely-shared estimate rather than a precisely measured statistic — but even as an approximation, it reflects a real and consistent pattern: raw data, whatever its source, arrives with problems that have to be resolved before any analysis built on it can be trusted.
The core categories of problems data cleaning addresses are consistent across nearly every dataset: missing values, duplicate records, inconsistent formatting, and outliers or invalid entries. Missing values show up as blank fields, nulls, or sentinel values (a date of "1900-01-01" used as a placeholder for "unknown," for instance) and require a decision — exclude the record, impute a replacement value, or flag it as missing rather than guessing. Duplicate records, often arising from repeated form submissions or merged data sources, inflate counts and distort aggregates unless identified and collapsed. Inconsistent formatting — "NY," "New York," and "ny" all referring to the same state — breaks grouping and joining operations that rely on exact string matches unless standardized first. Outliers and invalid entries (a negative age, an order date in the future) require judgment about whether they represent genuine data-entry errors to be fixed or removed, or legitimate extreme values that should be kept.
The most consequential discipline in data cleaning isn't the mechanics of fixing a value but documenting every cleaning decision made, because an undocumented cleaning step is functionally irreversible and unauditable — if an analyst silently drops rows with missing values, removes records flagged as outliers, or standardizes a category without recording the original values and the rule applied, nobody downstream can tell what was changed, why, or whether the decision was defensible. This matters beyond bookkeeping: a cleaning rule that happens to disproportionately remove one customer segment, one geography, or one time period introduces a systematic bias into every subsequent analysis, and that bias is invisible unless the cleaning logic itself is reviewable — which is why reproducible cleaning scripts (rather than manual, one-off spreadsheet edits) have become standard practice in professional data teams, since a script can be reviewed, rerun, and audited in a way a manual edit history usually can't be.
Validation rules — automated checks that flag records violating expected constraints (an age outside a plausible range, a required field left blank, a value outside its expected category set) — are the mechanism by which data cleaning scales beyond manual, case-by-case review, catching problems systematically rather than relying on an analyst noticing them by inspection. These rules typically run at the point data enters a pipeline rather than being applied retroactively, which is the more effective strategy generally: catching and rejecting (or flagging) a malformed record at ingestion is cheaper and less error-prone than discovering the same problem downstream, after it has already propagated through several transformation steps and potentially fed into an analysis or model that others are already relying on.
Deduplication specifically requires more judgment than it first appears to, because exact duplicate detection (two fully identical rows) misses the more common and harder case of fuzzy duplicates — the same customer entered as "Jon Smith" and "Jonathan Smith" with slightly different addresses, representing the same real-world entity despite no field matching exactly. Fuzzy matching techniques (string similarity measures like Levenshtein distance, or more advanced record-linkage methods) exist specifically for this problem, and getting the matching threshold wrong in either direction has a real cost: too strict, and genuine duplicates survive and inflate counts; too loose, and genuinely distinct records get incorrectly merged, silently combining two different customers' histories into one.
Read data cleaning as the unglamorous but disproportionately time-consuming work of making raw data trustworthy enough to analyze: the standard problem categories — missing values, duplicates, inconsistent formatting, invalid entries — recur across nearly every dataset regardless of domain, and the discipline that actually matters is not any individual fix but whether every cleaning decision is documented and reproducible enough that the gap between raw and clean data can be reviewed rather than taken on faith.