Data Validation
Data validation is the process of checking that data conforms to expected rules — correct type, valid range, required fields present, referential consistency — before it's accepted into a system or trusted in an analysis. Where data cleaning fixes problems after they're found, validation is the earlier, preventive discipline: a well-designed validation layer catches a malformed record at the moment it tries to enter a system, rejecting or flagging it before it has a chance to propagate downstream and corrupt whatever depends on it.
Validation rules fall into a small number of recurring categories that cover the large majority of real-world checks. Type validation confirms a value matches its expected data type — a "quantity" field must be an integer, not a string. Range validation confirms a value falls within a plausible bound — an age between 0 and 130, a percentage between 0 and 100. Format validation confirms a value matches an expected pattern — an email address containing an @ symbol, a phone number matching a specific digit structure, typically enforced with regular expressions. Referential validation confirms a value correctly references another record that actually exists — an order's customer_id must correspond to a real row in the customers table, the database-level version of this being a foreign key constraint. Uniqueness validation confirms a value that should be unique actually is — no two users sharing the same email address.
Where validation runs in the pipeline — at the point of entry versus after the fact — makes a significant practical difference, and the field's general preference is to validate as early as possible, a principle sometimes called "shift-left" validation borrowed from software testing terminology. A relational database's schema constraints (NOT NULL, type enforcement, foreign keys, unique indexes) validate at write time, rejecting a malformed insert immediately and making it structurally impossible for certain kinds of bad data to ever be stored. Application-level validation, running before data even reaches the database, catches errors even earlier, closer to the point a user or system actually generated the bad value, which generally makes the error easier to trace back to its root cause. Validating only after the fact — a nightly job that scans for malformed records already stored — catches problems later, after they may have already been read, aggregated, or acted upon by something else in the system, which is why reactive, after-the-fact validation is treated as a weaker safety net rather than a substitute for validation at the point of entry.
A dedicated data validation and testing layer has become standard in modern data pipelines, with frameworks like Great Expectations and dbt's built-in testing functionality letting teams define validation rules declaratively and run them automatically as part of a pipeline's execution, rather than relying on ad hoc, manually-written checks scattered across scripts. This formalization matters because validation rules, like code, need to be versioned, reviewed, and tested themselves — an informally applied validation check that exists only in one analyst's head isn't actually protecting anyone else working with the same data, which is the core argument for treating validation as pipeline infrastructure rather than a personal habit.
Statistical validation, distinct from the structural checks above, looks for anomalies a type or range check wouldn't catch — a sudden, unexplained shift in a column's distribution, an unusual spike in the proportion of a specific category, a correlation between two fields that historically didn't exist suddenly appearing. This category of check, often implemented as automated data quality monitoring that compares each new batch of data against the statistical profile of historical batches, exists because a record can pass every structural validation rule (correct type, valid range, properly formatted) while still being substantively wrong in a way only visible at the aggregate, statistical level — a sensor that starts reporting consistently biased readings after a recalibration, for instance, where every individual reading still looks like a plausible number.
Read data validation as the preventive counterpart to data cleaning: cleaning fixes problems that have already entered a dataset, while validation's entire purpose is to stop them from entering in the first place, and the discipline's clearest organizing principle — validate as early and as structurally as possible, and back structural checks up with statistical monitoring for the anomalies structure alone can't catch — is what separates a data system that can be trusted by default from one that requires manual inspection before every use.