Handling Missing Values is a critical data preprocessing step in machine learning because most algorithms cannot process NaN/Null values — requiring practitioners to choose between deletion (removing incomplete rows or columns), imputation (filling missing values with statistical estimates like mean, median, or model-based predictions), or using algorithms that handle missingness natively (XGBoost, LightGBM), with the choice depending on whether data is missing randomly or systematically, the percentage of missingness, and the dataset size.
What Are Missing Values?
- Definition: Data entries that have no recorded value — appearing as NaN, NULL, None, empty string, or sentinel values (-1, 999, "N/A") in datasets, caused by sensor failures, survey non-responses, data pipeline errors, or information that genuinely doesn't apply.
- Why It Matters: Most ML algorithms (linear regression, SVM, neural networks) crash or produce nonsensical results when given NaN values. Even algorithms that handle NaN natively (tree-based models) benefit from thoughtful missing value treatment.
- Types of Missingness: Understanding WHY data is missing determines the correct handling strategy.
Types of Missing Data
| Type | Meaning | Example | Implication |
|---|---|---|---|
| MCAR (Missing Completely At Random) | Missingness is unrelated to any variable | A sensor randomly malfunctions | Safe to delete rows |
| MAR (Missing At Random) | Missingness depends on observed variables | High-income people skip income questions | Impute using related variables |
| MNAR (Missing Not At Random) | Missingness depends on the missing value itself | People with low credit scores hide their score | Hardest — "missingness" itself is a signal |
Handling Strategies
| Strategy | Method | Pros | Cons | When to Use |
|---|---|---|---|---|
| Drop rows | Delete rows with NaN | Simple, preserves feature space | Loses data, biased if not MCAR | <5% missing, large dataset |
| Drop columns | Delete features with many NaN | Reduces complexity | Loses potentially useful features | >50% missing in a column |
| Mean/Median | Fill with column average | Simple, fast | Ignores relationships between features | Numeric features, MCAR |
| Mode | Fill with most frequent value | Works for categorical | May amplify majority class | Categorical features |
| KNN Imputer | Fill using K nearest complete neighbors | Captures local patterns | Slow for large datasets | MAR, moderate missingness |
| Iterative Imputer | Model each feature as a function of others | Most accurate | Computationally expensive | MAR, complex relationships |
| Indicator Variable | Add is_missing_feature column (0/1) | Preserves missingness signal | Doubles feature count | MNAR (missingness is informative) |
Python Implementation
from sklearn.impute import SimpleImputer, KNNImputer
# Mean imputation
mean_imp = SimpleImputer(strategy='mean')
X_filled = mean_imp.fit_transform(X)
# KNN imputation (uses neighbors)
knn_imp = KNNImputer(n_neighbors=5)
X_filled = knn_imp.fit_transform(X)
Common Mistakes
| Mistake | Problem | Fix |
|---|---|---|
| Imputing before train/test split | Test data leaks into imputer statistics | Fit imputer on train, transform both |
| Using mean for skewed data | Mean is pulled by outliers (salary: $50K mean but $35K median) | Use median for skewed distributions |
| Ignoring MNAR patterns | Missing values carry information you discard | Add indicator columns |
| One strategy for all columns | Different features need different approaches | Column-specific imputation strategies |
Handling Missing Values is the essential first step of data preprocessing — requiring practitioners to diagnose why data is missing, choose appropriate strategies based on missingness type and severity, and implement imputation correctly within cross-validation to prevent data leakage, because the model can only be as good as the data it receives.
Explore 500+ Semiconductor & AI Topics
From EUV lithography to CUDA optimization — search the full knowledge base or chat with our AI assistant.