Feature Selection is the process of identifying and keeping only the most informative variables for a machine learning model while discarding noisy, redundant, or irrelevant features — improving model accuracy (less noise = better signal), reducing overfitting (fewer parameters = better generalization), speeding up training and inference (fewer features = less computation), and improving interpretability (fewer features = easier to explain), making it a critical preprocessing step that sits between feature engineering and model training.
What Is Feature Selection?
- Definition: The systematic identification of the subset of input features that contribute most to prediction accuracy — using statistical tests, model-based importance scores, or iterative search to separate signal from noise.
- Why Not Keep Everything?: More features aren't always better. Irrelevant features add noise that models can overfit to. Redundant features (height_cm and height_inches) waste computation without adding information. The "curse of dimensionality" means that as features increase, the data becomes increasingly sparse in high-dimensional space.
- Feature Selection vs. Feature Extraction: Selection keeps a subset of original features. Extraction (PCA, autoencoders) creates new features that are combinations of originals. Selection preserves interpretability; extraction may not.
Three Categories of Methods
| Category | Approach | Speed | Quality | Example |
|---|---|---|---|---|
| Filter Methods | Rank features by statistical score, independent of model | Very fast | Good | Correlation, Chi-Square, Mutual Information |
| Wrapper Methods | Train model with different feature subsets, select the best | Slow | Best | Recursive Feature Elimination (RFE), Forward Selection |
| Embedded Methods | Model selects features during training | Moderate | Very good | L1 (Lasso), Tree Feature Importance, ElasticNet |
Filter Methods (Model-Independent)
| Method | Feature Type | What It Measures |
|---|---|---|
| Pearson Correlation | Continuous vs Continuous | Linear relationship strength |
| Chi-Square (χ²) | Categorical vs Categorical | Statistical independence |
| Mutual Information | Any | Non-linear dependency between feature and target |
| Variance Threshold | Any | Remove features with near-zero variance |
| ANOVA F-test | Continuous vs Categorical | Difference in means across classes |
Wrapper Methods (Model-Dependent)
| Method | Process | Trade-off |
|---|---|---|
| Forward Selection | Start empty, add best feature one at a time | Greedy, may miss feature interactions |
| Backward Elimination | Start with all, remove worst feature one at a time | Expensive for many features |
| RFE (Recursive Feature Elimination) | Train model, remove least important, repeat | Good balance, sklearn built-in |
Embedded Methods (During Training)
| Method | How It Selects | Best For |
|---|---|---|
| L1 Regularization (Lasso) | Drives weak feature coefficients to exactly zero | Linear/logistic regression |
| Tree Feature Importance | Features used in early splits are most important | Random Forest, XGBoost |
| ElasticNet (L1 + L2) | Combines L1 sparsity with L2 grouping | Correlated features |
Feature Selection is the essential preprocessing step that ensures models learn from signal rather than noise — using statistical tests, model-based importance, or iterative search to identify the features that actually matter, improving accuracy, reducing overfitting, speeding up training, and producing models that are easier to interpret and deploy.
Explore 500+ Semiconductor & AI Topics
From EUV lithography to CUDA optimization — search the full knowledge base or chat with our AI assistant.