feature selection
**Feature Selection** is the **process of identifying and keeping only the most informative variables for a machine learning model while discarding noisy, redundant, or irrelevant features** — improving model accuracy (less noise = better signal), reducing overfitting (fewer parameters = better generalization), speeding up training and inference (fewer features = less computation), and improving interpretability (fewer features = easier to explain), making it a critical preprocessing step that sits between feature engineering and model training.
**What Is Feature Selection?**
- **Definition**: The systematic identification of the subset of input features that contribute most to prediction accuracy — using statistical tests, model-based importance scores, or iterative search to separate signal from noise.
- **Why Not Keep Everything?**: More features aren't always better. Irrelevant features add noise that models can overfit to. Redundant features (height_cm and height_inches) waste computation without adding information. The "curse of dimensionality" means that as features increase, the data becomes increasingly sparse in high-dimensional space.
- **Feature Selection vs. Feature Extraction**: Selection keeps a subset of original features. Extraction (PCA, autoencoders) creates new features that are combinations of originals. Selection preserves interpretability; extraction may not.
**Three Categories of Methods**
| Category | Approach | Speed | Quality | Example |
|----------|---------|-------|---------|---------|
| **Filter Methods** | Rank features by statistical score, independent of model | Very fast | Good | Correlation, Chi-Square, Mutual Information |
| **Wrapper Methods** | Train model with different feature subsets, select the best | Slow | Best | Recursive Feature Elimination (RFE), Forward Selection |
| **Embedded Methods** | Model selects features during training | Moderate | Very good | L1 (Lasso), Tree Feature Importance, ElasticNet |
**Filter Methods (Model-Independent)**
| Method | Feature Type | What It Measures |
|--------|-------------|-----------------|
| **Pearson Correlation** | Continuous vs Continuous | Linear relationship strength |
| **Chi-Square (χ²)** | Categorical vs Categorical | Statistical independence |
| **Mutual Information** | Any | Non-linear dependency between feature and target |
| **Variance Threshold** | Any | Remove features with near-zero variance |
| **ANOVA F-test** | Continuous vs Categorical | Difference in means across classes |
**Wrapper Methods (Model-Dependent)**
| Method | Process | Trade-off |
|--------|---------|-----------|
| **Forward Selection** | Start empty, add best feature one at a time | Greedy, may miss feature interactions |
| **Backward Elimination** | Start with all, remove worst feature one at a time | Expensive for many features |
| **RFE (Recursive Feature Elimination)** | Train model, remove least important, repeat | Good balance, sklearn built-in |
**Embedded Methods (During Training)**
| Method | How It Selects | Best For |
|--------|---------------|----------|
| **L1 Regularization (Lasso)** | Drives weak feature coefficients to exactly zero | Linear/logistic regression |
| **Tree Feature Importance** | Features used in early splits are most important | Random Forest, XGBoost |
| **ElasticNet (L1 + L2)** | Combines L1 sparsity with L2 grouping | Correlated features |
**Feature Selection is the essential preprocessing step that ensures models learn from signal rather than noise** — using statistical tests, model-based importance, or iterative search to identify the features that actually matter, improving accuracy, reducing overfitting, speeding up training, and producing models that are easier to interpret and deploy.