Feature Selection Dimensionality Reduction Filtering Wrapper Methods

# Feature Selection & Dimensionality Reduction: Filtering & Wrapper Methods

## Introduction & Motivation

Feature selection removes irrelevant features, improving interpretability, training speed, and generalization. Filter methods (statistical tests), wrapper methods (model-based), embedded methods (algorithm-integrated). Contrast with dimensionality reduction (PCA) which creates new features.

Motivation: High-dimensional data: curse of dimensionality, overfitting, slow training. Feature selection keeps interpretable features. Reduces noise, improves model stability.

Applications: Gene expression analysis, text classification, medical diagnosis, real-time prediction systems.

---

## Core Concepts & Theory

### Filter Methods

Statistical tests: correlation, chi-squared, mutual information. Fast, model-agnostic.

### Wrapper Methods

Recursive Feature Elimination (RFE): train model, remove low-importance features, repeat.

### Embedded Methods

Feature selection during training: L1 regularization (Lasso), tree feature importance.

---

## Summary & Key Takeaways

Feature selection improves interpretability and efficiency via removal of irrelevant dimensions.

Principles:
1. Filter fast; wrapper accurate.
2. Embedded integrate selection into training.
3. Reduces overfitting and computational cost.
4. Maintains interpretability vs. PCA.
5. Domain knowledge guides selection.

---

---

## Appendix: Practical Labs

### Lab 1: Correlation-Based Selection

from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.datasets import load_iris
import pandas as pd

iris = load_iris()
X, y = iris.data, iris.target

selector = SelectKBest(score_func=f_classif, k=2)
X_selected = selector.fit_transform(X, y)

print(f"Original shape: {X.shape}, Selected: {X_selected.shape}")
assert X_selected.shape[1] == 2, "Should select 2 features"
print("✓ Correlation-based selection working")

if __name__ == "__main__":
 print("Lab 1: Correlation - PASSED")

### Lab 2: RFE

from sklearn.feature_selection import RFE
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import make_classification

X, y = make_classification(n_samples=100, n_features=20, random_state=42)

model = LogisticRegression(max_iter=200)
rfe = RFE(model, n_features_to_select=5)
X_selected = rfe.fit_transform(X, y)

print(f"Selected {X_selected.shape[1]} features from {X.shape[1]}")
assert X_selected.shape[1] == 5, "Should select 5 features"
print("✓ RFE working")

if __name__ == "__main__":
 print("Lab 2: RFE - PASSED")

### Lab 3: Tree-Based Selection

from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_iris
import numpy as np

iris = load_iris()
X, y = iris.data, iris.target

rf = RandomForestClassifier(random_state=42)
rf.fit(X, y)

importances = rf.feature_importances_
top_k = 2
top_indices = np.argsort(importances)[-top_k:]

print(f"Top {top_k} features: {top_indices}")
assert len(top_indices) == top_k, "Should select top features"
print("✓ Tree-based selection working")

if __name__ == "__main__":
 print("Lab 3: Tree-Based - PASSED")

### Lab 4: Mutual Information

from sklearn.feature_selection import mutual_info_classif
from sklearn.datasets import load_iris

iris = load_iris()
X, y = iris.data, iris.target

mi_scores = mutual_info_classif(X, y, random_state=42)

print(f"MI scores: {mi_scores}")
assert len(mi_scores) == X.shape[1], "Should score all features"
print("✓ Mutual information working")

if __name__ == "__main__":
 print("Lab 4: MI - PASSED")