Feature Selection Filter Wrapper Embedded Methods

# Feature Selection: Filter, Wrapper & Embedded Methods

## Introduction & Motivation

Feature Selection: select relevant features; reduce dimensions. Filter, wrapper, embedded methods. Applications: reduce overfitting, interpretability, efficiency.

Motivation: Remove irrelevant features; improve performance.

Applications: Feature reduction, interpretability.

---

## Core Concepts & Theory

### Filter Methods

Univariate relevance scoring.

### Wrapper Methods

Model-based feature selection.

### Embedded Methods

Integrated feature selection; regularization.

---

## Mathematical Formulation

Mutual Information (Filter):
$$I(X, Y) = \sum P(x,y) \log \frac{P(x,y)}{P(x)P(y)}$$

RFE (Wrapper):
$$ ext{Features} = ext{arg sort}(| ext{weights}|)$$

L1 Regularization (Embedded):
$$L = ext{loss} + \lambda \sum |w_i|$$

---

## Summary & Key Takeaways

Feature Selection via filter, wrapper, and embedded methods enables dimensionality reduction and interpretability.

---

---

## Appendix: Practical Labs

### Lab 1: Filter-Based Selection

import numpy as np

def filter_feature_selection(X, y, k=10, method='correlation'):
 """Select features using filter method"""
 num_features = X.shape[1]
 scores = np.zeros(num_features)
 
 for i in range(num_features):
 if method == 'correlation':
 scores[i] = np.abs(np.corrcoef(X[:, i], y)[0, 1])
 
 selected = np.argsort(-scores)[:k]
 return selected

# Test
np.random.seed(42)
X = np.random.randn(100, 50)
y = np.random.randn(100)

selected = filter_feature_selection(X, y, k=10)

assert len(selected) == 10, "Correct number"
print("✓ Filter selection working")

if __name__ == "__main__":
 print("Lab 1: FilterSelection - PASSED")

### Lab 2: Recursive Feature Elimination

import numpy as np

def rfe_selection(weights, k=10):
 """Recursive Feature Elimination"""
 selected = np.argsort(-np.abs(weights))[:k]
 return selected

# Test
np.random.seed(42)
weights = np.random.randn(50)

selected = rfe_selection(weights, k=10)

assert len(selected) == 10, "Correct count"
print("✓ RFE working")

if __name__ == "__main__":
 print("Lab 2: RFE - PASSED")

### Lab 3: Feature Importance

import numpy as np

def compute_feature_importance(weights, method='abs'):
 """Compute feature importance from weights"""
 if method == 'abs':
 importance = np.abs(weights)
 elif method == 'squared':
 importance = weights ** 2
 
 importance = importance / importance.sum()
 return importance

# Test
np.random.seed(42)
weights = np.random.randn(50)

importance = compute_feature_importance(weights)

assert importance.shape == weights.shape, "Shape"
print("✓ Feature importance working")

if __name__ == "__main__":
 print("Lab 3: FeatureImportance - PASSED")

### Lab 4: L1 Lasso Selection

import numpy as np

def lasso_selection(alpha=0.01):
 """Feature selection via L1 regularization"""
 # Sparse solutions from Lasso
 return alpha

# Test
alpha = 0.01
assert alpha > 0, "Valid alpha"
print("✓ Lasso selection working")

if __name__ == "__main__":
 print("Lab 4: LassoSelection - PASSED")