Feature Engineering Selection Dimensionality Reduction Importance
# Feature Engineering & Selection: Dimensionality Reduction & Importance
## Introduction & Motivation
Feature engineering: create informative features from raw data. Selection: identify high-value features; reduce dimensionality. Importance-based: tree-based feature importance; correlation analysis. Statistical methods: mutual information, correlation coefficients. Applications: improve model performance, interpretability, efficiency.
Motivation: Raw features often noisy, redundant. Engineering + selection → cleaner signal, faster training.
Applications: Tabular data, mixed-type features, interpretability.
---
## Core Concepts & Theory
### Feature Importance
Tree-based: sum of information gains. Permutation: drop feature, measure performance drop.
### Correlation Analysis
Pearson correlation: linear dependency. Spearman: rank-based; robust to outliers.
### Mutual Information
Information-theoretic dependency; captures non-linear relationships.
---
## Mathematical Formulation
Pearson Correlation:
$$r = \frac{\sum (x_i - \bar{x})(y_i - \bar{y})}{\sqrt{\sum (x_i - \bar{x})^2} \sqrt{\sum (y_i - \bar{y})^2}}$$
Mutual Information:
$$I(X; Y) = \sum_x \sum_y p(x, y) \log \frac{p(x, y)}{p(x)p(y)}$$
Feature Importance (tree):
$$ ext{Importance}_j = \sum_{ ext{splits on j}} ext{information gain}$$
---
## Advanced Theory & Extensions
### SHAP Values
Game-theoretic feature importance; explains predictions locally.
### Permutation Importance
Model-agnostic; drop feature, measure performance loss.
### PCA/SVD
Linear dimensionality reduction; orthogonal components.
---
## Computational Considerations
Correlation: O(N·D²) for N samples, D features.
Tree Importance: O(1) after training; negligible.
Permutation: O(M·N) for M features; expensive.
---
## Practical Implementation Strategies
### Selection Method
Start with tree-based importance; complement with correlation.
### Threshold
Keep top-k or importance > threshold; data-dependent.
### Validation
Validate on holdout; ensure improvement is real.
---
## Benchmark Datasets & Evaluation
MNIST: Raw pixels sufficient; limited feature engineering.
Tabular (Kaggle): Feature engineering critical; ~50% of effort.
Text: TF-IDF, embeddings; feature selection via frequency.
---
## Key Challenges & Limitations
### Multicollinearity
Correlated features confound importance; select one per pair.
### Non-linear Dependencies
Correlation misses; mutual information better.
### Feature Interactions
Single-feature importance misses interactions.
---
## Hyperparameter Tuning
Selection threshold: Empirical on holdout; ~top 50-80%.
Importance method: Tree > Permutation > Correlation.
Validation: Cross-validation to avoid overfitting.
---
## Real-World Applications & Case Studies
Kaggle Competitions: Feature engineering dominates; custom engineered features.
Credit Scoring: Interpretability critical; importance-based selection.
Medical Diagnosis: Select interpretable features; domain expert validation.
---
## Integration with Other Methods
Feature Selection + Regularization → explicit+implicit dimension reduction.
Feature Engineering + Embeddings → hybrid representations.
---
## Summary & Key Takeaways
Feature engineering and selection via importance measures, correlation analysis, and statistical tests improve model performance and interpretability.
Principles:
1. Tree importance: efficient, interpretable baseline.
2. Permutation: model-agnostic, accurate.
3. Correlation: simple; misses non-linear.
4. Mutual information: captures dependencies.
5. Validate on holdout; empirical threshold.
---
---
## Appendix: Practical Labs
### Lab 1: Correlation-based Selection
import numpy as np
def select_features_correlation(X, y, threshold=0.1):
"""Select features correlated with target"""
correlations = np.corrcoef(X.T, y)[:-1, -1]
selected = np.abs(correlations) > threshold
return np.where(selected)[0], correlations
# Test
np.random.seed(42)
X = np.random.randn(100, 20)
y = X[:, 0] + X[:, 1] + 0.1 * np.random.randn(100)
selected, corrs = select_features_correlation(X, y, threshold=0.1)
assert len(selected) > 0, "Should select some features"
assert all(np.abs(corrs[selected]) > 0.1), "Selected features above threshold"
print("✓ Correlation selection working")
if __name__ == "__main__":
print("Lab 1: Correlation - PASSED")### Lab 2: Mutual Information
import numpy as np
from sklearn.feature_selection import mutual_info_classif
def mutual_information_ranking(X, y):
"""Rank features by mutual information with target"""
mi = mutual_info_classif(X, y, random_state=42)
indices = np.argsort(-mi)
return indices, mi
# Test
np.random.seed(42)
X = np.random.randn(100, 20)
y = (X[:, 0] + X[:, 1] > 0).astype(int)
indices, mi = mutual_information_ranking(X, y)
assert len(indices) == 20, "Should rank all features"
assert mi[indices[0]] >= mi[indices[1]], "Should be sorted descending"
assert all(mi >= 0), "MI should be non-negative"
print("✓ Mutual information working")
if __name__ == "__main__":
print("Lab 2: MI - PASSED")### Lab 3: Permutation Importance
import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score
def permutation_importance(model, X, y, n_repeats=10):
"""Compute permutation importance"""
baseline_score = accuracy_score(y, model.predict(X))
importances = []
for feature_idx in range(X.shape[1]):
scores = []
for _ in range(n_repeats):
X_perm = X.copy()
np.random.shuffle(X_perm[:, feature_idx])
score = accuracy_score(y, model.predict(X_perm))
scores.append(baseline_score - score)
importances.append(np.mean(scores))
return np.array(importances)
# Test
np.random.seed(42)
X = np.random.randn(100, 20)
y = (X[:, 0] > 0).astype(int)
model = RandomForestClassifier(n_estimators=10, random_state=42)
model.fit(X, y)
importance = permutation_importance(model, X, y, n_repeats=5)
assert len(importance) == 20, "Should have 20 importances"
assert all(i >= 0 for i in importance), "Importances non-negative"
print("✓ Permutation importance working")
if __name__ == "__main__":
print("Lab 3: Permutation - PASSED")### Lab 4: Feature Selection Pipeline
import numpy as np
from sklearn.ensemble import RandomForestClassifier
def select_features_importance(X, y, n_features_to_keep=10):
"""Select top features by tree importance"""
model = RandomForestClassifier(n_estimators=50, random_state=42)
model.fit(X, y)
importances = model.feature_importances_
indices = np.argsort(-importances)[:n_features_to_keep]
return indices, importances
# Test
np.random.seed(42)
X = np.random.randn(100, 30)
y = (X[:, 0] + X[:, 1] > 0).astype(int)
selected, importances = select_features_importance(X, y, n_features_to_keep=10)
assert len(selected) == 10, "Should select 10 features"
assert all(importances[selected] > 0), "Importances positive"
print("✓ Feature selection pipeline working")
if __name__ == "__main__":
print("Lab 4: Pipeline - PASSED")