regression analysis linear ridge lasso regression
# Regression Analysis: Linear, Ridge & Lasso Regression
## Introduction & Motivation
Linear regression models relationships via linear combination: ŷ = w^T x + b. Ridge (L2) and Lasso (L1) regularization prevent overfitting via weight penalties. Foundation for interpretable prediction; closed-form solutions available.
Motivation: Interpretable—coefficients show feature contribution. Fast—closed form or simple optimization. Baseline for regression tasks; strong performance on linear problems.
Applications: Pricing, forecasting, causal inference, feature selection (Lasso).
---
## Core Concepts & Theory
### Ordinary Least Squares
Minimize squared residuals: min Σ(y_i - ŷ_i)².
Solution: w = (X^T X)^{-1} X^T y (closed form).
### Ridge Regression
Add L2 penalty: min Σ(y_i - ŷ_i)² + λ Σ w_j².
Shrinks weights; reduces variance, increases bias.
### Lasso Regression
Add L1 penalty: min Σ(y_i - ŷ_i)² + λ Σ|w_j|.
Sparse weights; automatic feature selection.
---
## Mathematical Formulation
Ridge objective:
$$\min_w \|\mathbf{y} - \mathbf{X}\mathbf{w}\|^2 + \lambda \|\mathbf{w}\|_2^2$$
Closed form: w = (X^T X + λI)^{-1} X^T y.
Lasso objective:
$$\min_w \|\mathbf{y} - \mathbf{X}\mathbf{w}\|^2 + \lambda \|\mathbf{w}\|_1$$
No closed form; solved via coordinate descent, proximal methods.
---
## Advanced Theory & Extensions
### Elastic Net
Combines L1 and L2: λ₁||w||₁ + λ₂||w||₂². Balanced regularization.
### Standardization
Scale X to [0,1]; ensures comparable regularization across features.
---
## Computational Considerations
OLS: O(d³) via matrix inversion.
Ridge: O(d³) similarly.
Lasso: O(iterations × d) via iterative optimization.
---
## Practical Implementation Strategies
### Regularization Strength
λ via cross-validation; typical range 10^{-3} to 10^3.
### Feature Scaling
Critical; standardize to mean=0, std=1.
### Multicollinearity
Ridge handles via regularization; Lasso via selection.
---
## Benchmark Datasets & Evaluation
Boston Housing: Regression benchmark. Metric: RMSE, R².
California Housing: Larger dataset.
---
## Key Challenges & Limitations
### Assumptions
Assumes linear relationship. Non-linearity requires feature engineering.
### Extrapolation
Unreliable outside training range.
---
## Hyperparameter Tuning
α (regularization) via GridSearchCV; typical 0.001 to 100.
---
## Real-World Applications & Case Studies
Real Estate: Price prediction via linear features.
Economics: Coefficient interpretation for causal inference.
---
## Integration with Other Methods
Regression + Polynomial Features → Non-linear fitting.
---
## Summary & Key Takeaways
Linear regression with Ridge/Lasso regularization balances fit and simplicity, with closed-form/efficient solutions and interpretable coefficients.
Principles:
1. OLS minimizes squared error.
2. Ridge reduces variance via L2 penalty.
3. Lasso enables feature selection via L1 penalty.
4. Feature scaling critical for regularized models.
5. Cross-validation selects optimal regularization.
---
---
## Appendix: Practical Labs
### Lab 1: OLS vs Ridge
from sklearn.linear_model import LinearRegression, Ridge
from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split
from sklearn.metrics import r2_score
X, y = make_regression(n_samples=100, n_features=20, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
ols = LinearRegression().fit(X_train, y_train)
ridge = Ridge(alpha=1.0).fit(X_train, y_train)
ols_r2 = r2_score(y_test, ols.predict(X_test))
ridge_r2 = r2_score(y_test, ridge.predict(X_test))
print(f"OLS R²: {ols_r2:.4f}, Ridge R²: {ridge_r2:.4f}")
assert ols_r2 > 0 and ridge_r2 > 0, "Both should be positive"
print("✓ OLS vs Ridge working")
if __name__ == "__main__":
print("Lab 1: OLS vs Ridge - PASSED")### Lab 2: Lasso Feature Selection
from sklearn.linear_model import Lasso
from sklearn.datasets import make_regression
import numpy as np
X, y = make_regression(n_samples=100, n_features=20, random_state=42)
lasso = Lasso(alpha=0.1).fit(X, y)
non_zero = np.sum(lasso.coef_ != 0)
print(f"Non-zero coefficients: {non_zero}/{len(lasso.coef_)}")
assert non_zero > 0 and non_zero <= 20, "Should have selective features"
print("✓ Lasso feature selection working")
if __name__ == "__main__":
print("Lab 2: Lasso - PASSED")### Lab 3: Regularization Strength
from sklearn.linear_model import Ridge
from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split
from sklearn.metrics import r2_score
X, y = make_regression(n_samples=100, n_features=20, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
alphas = [0.001, 0.01, 0.1, 1, 10]
r2_scores = []
for alpha in alphas:
ridge = Ridge(alpha=alpha).fit(X_train, y_train)
r2 = r2_score(y_test, ridge.predict(X_test))
r2_scores.append(r2)
print(f"R² scores: {r2_scores}")
assert len(r2_scores) == 5, "Should have 5 scores"
print("✓ Regularization strength working")
if __name__ == "__main__":
print("Lab 3: Regularization - PASSED")### Lab 4: Elastic Net
from sklearn.linear_model import ElasticNet, Ridge, Lasso
from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split
from sklearn.metrics import r2_score
X, y = make_regression(n_samples=100, n_features=20, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
ridge = Ridge(alpha=1.0).fit(X_train, y_train)
lasso = Lasso(alpha=0.1).fit(X_train, y_train)
elastic = ElasticNet(alpha=1.0, l1_ratio=0.5).fit(X_train, y_train)
ridge_r2 = r2_score(y_test, ridge.predict(X_test))
lasso_r2 = r2_score(y_test, lasso.predict(X_test))
elastic_r2 = r2_score(y_test, elastic.predict(X_test))
print(f"Ridge: {ridge_r2:.4f}, Lasso: {lasso_r2:.4f}, ElasticNet: {elastic_r2:.4f}")
assert all(r > -1 for r in [ridge_r2, lasso_r2, elastic_r2]), "All should be valid"
print("✓ Elastic Net working")
if __name__ == "__main__":
print("Lab 4: Elastic Net - PASSED")