Logistic Regression Probabilistic Classification Log-Odds
# Logistic Regression: Probabilistic Classification & Log-Odds
## Introduction & Motivation
Logistic regression extends linear regression to classification via logistic (sigmoid) function, mapping predicted values to [0,1] probability range. Despite name, solves classification not regression. Probability output enables threshold-tuning, cost-sensitive classification, and confidence estimates. Foundational supervised learning algorithm.
Motivation: Linear regression outputs unbounded values unsuitable for classification. Sigmoid transformation squashes to valid probability. Interpretable coefficients guide feature importance. Standard baseline for binary/multi-class problems.
Applications: Medical diagnosis, credit approval, spam detection, customer churn. Industry standard for interpretable classification.
---
## Core Concepts & Theory
### Sigmoid Function
Map linear predictor to probability:
$$P(y=1|\mathbf{x}) = \sigma(\mathbf{w}^T\mathbf{x} + b) = \frac{1}{1 + e^{-(\mathbf{w}^T\mathbf{x} + b)}}$$
Range (0,1); interpretation as posterior probability.
### Log-Odds
Model log-odds (logit) as linear:
$$\log\frac{P(y=1)}{P(y=0)} = \mathbf{w}^T\mathbf{x} + b$$
Linearity in log-odds justifies linear model.
---
## Mathematical Formulation
Binary cross-entropy loss:
$$\mathcal{L} = -\frac{1}{n}\sum_{i=1}^{n}[y_i \log(\hat{p}_i) + (1-y_i)\log(1-\hat{p}_i)]$$
Gradient w.r.t. weights:
$$\frac{\partial \mathcal{L}}{\partial \mathbf{w}} = \frac{1}{n}\sum_{i=1}^{n}(\hat{p}_i - y_i)\mathbf{x}_i$$
Solved via gradient descent or closed-form (for small n).
---
## Advanced Theory & Extensions
### Regularization
L2 (Ridge): \mathcal{L} + \frac{\lambda}{2}\|\mathbf{w}\|^2 reduces overfitting.
L1 (Lasso): \mathcal{L} + \lambda\|\mathbf{w}\|_1 enables feature selection.
### Multi-class Extension
Softmax for K classes:
$$P(y=k|\mathbf{x}) = \frac{e^{\mathbf{w}_k^T\mathbf{x}}}{\sum_j e^{\mathbf{w}_j^T\mathbf{x}}}$$
---
## Computational Considerations
Training: O(n imes d imes iterations) via gradient descent.
Inference: O(d) per sample.
Memory: O(d) for weights.
---
## Practical Implementation Strategies
### Feature Scaling
Critical: Standardize features to [0,1]. Affects convergence speed and coefficient magnitude.
### Handling Class Imbalance
Adjust class weights or threshold. Weight minority class higher.
### Convergence
Check gradient magnitude; typical tolerance 1e-4.
---
## Benchmark Datasets & Evaluation
Binary: Breast Cancer, Default prediction. Metric: AUC, F1.
Multi-class: Iris, MNIST digits. Metric: Accuracy, macro F1.
---
## Key Challenges & Limitations
### Linear Decision Boundary
Non-linearly separable problems require feature engineering or different model.
### Multicollinearity
Correlated features inflate weights. Use regularization or feature selection.
---
## Hyperparameter Tuning
Regularization strength C \in {0.001, 0.01, 0.1, 1, 10, 100\}. Solver {lbfgs, liblinear, saga\}.
---
## Real-World Applications & Case Studies
Medical: Logistic regression baseline for disease prediction; interpretable for clinicians.
Finance: Credit scoring, default prediction; regulatory compliance via explainability.
---
## Integration with Other Methods
Logistic Regression + Feature Selection → Improved interpretability.
Logistic Regression + Calibration → Probability estimates aligned with reality.
---
## Future Research Directions
Bayesian logistic regression with uncertainty; fairness constraints; sparse inference.
---
## Summary & Key Takeaways
Logistic regression is foundational probabilistic classifier via sigmoid function mapping linear predictor to probability.
Principles:
1. Sigmoid transforms unbounded predictions to [0,1].
2. Log-odds linearity justifies linear model.
3. Cross-entropy loss optimized via gradient descent.
4. Regularization prevents overfitting.
5. Feature scaling critical for convergence.
---
---
## Appendix: Practical Labs
### Lab 1: Logistic Regression from Scratch
import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
class LogisticRegression:
def __init__(self, learning_rate=0.01, n_iterations=100):
self.lr = learning_rate
self.n_iter = n_iterations
self.w = None
self.b = None
def sigmoid(self, z):
return 1 / (1 + np.exp(-np.clip(z, -500, 500)))
def fit(self, X, y):
n_samples, n_features = X.shape
self.w = np.zeros(n_features)
self.b = 0
for _ in range(self.n_iter):
z = np.dot(X, self.w) + self.b
p = self.sigmoid(z)
dw = np.dot(X.T, (p - y)) / n_samples
db = np.mean(p - y)
self.w -= self.lr * dw
self.b -= self.lr * db
return self
def predict_proba(self, X):
z = np.dot(X, self.w) + self.b
return self.sigmoid(z)
def predict(self, X, threshold=0.5):
return (self.predict_proba(X) >= threshold).astype(int)
X, y = make_classification(n_samples=100, n_features=10, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
lr = LogisticRegression(learning_rate=0.01, n_iterations=100)
lr.fit(X_train, y_train)
y_pred = lr.predict(X_test)
accuracy = np.mean(y_pred == y_test)
print(f"Accuracy: {accuracy:.4f}")
assert accuracy > 0.5, "Should exceed random"
assert lr.w is not None, "Should be trained"
print("✓ Logistic Regression from scratch working")
if __name__ == "__main__":
print("Lab 1: Logistic Regression - PASSED")### Lab 2: Binary vs Multi-class
import numpy as np
from sklearn.datasets import make_classification, load_iris
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
# Binary
X_bin, y_bin = make_classification(n_samples=150, n_features=15, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X_bin, y_bin, test_size=0.3, random_state=42)
lr_bin = LogisticRegression(max_iter=200)
lr_bin.fit(X_train, y_train)
acc_bin = accuracy_score(y_test, lr_bin.predict(X_test))
# Multi-class (Iris)
iris = load_iris()
X_multi, y_multi = iris.data, iris.target
X_train, X_test, y_train, y_test = train_test_split(X_multi, y_multi, test_size=0.3, random_state=42)
lr_multi = LogisticRegression(max_iter=200)
lr_multi.fit(X_train, y_train)
acc_multi = accuracy_score(y_test, lr_multi.predict(X_test))
print(f"Binary accuracy: {acc_bin:.4f}, Multi-class accuracy: {acc_multi:.4f}")
assert acc_bin > 0.6 and acc_multi > 0.6, "Both should exceed baseline"
print("✓ Binary vs Multi-class working")
if __name__ == "__main__":
print("Lab 2: Binary vs Multi-class - PASSED")### Lab 3: Regularization Effect
import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
X, y = make_classification(n_samples=100, n_features=20, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
C_values = [0.001, 0.01, 0.1, 1, 10]
accuracies = []
for C in C_values:
lr = LogisticRegression(C=C, max_iter=200)
lr.fit(X_train, y_train)
acc = accuracy_score(y_test, lr.predict(X_test))
accuracies.append(acc)
print(f"C={C}: Accuracy={acc:.4f}")
assert len(accuracies) == 5, "Should have 5 C values"
assert all(0.4 < acc < 1.0 for acc in accuracies), "Accuracies should be valid"
print("✓ Regularization effect working")
if __name__ == "__main__":
print("Lab 3: Regularization - PASSED")### Lab 4: Probability Calibration
import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
X, y = make_classification(n_samples=150, n_features=12, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
lr = LogisticRegression(max_iter=200)
lr.fit(X_train, y_train)
probs = lr.predict_proba(X_test)[:, 1]
print(f"Probability range: [{probs.min():.4f}, {probs.max():.4f}]")
print(f"Mean probability: {probs.mean():.4f}")
assert 0 <= probs.min() and probs.max() <= 1, "Probabilities should be in [0,1]"
assert probs.mean() > 0.2 and probs.mean() < 0.8, "Mean should be reasonable"
print("✓ Probability calibration working")
if __name__ == "__main__":
print("Lab 4: Probability Calibration - PASSED")