Quantile Regression Robust Estimation Conditional Quantiles

# Quantile Regression: Robust Estimation & Conditional Quantiles

## Introduction & Motivation

Quantile regression estimates conditional quantiles instead of mean. τ-th quantile: minimize absolute deviations weighted by τ. Robust to outliers; asymmetric loss captures heteroskedasticity. Applications: conditional value-at-risk (finance), income distribution (economics), medical prognosis (varying patient outcomes).

Motivation: OLS assumes homoskedasticity; breaks with heteroskedasticity. Quantile regression reveals full conditional distribution, not just mean.

Applications: Risk assessment, pricing, medical diagnosis, economic inequality.

---

## Core Concepts & Theory

### Quantile Definition

τ-th quantile: value where τ proportion of data below. Linear quantile: β(τ) = argmin Σ ρ_τ(y_i - x_i^T β)
where ρ_τ(u) = u(τ - I(u<0)).

### Asymmetric Loss

Loss weighs underprediction τ times overprediction; reflects quantile level.

---

## Mathematical Formulation

Quantile loss (check function):
$$ ho_ au(u) = u( au - \mathbb{I}_{u < 0})$$

Quantile regression objective:
$$\min_\beta \sum_{i=1}^n ho_ au(y_i - \mathbf{x}_i^T \beta)$$

Conditional quantile:
$$Q_y( au | \mathbf{x}) = \mathbf{x}^T \beta^*( au)$$

---

## Advanced Theory & Extensions

### Quantile Crossing

Quantile curves may cross; solutions: rearrangement operators.

### Conditional Autoregressive Expectiles

CARE models: dynamic quantile estimation.

---

## Computational Considerations

Linear: O(n × d) via linear programming or gradient descent.

Non-parametric: O(n²) kernel bandwidth selection.

---

## Practical Implementation Strategies

### Cross-Validation

Select τ and regularization via CV on quantile loss.

### Multiple Quantiles

Estimate portfolio of quantiles (0.1, 0.5, 0.9) simultaneously.

---

## Benchmark Datasets & Evaluation

Boston Housing: Heteroskedastic regression.

Income Quantiles: Earnings prediction at multiple levels.

Metrics: Quantile loss, coverage (% observations below predicted), interval width.

---

## Key Challenges & Limitations

### Non-Crossing Quantiles

Quantile curves cross; violated monotonicity. Use isotonic regression.

### High Dimension

Variable selection in quantile regression challenging.

---

## Hyperparameter Tuning

τ (quantile level): 0.1-0.9; lower = lower tail.

Regularization λ: Ridge/Lasso via CV.

---

## Real-World Applications & Case Studies

Finance: Value-at-Risk (VaR) via quantile regression.

Healthcare: Personalized medicine; conditional treatment effects by patient severity.

Econometrics: Wage distribution; inequality measurement.

---

## Integration with Other Methods

Quantile + Ensemble → Random forests for quantiles.

---

## Summary & Key Takeaways

Quantile regression estimates conditional quantiles via asymmetric loss, enabling robust heteroskedastic regression.

Principles:
1. Asymmetric loss weights underestimation by τ.
2. Estimates full conditional distribution.
3. Robust to outliers.
4. Multiple quantiles reveal heteroskedasticity.
5. Non-crossing quantiles may require post-hoc adjustment.

---

---

## Appendix: Practical Labs

### Lab 1: Basic Quantile Regression

import numpy as np
from scipy.optimize import minimize

def quantile_loss(y, X, beta, tau):
 residuals = y - X @ beta
 return np.sum(np.where(residuals >= 0, tau * residuals, (tau - 1) * residuals))

np.random.seed(42)
n = 100
X = np.column_stack([np.ones(n), np.random.randn(n)])
y = 2 + 1.5 * X[:, 1] + np.random.randn(n) * (0.5 + X[:, 1]**2)

for tau in [0.1, 0.5, 0.9]:
 result = minimize(lambda b: quantile_loss(y, X, b, tau), [0, 0])
 beta = result.x
 print(f"Tau {tau}: {beta}")

print("✓ Quantile regression working")

if __name__ == "__main__":
 print("Lab 1: Quantile - PASSED")

### Lab 2: Multiple Quantiles

import numpy as np

def estimate_quantiles(y, X, taus):
 quantiles = {}
 for tau in taus:
 loss = lambda b: np.sum(np.where(y - X@b >= 0, tau*(y-X@b), (tau-1)*(y-X@b)))
 from scipy.optimize import minimize
 result = minimize(loss, [0, 0])
 quantiles[tau] = result.x
 return quantiles

np.random.seed(42)
X = np.column_stack([np.ones(50), np.random.randn(50)])
y = 1 + 2*X[:, 1] + np.random.randn(50)

taus = [0.25, 0.5, 0.75]
quants = estimate_quantiles(y, X, taus)

assert len(quants) == 3, "Should estimate 3 quantiles"
print("✓ Multiple quantiles working")

if __name__ == "__main__":
 print("Lab 2: Quantiles - PASSED")

### Lab 3: Quantile Loss Comparison

import numpy as np

def quantile_error(y_true, y_pred, tau):
 residuals = y_true - y_pred
 return np.sum(np.where(residuals >= 0, tau*residuals, (tau-1)*residuals))

np.random.seed(42)
y_true = np.random.randn(100)
y_pred = y_true + np.random.randn(100)*0.2

tau_list = [0.1, 0.5, 0.9]
errors = [quantile_error(y_true, y_pred, tau) for tau in tau_list]

print(f"Quantile errors: {errors}")
assert all(e > 0 for e in errors), "Errors should be positive"
print("✓ Quantile loss working")

if __name__ == "__main__":
 print("Lab 3: Loss - PASSED")

### Lab 4: Heteroskedasticity Detection

import numpy as np

def quantile_interval_width(y, X, tau_low=0.1, tau_high=0.9):
 from scipy.optimize import minimize
 
 intervals = []
 for i in range(len(X)):
 x_i = X[i:i+1]
 
 loss_low = lambda b: np.sum(np.where(y - x_i@b >= 0, tau_low*(y-x_i@b), (tau_low-1)*(y-x_i@b)))
 loss_high = lambda b: np.sum(np.where(y - x_i@b >= 0, tau_high*(y-x_i@b), (tau_high-1)*(y-x_i@b)))
 
 r_low = minimize(loss_low, [0, 0]).x
 r_high = minimize(loss_high, [0, 0]).x
 
 width = (x_i @ r_high - x_i @ r_low)[0]
 intervals.append(width)
 
 return np.array(intervals)

np.random.seed(42)
X = np.column_stack([np.ones(30), np.random.randn(30)])
y = X[:, 1] + np.random.randn(30) * (0.1 + abs(X[:, 1]))

widths = quantile_interval_width(y, X)
print(f"Interval widths: {widths[:5]}")
assert len(widths) == 30, "Should have 30 intervals"
print("✓ Heteroskedasticity detection working")

if __name__ == "__main__":
 print("Lab 4: Heteroskedasticity - PASSED")