box cox
**Box-Cox Transformation** is a **power transformation that automatically finds the optimal mathematical function to normalize data** — searching over a parameter λ (lambda) to determine whether the data needs a log transform (λ=0), square root (λ=0.5), reciprocal (λ=-1), no transform (λ=1), or any other power between them, making it the data-driven alternative to manually guessing which transformation to apply to skewed features.
**What Is the Box-Cox Transformation?**
- **Definition**: A family of power transformations parameterized by λ that transforms the data to be as close to a normal distribution as possible — the algorithm finds the optimal λ using maximum likelihood estimation.
- **The Formula**:
- If $lambda
eq 0$: $y_{new} = frac{y^{lambda} - 1}{lambda}$
- If $lambda = 0$: $y_{new} = log(y)$
- **Why Not Just Use Log?**: Log transformation assumes the data needs logarithmic compression. But some data needs square root (λ=0.5), cube root (λ=0.33), or even no transformation (λ=1). Box-Cox finds the optimal power automatically.
**Lambda Values Explained**
| λ (Lambda) | Transformation | When Optimal | Effect |
|-----------|---------------|-------------|--------|
| -1 | Reciprocal ($1/y$) | Heavily right-skewed | Extreme compression |
| -0.5 | Reciprocal square root ($1/sqrt{y}$) | Very right-skewed | Strong compression |
| 0 | Log($y$) | Moderately right-skewed | Logarithmic compression |
| 0.5 | Square root ($sqrt{y}$) | Mildly right-skewed | Mild compression |
| 1 | No transformation ($y$ itself) | Already normal | No change needed |
| 2 | Square ($y^2$) | Left-skewed | Expansion (rare) |
**How Box-Cox Finds Optimal λ**
| Step | Process |
|------|---------|
| 1. Try many λ values | Test λ from -5 to +5 in small increments |
| 2. For each λ, transform data | Apply $y^{(lambda)}$ formula |
| 3. Measure normality | Log-likelihood of the transformed data under a normal distribution |
| 4. Select best λ | The λ that maximizes log-likelihood (makes data most normal) |
**Python Implementation**
```python
from scipy.stats import boxcox
from sklearn.preprocessing import PowerTransformer
# SciPy (returns transformed data + optimal lambda)
data_transformed, optimal_lambda = boxcox(data)
print(f"Optimal lambda: {optimal_lambda:.2f}")
# Scikit-learn (fits inside pipeline, handles inverse)
pt = PowerTransformer(method='box-cox') # requires positive data
X_transformed = pt.fit_transform(X)
```
**Box-Cox vs Yeo-Johnson**
| Property | Box-Cox | Yeo-Johnson |
|----------|---------|-------------|
| **Input requirement** | Strictly positive ($y > 0$) | Any value (positive, zero, negative) |
| **Zero handling** | Cannot handle zeros | Yes |
| **Negative values** | Cannot handle | Yes |
| **Optimal for** | Positive continuous data | General-purpose |
| **Scikit-learn** | `PowerTransformer(method='box-cox')` | `PowerTransformer(method='yeo-johnson')` |
**When to Use**
| Use Box-Cox / Yeo-Johnson | Don't Use |
|---------------------------|----------|
| Linear models that assume normality | Tree-based models (don't need normality) |
| Right or left-skewed features | Already normally distributed data |
| When you don't know which transform to apply | When you know log transform is correct |
| Preprocessing for statistical tests | Categorical or binary features |
**Box-Cox Transformation is the automated alternative to manual transformation selection** — finding the optimal power parameter λ through maximum likelihood estimation to produce the most normal-like distribution possible, with Yeo-Johnson as its generalization that handles the zero and negative values that Box-Cox cannot.