log transform
**Log Transformation** is a **data preprocessing technique that applies the logarithm function to compress large values and spread out small values** — converting right-skewed distributions (income, house prices, website traffic) into approximately normal distributions that linear models, neural networks, and statistical tests assume, while stabilizing variance so that predictions are equally reliable across the range rather than more accurate for small values and wildly inaccurate for large values.
**What Is Log Transformation?**
- **Definition**: A mathematical transformation that replaces each value $x$ with $log(x)$ — typically using the natural logarithm (ln) or $log(x + 1)$ (log1p) to handle zeros, compressing the dynamic range of the data.
- **Why It's Needed**: Many real-world variables have right-skewed distributions — a few CEOs earn $10M+ while most employees earn $50-100K. The raw distribution has a long right tail that violates normality assumptions, inflates the mean, and makes outlier detection unreliable. Log transformation compresses the tail.
- **Formula**: $X_{new} = log(X + 1)$ — the +1 handles zero values since $log(0)$ is undefined.
**When to Use Log Transformation**
| Data Type | Skew | Example | Effect of Log |
|-----------|------|---------|--------------|
| **Income/Salary** | Heavy right skew | $30K, $50K, $80K, $500K, $10M | Compresses outlier salaries |
| **House Prices** | Moderate right skew | $200K, $400K, $2M, $50M | Makes distribution more symmetric |
| **Website Traffic** | Heavy right skew | 10, 50, 200, 1M page views | Equalizes small and large sites |
| **Count Data** | Right skew | 0, 1, 3, 5, 500 retweets | Spreads low counts, compresses high |
| **Elapsed Time** | Right skew | 1s, 5s, 30s, 600s response times | Normalizes response time distribution |
**Before and After Example**
| Original Salary | Log(Salary + 1) | Effect |
|----------------|-----------------|--------|
| $30,000 | 10.31 | Slightly compressed |
| $50,000 | 10.82 | Slightly compressed |
| $80,000 | 11.29 | Slightly compressed |
| $500,000 | 13.12 | Moderately compressed |
| $10,000,000 | 16.12 | Heavily compressed |
The range went from $30K-$10M (333× ratio) to 10.31-16.12 (1.56× ratio) — dramatically reducing the impact of extreme values.
**Python Implementation**
```python
import numpy as np
import pandas as pd
# Log1p (handles zeros safely)
df["log_salary"] = np.log1p(df["salary"])
# Reverse: expm1 to get back original scale
df["original"] = np.expm1(df["log_salary"])
```
**Common Alternatives**
| Transform | Formula | When to Use |
|-----------|---------|------------|
| **Log (ln)** | $log(x + 1)$ | Standard for right-skewed data |
| **Square Root** | $sqrt{x}$ | Less aggressive compression than log |
| **Box-Cox** | Finds optimal λ | When the best transform is unknown |
| **Yeo-Johnson** | Modified Box-Cox | Works with negative values (Box-Cox requires positive) |
**Log Transformation is the standard preprocessing technique for right-skewed data** — normalizing distributions that violate model assumptions, stabilizing variance across the value range, and compressing extreme outliers, making it one of the first transformations to try when features span multiple orders of magnitude.