decision tree
**Decision Trees** are a **supervised machine learning algorithm that makes predictions by learning a series of if-then-else decision rules from training data, organized as a tree structure** — where each internal node asks a question about a feature ("Is income > $50K?"), each branch represents an answer, and each leaf node provides the prediction, making them the most interpretable ML model (you can literally visualize and explain every decision), while **Random Forests** aggregate hundreds of decision trees to eliminate overfitting and achieve production-grade accuracy.
**What Is a Decision Tree?**
- **Definition**: A tree-shaped model where data flows from the root through internal decision nodes (questions) to leaf nodes (predictions) — used for both classification ("Will this customer churn? Yes/No") and regression ("What will the house price be?").
- **Interpretability**: The #1 advantage — you can print the tree and explain every prediction to a non-technical stakeholder: "The model predicted churn because: tenure < 6 months AND support tickets > 3 AND plan = Basic."
- **Human-Like Reasoning**: Decision trees mimic how humans make decisions — a doctor diagnosing a patient goes through a mental decision tree: "Does the patient have a fever? → Is it above 103°F? → Does the patient have a rash?"
**How Trees Learn: Splitting Criteria**
| Criterion | Formula | Used For | Intuition |
|-----------|---------|----------|-----------|
| **Gini Impurity** | $1 - sum p_i^2$ | Classification | How "mixed" are the labels in this node? |
| **Entropy (Info Gain)** | $-sum p_i log_2 p_i$ | Classification | How much uncertainty is reduced by this split? |
| **MSE (Mean Squared Error)** | $frac{1}{n}sum(y_i - ar{y})^2$ | Regression | How well does the mean predict all values? |
The tree picks the feature and threshold that produces the "purest" child nodes — splitting data so that each branch contains mostly one class.
**The Overfitting Problem**
A single decision tree will memorize the training data if grown without constraints — achieving 100% training accuracy but poor generalization. Solutions:
| Technique | Approach | Effect |
|-----------|---------|--------|
| **Max Depth** | Limit tree depth (e.g., max_depth=5) | Prevents overly specific rules |
| **Min Samples** | Require minimum samples per leaf | Prevents single-example leaves |
| **Pruning** | Remove branches that don't improve validation accuracy | Simplifies after training |
| **Random Forest** | Aggregate hundreds of trees | The standard solution |
**Random Forest**
- **How**: Train 100-1000 decision trees, each on a random subset of data (bagging) and features. Final prediction = majority vote (classification) or average (regression).
- **Why It Works**: Individual trees overfit in different ways — averaging their predictions cancels out individual errors, producing a stable, accurate model.
- **When to Use**: Tabular data (spreadsheets, databases, structured features) — Random Forests are often the #1 choice for tabular ML before trying deep learning.
**Decision Trees and Random Forests are the most practical ML algorithms for structured/tabular data** — providing interpretable predictions through human-readable decision rules in single trees, and production-grade accuracy through Random Forest ensembles that combine hundreds of trees to eliminate overfitting, making them the first algorithm to try for classification and regression on tabular datasets.