Decision Trees are a supervised machine learning algorithm that makes predictions by learning a series of if-then-else decision rules from training data, organized as a tree structure — where each internal node asks a question about a feature ("Is income > $50K?"), each branch represents an answer, and each leaf node provides the prediction, making them the most interpretable ML model (you can literally visualize and explain every decision), while Random Forests aggregate hundreds of decision trees to eliminate overfitting and achieve production-grade accuracy.
What Is a Decision Tree?
- Definition: A tree-shaped model where data flows from the root through internal decision nodes (questions) to leaf nodes (predictions) — used for both classification ("Will this customer churn? Yes/No") and regression ("What will the house price be?").
- Interpretability: The #1 advantage — you can print the tree and explain every prediction to a non-technical stakeholder: "The model predicted churn because: tenure < 6 months AND support tickets > 3 AND plan = Basic."
- Human-Like Reasoning: Decision trees mimic how humans make decisions — a doctor diagnosing a patient goes through a mental decision tree: "Does the patient have a fever? → Is it above 103°F? → Does the patient have a rash?"
How Trees Learn: Splitting Criteria
| Criterion | Formula | Used For | Intuition |
|---|---|---|---|
| Gini Impurity | $1 - sum p_i^2$ | Classification | How "mixed" are the labels in this node? |
| Entropy (Info Gain) | $-sum p_i log_2 p_i$ | Classification | How much uncertainty is reduced by this split? |
| MSE (Mean Squared Error) | $frac{1}{n}sum(y_i - ar{y})^2$ | Regression | How well does the mean predict all values? |
The tree picks the feature and threshold that produces the "purest" child nodes — splitting data so that each branch contains mostly one class.
The Overfitting Problem
A single decision tree will memorize the training data if grown without constraints — achieving 100% training accuracy but poor generalization. Solutions:
| Technique | Approach | Effect |
|---|---|---|
| Max Depth | Limit tree depth (e.g., max_depth=5) | Prevents overly specific rules |
| Min Samples | Require minimum samples per leaf | Prevents single-example leaves |
| Pruning | Remove branches that don't improve validation accuracy | Simplifies after training |
| Random Forest | Aggregate hundreds of trees | The standard solution |
Random Forest
- How: Train 100-1000 decision trees, each on a random subset of data (bagging) and features. Final prediction = majority vote (classification) or average (regression).
- Why It Works: Individual trees overfit in different ways — averaging their predictions cancels out individual errors, producing a stable, accurate model.
- When to Use: Tabular data (spreadsheets, databases, structured features) — Random Forests are often the #1 choice for tabular ML before trying deep learning.
Decision Trees and Random Forests are the most practical ML algorithms for structured/tabular data — providing interpretable predictions through human-readable decision rules in single trees, and production-grade accuracy through Random Forest ensembles that combine hundreds of trees to eliminate overfitting, making them the first algorithm to try for classification and regression on tabular datasets.
Explore 500+ Semiconductor & AI Topics
From EUV lithography to CUDA optimization — search the full knowledge base or chat with our AI assistant.