contextual bandit
**A contextual bandit** is a reinforcement learning framework where an agent makes decisions based on **context (features/state)** available at decision time, receives a reward for its choice, but doesn't observe what would have happened with other choices. It sits between simple multi-armed bandits (no context) and full RL (sequential decisions).
**How Contextual Bandits Work**
- **Observe Context**: The agent receives a context vector $x$ — features describing the current situation.
- **Select Action**: Based on the context, the agent selects an action $a$ from a set of possible actions.
- **Receive Reward**: The environment returns a reward $r(x, a)$ for the chosen action.
- **Learn**: The agent updates its policy to improve future action selection.
- **No observation** of rewards for actions not taken (the counterfactual problem).
**Examples**
- **News Recommendation**: Context = user profile + time of day. Actions = articles to show. Reward = whether the user clicked.
- **Ad Placement**: Context = user demographics + page content. Actions = which ad to display. Reward = click or purchase.
- **LLM Routing**: Context = query characteristics. Actions = which model to send the query to. Reward = response quality score.
- **Clinical Trials**: Context = patient characteristics. Actions = treatment options. Reward = health outcome.
**Key Algorithms**
- **LinUCB**: Linear model for each action with Upper Confidence Bound exploration. Balances exploitation of known-good actions with exploration of uncertain ones.
- **Thompson Sampling**: Bayesian approach — maintain a posterior distribution over expected rewards for each action and sample from it to select actions.
- **Epsilon-Greedy**: With probability ε, explore randomly; otherwise, exploit the best-estimated action.
- **Neural Contextual Bandits**: Use neural networks to model the context-reward relationship for complex, high-dimensional contexts.
**Contextual Bandits vs. Full RL**
| Aspect | Contextual Bandit | Full RL |
|--------|-------------------|--------|
| **State** | Single observation | Sequential states |
| **Actions** | One decision | Sequence of decisions |
| **Consequence** | Immediate reward | Delayed rewards |
| **Complexity** | Moderate | High |
Contextual bandits are the **sweet spot** for many real-world decision problems — they handle personalization and context while being simpler and more data-efficient than full reinforcement learning.