A contextual bandit is a reinforcement learning framework where an agent makes decisions based on context (features/state) available at decision time, receives a reward for its choice, but doesn't observe what would have happened with other choices. It sits between simple multi-armed bandits (no context) and full RL (sequential decisions).

How Contextual Bandits Work

• Observe Context: The agent receives a context vector $x$ — features describing the current situation.
• Select Action: Based on the context, the agent selects an action $a$ from a set of possible actions.
• Receive Reward: The environment returns a reward $r(x, a)$ for the chosen action.
• Learn: The agent updates its policy to improve future action selection.
• No observation of rewards for actions not taken (the counterfactual problem).

Examples

• News Recommendation: Context = user profile + time of day. Actions = articles to show. Reward = whether the user clicked.
• Ad Placement: Context = user demographics + page content. Actions = which ad to display. Reward = click or purchase.
• LLM Routing: Context = query characteristics. Actions = which model to send the query to. Reward = response quality score.
• Clinical Trials: Context = patient characteristics. Actions = treatment options. Reward = health outcome.

Key Algorithms

• LinUCB: Linear model for each action with Upper Confidence Bound exploration. Balances exploitation of known-good actions with exploration of uncertain ones.
• Thompson Sampling: Bayesian approach — maintain a posterior distribution over expected rewards for each action and sample from it to select actions.
• Epsilon-Greedy: With probability ε, explore randomly; otherwise, exploit the best-estimated action.
• Neural Contextual Bandits: Use neural networks to model the context-reward relationship for complex, high-dimensional contexts.

Contextual Bandits vs. Full RL

Contextual bandits are the sweet spot for many real-world decision problems — they handle personalization and context while being simpler and more data-efficient than full reinforcement learning.