Value Learning
Value learning is the family of approaches in which an AI system is not given a fixed, hand-written objective but instead infers what humans want from evidence — their behavior, choices, feedback, demonstrations or stated preferences — and acts on that inferred understanding, usually while remaining uncertain about it. It is motivated by a simple difficulty: human values are complex, context-dependent and hard to write down, and an AI that optimizes a literal but flawed specification can satisfy the letter of an instruction while defeating its purpose.
The foundational technique is inverse reinforcement learning (IRL), introduced by Andrew Ng and Stuart Russell in their 2000 paper "Algorithms for Inverse Reinforcement Learning." Ordinary reinforcement learning starts from a reward function and finds good behavior; IRL runs the problem backwards, observing behavior from an expert and searching for a reward function that would explain it. Pieter Abbeel and Ng's 2004 work on apprenticeship learning showed that learning from demonstrated driving could succeed this way, and Brian Ziebart and colleagues' 2008 maximum-entropy formulation gave a probabilistic version that is more robust to imperfect demonstrators. A basic difficulty was noted from the start: the problem is ill-posed, since many different reward functions, including trivial ones, are consistent with the same observed behavior, so some additional assumptions or priors are needed to choose between them.
Cooperative inverse reinforcement learning (CIRL), proposed by Dylan Hadfield-Menell, Stuart Russell, Pieter Abbeel and Anca Dragan in 2016, reframed the problem as a cooperative game. In CIRL, a human knows the reward function and the robot does not, but both are rewarded according to it, so the optimal solution for the pair involves the human teaching actively rather than acting naively and the robot asking questions, deferring and acting cautiously where it is uncertain. This differs from classical IRL, which treats the human as a passive source of data, and it is the basis of Stuart Russell's 2019 book Human Compatible, which argues that advanced systems should be designed with objectives that they are explicitly uncertain about. The same idea underlies the off-switch game discussed under corrigibility.
Reinforcement learning from human feedback (RLHF) is the practical cousin that actually reached products. Paul Christiano and colleagues' 2017 paper "Deep Reinforcement Learning from Human Preferences" trained a reward model from human comparisons of behavior clips, avoiding the need to hand-specify a reward; OpenAI's 2022 InstructGPT paper by Ouyang and colleagues applied the same pattern to language models, and variants such as Anthropic's Constitutional AI and direct preference optimization changed the details. RLHF shares IRL's logic of learning a reward from human signals, but in practice it fits a single reward model to aggregated labeler judgments and then optimizes the policy against it, with no explicit treatment of ongoing uncertainty about values. The reward model and DPO have separate entries.
The approach has deep limitations that value-learning researchers emphasize. Humans are not perfectly rational, so inferring values from behavior requires a model of human error, and Stuart Armstrong and Sören Mindermann's 2017 analysis argued that simplicity priors alone cannot separate a person's preferences from their biases. Preferences differ between individuals and cultures, change over time and may be shaped by the very system that learns them. Learned reward models can be over-optimized, so that a policy exploits their flaws, a failure called reward hacking or Goodhart's law. And evidence from ordinary users may not reflect what they would endorse on reflection.
A common misconception is that value learning means teaching an AI the one correct set of human values. The goal in most formulations is more modest and procedural: build systems that stay responsive to human guidance, express uncertainty and keep learning, with questions of whose values count remaining political and ethical choices rather than technical ones. A second pitfall is to treat RLHF as having solved the problem; it improves helpfulness and reduces many harmful outputs, but it trains the appearance of preferred behavior and does not guarantee that the model's internal goals match the intended ones, a concern pursued under inner alignment.
Read value learning as the shift from telling machines what to optimize to letting them find out what people want: its progress runs from inferring a reward from expert behavior to cooperative games and preference-trained models, and its core lesson is that an AI should remain uncertain about values, and humble in acting on them.