reward learning

**Reward Learning** is the **general problem of learning a reward function from various forms of human feedback** — including demonstrations, preferences, corrections, natural language instructions, and other signals, to define the objective for RL without manual reward engineering. **Forms of Reward Feedback** - **Demonstrations**: Expert trajectories (IRL) — infer the reward from observed expert behavior. - **Preferences**: Pairwise comparisons — learn which behaviors are preferred. - **Corrections**: Physical corrections or partial feedback — the human adjusts the agent's behavior. - **Language**: Natural language descriptions of desired behavior — ground language to reward. **Why It Matters** - **Reward Design is Hard**: Manual reward engineering leads to reward hacking — reward learning avoids this. - **Alignment**: Learning rewards from human feedback is the core of AI alignment — ensure AI optimizes for human values. - **Specification**: Reward learning turns informal human preferences into formal optimization objectives. **Reward Learning** is **letting humans define the objective** — automatically constructing reward functions from diverse forms of human feedback.