Outcome Reward Model
An outcome reward model (ORM) is a reward model that scores a response by whether its final result is good, ignoring the path taken to reach it. For a math problem the label is simply whether the final answer is correct; for a coding task, whether the tests pass. The model is trained to predict that end-of-sequence label, and it is then used to rank candidate solutions or to supply a training signal. The defining contrast is with a process reward model (PRM), which scores each intermediate step instead.
The clearest early comparison is Uesato et al. (2022, "Solving math word problems with process- and outcome-based feedback", DeepMind), which trained models on GSM8K under both regimes. Their finding was nuanced: outcome-based supervision reached similar final-answer accuracy to process-based supervision, but process-based supervision produced noticeably fewer solutions in which the final answer was right while the reasoning was wrong. Lightman et al. (2023, "Let's Verify Step by Step", OpenAI) then compared the two on harder competition mathematics using 800,000 human step-level labels (PRM800K) and found the process-supervised reward model chose correct solutions more reliably than an outcome-supervised one. Together these papers established that ORMs are the cheap baseline and PRMs the more precise alternative.
The central weakness of an ORM is credit assignment: one bit of feedback at the end of a long chain of reasoning says nothing about which step was the mistake. A solution with ten steps and one subtle error receives the same "wrong" label as a solution that was nonsense from the first line, and a solution that stumbles into the right answer through two cancelling errors receives full marks. Training an ORM therefore teaches it to correlate overall style and plausibility with success, which can reward lucky or even unfaithful reasoning, and it gives a generator no gradient about where it went astray. The benefit, in exchange, is that labels are nearly free whenever the answer can be checked automatically, with no human annotating steps.
The term "outcome reward" is used in two related but different senses, and keeping them apart avoids confusion. In the ORM sense, a neural network is trained to predict final-answer correctness and is used as a learned judge. In the rule-based sense, no network is needed at all: an exact-match check or a unit-test run returns the reward directly. DeepSeek's R1 work (2025) is a well-known example of the second kind, using rule-based accuracy and format rewards for reinforcement learning on reasoning and deliberately avoiding neural reward models for that stage, citing the risk of reward hacking. Reinforcement learning with verifiable rewards (RLVR) follows the same logic, as in the Tulu 3 training recipe from the Allen Institute for AI (2024).
An ORM should not be confused with the general reward-model of RLHF, which learns human preferences over open-ended outputs from pairwise comparisons. A preference reward model answers which of two responses a person would prefer, while an ORM answers whether a task with a checkable answer was solved. The two share an architecture, typically a language model with a scalar head, but differ in labels, domain, and failure modes. An ORM also differs from a verifier-model used for best-of-n only in emphasis: many verifiers are simply outcome-supervised classifiers applied to sampled solutions.
The pitfalls are the usual ones for any learned reward, plus a few specific ones. Optimizing hard against an ORM invites reward hacking, where a generator learns surface patterns that the model likes rather than correct reasoning. Outcome rewards are also sparse, which makes learning slow on very hard problems where nearly every attempt fails, and they cannot be defined at all for tasks with no checkable answer, such as essays or strategy advice. Teams often mix them with step-level signals, length penalties, or consistency checks to compensate.
Read an outcome reward model as a grader who looks only at the answer key: that makes grading cheap, scalable, and easy to automate, but it cannot tell a sound derivation from a lucky guess, which is exactly the gap that process supervision was invented to close.