Process Reward Model (PRM) is a reward model that assigns scores to each intermediate reasoning step rather than only the final answer — enabling fine-grained training signal for multi-step reasoning tasks where step-level correctness matters more than final outcome.

ORM vs. PRM

• ORM (Outcome Reward Model): Single reward for correct/incorrect final answer. Simple but sparse signal.
• PRM (Process Reward Model): Score each reasoning step (correct/incorrect/uncertain). Dense, step-level signal.
• ORM limitation: Wrong reasoning that accidentally reaches correct answer gets full reward.
• PRM advantage: Penalizes incorrect reasoning steps even if final answer is correct — promotes genuine understanding.

PRM Training

• Requires annotated reasoning chains: Each step labeled correct/incorrect by human or automated checker.
• OpenAI PRM800K: 800K step-level human annotations of math reasoning chains.
• Training: Train classifier to predict step-level correctness.
• Inference: Use PRM scores to guide beam search or MCTS over reasoning trees.

PRM Applications

• Best-of-N with PRM: Generate N chains; select the one with highest PRM score.
• More discriminative than ORM for reasoning tasks.
• MCTS with PRM: Tree search guided by PRM step scores — AlphaGo-style for math.
• Training signal for RLHF: Dense step-level rewards improve PPO training stability.

Math Reasoning Results

• DeepMind Gemini with PRM: 51% on AIME 2024 (vs. 9% without).
• OpenAI o1: Combines PRM + extended "thinking time" — internal reasoning chain.
• Scaled inference compute + PRM: Log-linear relationship between compute and accuracy.

Challenges

• Annotation cost: Step-level labeling is expensive.
• Automated verification: Only feasible where answers are checkable (math, code).
• Reward hacking: PRM itself can be exploited — adversarial steps that score well but are wrong.

Process reward models are the key to closing the gap between raw reasoning capability and reliable problem-solving — by rewarding correct thinking processes rather than just correct answers, PRMs enable the kind of robust multi-step reasoning that characterizes mathematical expertise.