process reward model

**Process Reward Model (PRM)** is a **reward model that assigns scores to each intermediate reasoning step rather than only the final answer** — enabling fine-grained training signal for multi-step reasoning tasks where step-level correctness matters more than final outcome. **ORM vs. PRM** - **ORM (Outcome Reward Model)**: Single reward for correct/incorrect final answer. Simple but sparse signal. - **PRM (Process Reward Model)**: Score each reasoning step (correct/incorrect/uncertain). Dense, step-level signal. - ORM limitation: Wrong reasoning that accidentally reaches correct answer gets full reward. - PRM advantage: Penalizes incorrect reasoning steps even if final answer is correct — promotes genuine understanding. **PRM Training** - Requires annotated reasoning chains: Each step labeled correct/incorrect by human or automated checker. - OpenAI PRM800K: 800K step-level human annotations of math reasoning chains. - Training: Train classifier to predict step-level correctness. - Inference: Use PRM scores to guide beam search or MCTS over reasoning trees. **PRM Applications** - **Best-of-N with PRM**: Generate N chains; select the one with highest PRM score. - More discriminative than ORM for reasoning tasks. - **MCTS with PRM**: Tree search guided by PRM step scores — AlphaGo-style for math. - **Training signal for RLHF**: Dense step-level rewards improve PPO training stability. **Math Reasoning Results** - DeepMind Gemini with PRM: 51% on AIME 2024 (vs. 9% without). - OpenAI o1: Combines PRM + extended "thinking time" — internal reasoning chain. - Scaled inference compute + PRM: Log-linear relationship between compute and accuracy. **Challenges** - Annotation cost: Step-level labeling is expensive. - Automated verification: Only feasible where answers are checkable (math, code). - Reward hacking: PRM itself can be exploited — adversarial steps that score well but are wrong. Process reward models are **the key to closing the gap between raw reasoning capability and reliable problem-solving** — by rewarding correct thinking processes rather than just correct answers, PRMs enable the kind of robust multi-step reasoning that characterizes mathematical expertise.