Verifier Model
A verifier model is a separate model trained to judge whether a candidate solution is correct, typically by outputting a score for a problem and a proposed answer. Rather than asking a generator to be right on its first try, the system lets the generator propose many candidates and lets the verifier pick. This splits one hard job, producing correct reasoning, into two jobs of which the second, recognizing correct reasoning, is often easier for a model to learn than the first.
The canonical reference is Cobbe et al. (2021, "Training Verifiers to Solve Math Word Problems"), the OpenAI paper that introduced the GSM8K dataset of roughly 8,500 grade-school math word problems. They fine-tuned a generator, sampled many candidate solutions per problem at test time, and used a trained verifier to rank them and return the highest-scoring one. The verifier was trained on generator samples labelled correct or incorrect according to whether the final answer matched. Their headline finding was that verification scaled better than plain fine-tuning: a much smaller model with a verifier could match or beat a far larger model fine-tuned on the same data, and the gap widened as the training set grew. The paper also noted that verification worked partly because it gave a model a way to check many different attempts instead of betting on one.
Best-of-n is the simplest way to use a verifier: draw n independent samples from the generator, score each one, and keep the top scorer. Accuracy rises with n at first and then flattens or even declines, because with more candidates the verifier has more opportunities to be fooled by a wrong solution that merely looks plausible. This is a form of spending test-time compute to buy accuracy, and it sits next to self-consistency (Wang et al., 2022), which takes a majority vote over sampled final answers with no learned verifier at all. A verifier can beat majority vote when correct answers are rare but recognizable, whereas majority vote wins when the verifier is poorly calibrated.
What the verifier actually judges is a design choice with real consequences. An outcome-style verifier looks only at whether the whole solution is right, while a step-level verifier scores each reasoning step; the first is cheaper to label, the second pinpoints errors. Lightman et al. (2023, "Let's Verify Step by Step") compared the two on competition mathematics with human step labels (the PRM800K dataset) and found step-level supervision more reliable. The verifier can also be a program rather than a network: unit tests for code and exact-match answer checks for arithmetic are verifiers with no learned parameters, and these are the signals that reinforcement learning on reasoning models uses at scale.
The pitfall is verifier exploitation, a case of Goodhart's law. A learned verifier is itself an imperfect model, so the more aggressively a generator is optimized against it, or the larger n becomes, the more likely the selected answer is one that fools the verifier rather than one that is correct. Verifiers also tend to transfer poorly out of distribution: one trained on grade-school arithmetic cannot be assumed to judge olympiad proofs. A verifier is not the same thing as a reward-model used in RLHF, which encodes human preferences over open-ended text; a verifier targets correctness, and it is mostly used at inference time to choose among candidates, though its scores can also be turned into training data.
Verifiers matter because they convert extra compute into reliability without retraining the generator. Systems that combine sampling, ranking, and tool-based checking underlie much of the progress on math and code benchmarks, and the same pattern appears in agent pipelines where a checker approves or rejects each proposed action. The cost is straightforward: n samples cost roughly n times the generation compute, so the technique is best reserved for tasks where an answer can be checked cheaply and a wrong answer is expensive.
Read a verifier model as a second opinion that is cheaper to train than a perfect solver: best-of-n lets a generator gamble many times while the verifier keeps only the winning ticket, and the whole scheme is exactly as trustworthy as the verifier's ability to tell a correct solution from a convincing one.