Reasoning Model

A reasoning model is trained with reinforcement learning specifically to generate an extended internal reasoning trace before answering, rather than simply being prompted to do so — the distinction between this and the prompted chain-of-thought technique is that the reasoning behavior here is baked into the model's training, not requested at inference time. OpenAI's o1, released in September 2024, was the first widely known model built this way, generating a reasoning trace largely hidden from the user before producing a final answer; DeepSeek's R1, released openly in January 2025, demonstrated the same reasoning capability could be trained using a reinforcement learning method called GRPO — Group Relative Policy Optimization, introduced in DeepSeek's own 2024 DeepSeekMath paper — at a training cost dramatically lower than what had been assumed necessary to reach this level of capability.

Whether the reward only checks the final answer or scores every intermediate step turns out to matter enormously, and OpenAI's own research settled this question before reasoning models became a product category. Lightman and colleagues' 2023 paper "Let's Verify Step by Step" compared outcome reward models, which only score whether the final answer was correct, against process reward models, which score the correctness of each individual reasoning step along the way, and found process supervision produced meaningfully more reliable reasoning — a model can stumble onto a correct final answer through flawed intermediate logic, and an outcome-only reward has no way to catch or discourage that, while a process reward model can penalize the flawed step even when the final answer happens to be right.

DeepSeek's GRPO earns its place in reasoning-model training specifically because it removes a piece of infrastructure standard reinforcement learning methods normally require. Where a standard policy-gradient method like PPO trains a separate value network to estimate how good a given state is, GRPO instead generates a whole group of candidate responses to the same prompt, scores each one, and uses each response's rank relative to the rest of that same group as the training signal — no separate value network needed, which is a meaningful part of why DeepSeek was able to train R1-level reasoning capability with substantially less computational infrastructure than a value-network-based approach would have required.

Extended reasoning costs real tokens, and that cost is the whole reason reasoning models aren't used for everything by default. A reasoning model's internal thinking phase can run to thousands of tokens on a hard problem before it ever produces a visible answer, and since every one of those tokens costs compute and money the same way any generated token does, reasoning models run meaningfully slower and more expensively per query than a standard model answering directly — which is exactly why providers expose a configurable "thinking budget" or reasoning effort setting, letting a team spend that extra cost only on problems — competition math, debugging, multi-step planning — where the accuracy gain is actually worth paying for, rather than applying it by default to every request.

Reasoning models are the product-ized version of the same insight behind chain-of-thought — that a model's accuracy on hard problems is a dial tied to how much it computes before answering — except here that dial is set by training with reinforcement learning rather than by a user's prompt, and the specific training choices covered here (process supervision over outcome-only supervision, GRPO's group-relative signal instead of a separate value network) are the engineering answers to making that training actually work at a cost labs could afford to pay.