Advanced Reasoning Models: Scaling Test-Time Compute — LLMs with extended thinking for math, coding, and science tasks

OpenAI o1, o3, and DeepSeek-R1 introduce extended thinking (reasoning steps) at test time, allocating significant compute per problem (not just forward pass). This test-time scaling achieves breakthrough performance on challenging benchmarks.

Extended Thinking and Process Supervision

o1 (OpenAI, 2024): generates internal reasoning (hidden from user) before outputting final answer. Reasoning trajectory (chain-of-thought in latent space): explores problem space, backtracks, validates intermediate results. Training: reinforcement learning on correctness of final answer (outcome reward) plus intermediate reasoning quality (process reward). o3 (announced 2025): improved reasoning, claimed state-of-the-art on AIME (99.2%), GPQA (92%, human expert ~80%).

Process Reward Models

PRM: supervise intermediate steps during reasoning, not just final answer. Label each step in reasoning trajectory (correct/incorrect/helpful). Training: classifier predicts step correctness. Inference: generate step, score with PRM, if incorrect, prune and backtrack—guided search through reasoning space. Iterative refinement: rewrite steps, validate, continue. Significantly outperforms outcome reward model (RM) which only scores final answers.

GRPO: Grounded Reason-Preference Optimization

DeepSeek-R1 (DeepSeek, 2024) uses GRPO training: RL method combining RM scores with language model objectives. Generate reasoning + answer, score via RM, compute preference pairs (good reasoning > bad reasoning), update policy. 671B parameter model, trained on standard + reasoning-heavy datasets. Performance: AIME 96%, SWE-bench 96% (programming), GPQA 90% (science), competitive with o1.

Thinking Budget and Inference Cost

Reasoning phase: generates 5,000-30,000 tokens per query (10-100x normal completion). Cost/latency: 10-100x higher than standard LLM inference. Thinking budget: configurable maximum reasoning tokens (trade-off accuracy vs. cost). Applications: high-value problems (competition math, scientific research, debugging) justify cost; routine tasks don't benefit. Business model: pricing reasoning tokens separately, encourage selective usage.

Benchmark Performance

AIME (American Invitational Mathematics Examination): 30 competition math problems, human experts ~55-80% correct. o1: 85-92%, o3: 99%+ (anomalous—possibly overfitting or benchmark contamination). SWE-bench (Software Engineering benchmark): solve real GitHub issues, modify code, run tests. o1: 71.3% accuracy, o3: 96% (claimed), DeepSeek-R1: 96%. GPQA (difficult science Q&A): o1: 92%, o3: 92%+. Limitations: no verified independent evaluation (benchmarks not held out), reasoning quality hard to assess, generalization beyond benchmarks unknown.

Distillation and Efficiency

o1-style reasoning generates expensive reasoning tokens. Distillation: knowledge transfer to smaller models. Marco-o1 (research), attempts to capture reasoning capability in 7B-13B parameter models via data synthesis. Efficiency gain modest: smaller reasoning models still expensive (vs. standard 7B inference). Scalability: not clear if reasoning approach scales to 10T+ token sequences or 10B+ parameter models.