"Let's Verify Step by Step" is a paper by Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever and Karl Cobbe of OpenAI, posted to arXiv as 2305.20050 in May 2023 (Lightman, Kosaraju, Burda and Cobbe are marked as primary authors). It compares two ways of training reward models to judge mathematical solutions, outcome supervision and process supervision, on the MATH dataset, releases the PRM800K dataset of step-level human labels, and in doing so defines the 500-problem MATH test subset that later reporting calls MATH-500.
Summary
The paper starts from the observation that language models solving multi-step problems "regularly produce logical mistakes," and that a reward model used for reinforcement learning or rejection sampling makes the resulting system "only as reliable as the reward model itself." Following Uesato et al. (2022), it distinguishes outcome-supervised reward models (ORMs), trained only on whether a solution's final result is correct, from process-supervised reward models (PRMs), which receive feedback on each step. The authors cite three reasons to prefer process supervision: more precise feedback locating the error; easier human interpretation; and more direct reward for "following a human-endorsed chain-of-thought." Uesato et al. had found the two methods gave similar final performance on grade-school math; this paper differs in using a more capable base model (GPT-4), "significantly more human feedback," and the harder MATH dataset.
Method
A fixed generator, fine-tuned from GPT-4 (plus a further pretraining pass on a roughly 1.5B-token math corpus called MathMix), produces newline-delimited step-by-step solutions. Reward models are evaluated by best-of-N search: for each test problem the highest-scoring sampled solution is auto-graded on its final answer. The generator is not improved with reinforcement learning; the paper is "exclusively" about training the most reliable reward model.
Process labels were collected from human labellers who marked each step positive, negative or neutral. Solutions surfaced for labelling were chosen by active learning: "convincing wrong-answer solutions," meaning those rated highly by the current best PRM but reaching an incorrect final answer, with the PRM retrained between generations. The PRM scores a solution as the product of its per-step correctness probabilities. Process supervision is given only up to the first incorrect step, to keep the comparison with outcome supervision straightforward.
PRM800K's training set contains 800K step-level labels across 75K solutions to 12K problems; Appendix B records 1,085,590 labels over 101,599 samples before filtering. To avoid overfitting the 7,500 MATH training problems, 4,500 of the 5,000 MATH test problems were moved into training, so evaluation uses only the remaining 500, "selected uniformly at random" and shown to be representative of the full test set by difficulty level and subject. Labellers passed a screening test of 30 quality-control questions at 75% agreement with gold labels.
Results
At large scale, the PRM trained on PRM800K solved 78.2% of the 500 test problems by best-of-1,860 selection, against 72.4% for an ORM trained on 100 uniform samples per problem and 69.6% for majority voting; the gap widens as N increases. The authors note the two training sets are not directly comparable (the PRM set is actively selected, biased to wrong answers, and an order of magnitude smaller), so they run a small-scale comparison with models pretrained with roughly 200 times less compute, using the large PRM as a labelling oracle. There, process supervision "significantly outperforms both forms of outcome supervision at all data collection scales," and active learning is estimated to be approximately 2.6x more data-efficient than uniform labelling.
On a held-out set of recent AP Calculus, AP Chemistry, AP Physics, AMC 10 and AMC 12 questions released after the pretraining data was compiled, best-of-100 aggregate accuracy was 72.9% for the PRM, 63.8% for the ORM and 61.3% for majority voting. The text gives the set as 224 questions while Table 1's per-test counts sum to 234.
Discussion
The authors argue process supervision eases credit assignment and has alignment advantages: more interpretable reasoning, direct reward for "an aligned chain-of-thought rather than relying on outcomes as a proxy," and, since it also performed better, "a negative alignment tax." On contamination, they state that MATH test problems "are discussed in several online venues," that string-matching removal from MathMix cannot give "strong guarantees," and that "some degree of contamination has slightly inflated our performance on the MATH test set" is possible, while expecting any effect to apply across methods equally.
Key claims
- 78.2% on a 500-problem MATH test subset with a GPT-4-based PRM (best-of-1,860); ORM 72.4%; majority voting 69.6%.
- The 500-problem subset was chosen uniformly at random after 4,500 test problems were moved into training; it is published at github.com/openai/prm800k.
- PRM800K: about 800,000 step-level labels over 75,000 solutions (1,085,590 raw labels), released publicly.
- Active learning on convincing wrong-answer solutions is about 2.6x more data-efficient than uniform labelling.
- Out-of-distribution STEM set: PRM 72.9%, ORM 63.8%, majority voting 61.3% (best-of-100).
- Stated limitation: possible test-set contamination of MATH; generalisation beyond mathematics untested.
Provenance
Retrieved 22 September 2026 from the arXiv HTML rendering of version 1 (arxiv.org); the abstract page is arxiv.org. Dataset and test subset at github.com.
Relationships
- supports: MATH and MATH-500 — defines the MATH-500 subset and supplies the 2023 process-reward-model results.
- depends-on: Measuring Mathematical Problem Solving With the MATH Dataset (Hendrycks et al., 2021) — the dataset the paper trains and evaluates on.
- depends-on: Training Verifiers to Solve Math Word Problems (Cobbe et al., OpenAI, 2021) — the ORM training method follows that paper's verifier approach.
- related: RLHF (Reinforcement Learning from Human Feedback) — reward-model training for reinforcement learning.
- instance-of: AI Benchmarks and Evaluation