"Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation" is a paper by Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang and Lingming Zhang of the University of Illinois Urbana-Champaign and Nanjing University, first posted to arXiv as 2305.01210 on 2 May 2023. It introduces EvalPlus, a framework that augments code-synthesis benchmarks with automatically generated test inputs, and applies it to HumanEval to produce HumanEval+. The authors mark all four as equal contributors, with author order decided by a game of nigiri.
Summary
Argument
The paper argues that existing code benchmarks judge functional correctness with too few and too simple tests, so that logically wrong programs pass and reported scores over-approximate correctness. It identifies two limitations: insufficient testing, with benchmarks averaging fewer than 10 tests per problem, and imprecise problem descriptions, where a docstring omits the expected input domain or error handling so that a capable model can be judged wrong. Figure 1 shows a ChatGPT solution to HumanEval problem 58 that passes the base tests but converts a sorted list to a set, losing the required order. The authors state that theirs is the first study of test inadequacy in these benchmarks.
Approach
EvalPlus takes a benchmark's ground-truth solutions and base test inputs and generates new inputs in two stages. ChatGPT is prompted with the ground-truth implementation, sample inputs and an instruction to produce "interesting" seed inputs that respect the problem's input format; the authors say this handles semantic constraints, such as an input that must be a palindrome, that traditional generators cannot. Because querying a large model is slow and costly, the seeds are then expanded by type-aware mutation, a fuzzing loop that applies type-specific edits (incrementing numbers, removing or repeating list items or substrings, recursively mutating dictionary entries) and re-uses observed data fragments. Generated code is checked by differential testing against the ground truth. Because docstrings can be vague, the authors annotate function pre-conditions as assertions ("program contracts") to filter invalid inputs; 83 of the 164 HumanEval tasks carry hand-written contracts. An optional test-suite reduction step uses greedy set covering over branch coverage, mutant kills and empirical "sample kills" (wrong LLM samples a test catches) to build a smaller suite of equivalent effectiveness.
HumanEval+ and results
For each task, around 30 ChatGPT seeds from three prompts were mutated for one hour to produce 1,000 additional inputs. HumanEval+ raises the average number of tests per task from 9.6 to 764.1 (Table 2; the caption of Table 4 gives 774.8), which the paper describes as an 80-fold increase, and HumanEval+-mini reduces this to 16.1 tests per task, 47 times fewer, while preserving nearly the same pass@1 drop under leave-one-out cross-validation. Inspecting the original ground truths, the authors found more than 10% incorrectly implemented and re-implemented all of them; the defects number 18 (11% of problems): five unhandled edge cases, ten cases of bad logic and three performance problems. Problem 124 (validate_date) is given as an example, where operator precedence made the reference accept either of two conditions instead of both, so that 12-31-1999 was marked invalid, an error none of the base tests exposed.
The evaluation covers 26 models using the unbiased pass@k estimator of the Codex paper, with 200 samples at each of four temperatures (0.2, 0.4, 0.6, 0.8) plus greedy decoding; GPT-4 was evaluated only under greedy decoding and ChatGPT additionally at temperature 0.8. On HumanEval+, almost all pass@k results drop, by up to 23.1% (greedy pass@1), 19.3% (pass@1), 24.9% (pass@10) and 28.9% (pass@100). Greedy pass@1 fell from 88.4% to 76.2% for GPT-4 (a 13.1% reduction), from 73.2% to 63.4% for ChatGPT (12.6%), from 32.9% to 26.8% for CodeGen-16B (18.5%), from 51.8% to 42.7% for CodeLlama-34B (17.6%) and from 34.1% to 29.3% for StarCoder (14.1%). Rankings changed: WizardCoder-CodeLlama-34B (73.2% base, 64.6% plus) and Phind-CodeLlama-34B (71.3%, 67.1%) were no better than ChatGPT on HumanEval but ahead of it on HumanEval+, and SantaCoder-1.1B overtook InCoder-6.7B while Vicuna-7B overtook InCoder-1.3B. Optimal temperatures stayed largely unchanged, with lower temperatures best for small k and higher for large k. The hardest problems involved multiple conditions, completeness (handling negative numbers in a primality check), reasoning (the Tribonacci sequence) and efficiency (the n-th prime Fibonacci number).
The paper closes by reporting that the EvalPlus PyPI package had been installed more than 6,000 times in five months, that a leaderboard is maintained at evalplus.github.io, and that the authors plan to extend the method to MBPP and to combine it with formal verification.
Key claims
- HumanEval averages 9.6 tests per task (median 7); HumanEval+ averages 764.1 (median 982.5, range 12–1,100); HumanEval+-mini averages 16.1. (high)
- 18 of 164 HumanEval reference solutions (11%) are defective: 5 edge-case failures, 10 logic errors, 3 performance problems. (high)
- Across 26 models, pass@k on HumanEval+ is lower by up to 19.3% (k = 1), 24.9% (k = 10) and 28.9% (k = 100); greedy pass@1 by up to 23.1%. (high)
- GPT-4 greedy pass@1: 88.4% base, 76.2% plus; ChatGPT: 73.2% base, 63.4% plus. (high)
- Two open models, WizardCoder-CodeLlama and Phind-CodeLlama, rank above ChatGPT on HumanEval+ but not on HumanEval. (high)
- Test generation uses ChatGPT-produced seeds expanded by type-aware mutation, with differential testing against re-implemented ground truths. (high)
Provenance
Retrieved 22 September 2026 from the arXiv HTML rendering (arxiv.org), which corresponds to version 3 of the preprint; the abstract page is arxiv.org. Tools, datasets and generated code are at github.com. The work was partially supported by NSF grants CCF-2131943 and CCF-2141474 and by Kwai Inc.
Relationships
- supports: HumanEval — the primary text for the HumanEval+ variant and the reference-solution defect count.
- depends-on: Evaluating Large Language Models Trained on Code (Chen et al., OpenAI, 2021) — extends the dataset and re-uses the unbiased pass@k estimator that paper introduced.
- supports: AI Benchmarks and Evaluation — evidence that test adequacy changes both scores and rankings.
- instance-of: AI Benchmarks and Evaluation