"Evaluating Large Language Models Trained on Code" is the paper in which OpenAI introduced Codex, a GPT language model fine-tuned on public GitHub code, together with the HumanEval evaluation set and the pass@k metric used to score it. It was posted to arXiv as 2107.03374 on 7 July 2021 and lists 58 authors, led by Mark Chen, Jerry Tworek, Heewoo Jun and Qiming Yuan, all at OpenAI at the time; several (Jared Kaplan, Nicholas Joseph, Dario Amodei, Sam McCandlish) are listed with Anthropic affiliations for work performed while at OpenAI. The paper states that a distinct production version of Codex powers GitHub Copilot.
Summary
The paper's stated task is generating standalone Python functions from docstrings and checking the samples automatically with unit tests, in contrast to natural-language generation, where outputs are judged by heuristics or human raters. Section 2 sets out the evaluation framework, Sections 3 and 4 the fine-tuning of Codex and the further-tuned Codex-S, Section 5 a docstring-generation model, Section 6 the model's limitations, and Section 7 a hazard analysis covering over-reliance, misalignment, bias, labour markets, security, environmental and legal questions.
Functional correctness and pass@k
The authors argue that match-based metrics such as BLEU cannot account for the large space of programs functionally equivalent to a reference solution, and that functional correctness, a sample passing a set of unit tests, is the standard human developers use. They adopt the pass@k metric of Kulal et al. (2019), in which a problem counts as solved if any of k samples passes the tests, but replace the direct calculation, which they say has high variance, with an unbiased estimator: n ≥ k samples are generated per task (n = 200, k ≤ 100 in the paper), the number c of correct samples is counted, and pass@k is the expectation over problems of 1 − C(n−c, k)/C(n, k). A numerically stable numpy implementation is given in Figure 3, and Appendix A shows that the naive estimate 1 − (1 − p̂)^k is biased. Later in the paper they plot BLEU-score distributions for correct and incorrect Codex-12B samples and find them overlapping, concluding that improvements in BLEU "may not indicate improved rates of functional correctness in practice."
HumanEval
HumanEval is a set of 164 hand-written programming problems, each with a function signature, docstring, body and several unit tests, averaging 7.7 tests per problem. The problems "assess language comprehension, reasoning, algorithms, and simple mathematics," with some described as comparable to simple software interview questions. The authors say hand-writing was necessary because the models are trained on a large fraction of GitHub, which already contains solutions to problems from many sources; they note more than ten public repositories holding solutions to the Codeforces problems that form part of the APPS dataset. The dataset and evaluation framework were released at github.com/openai/human-eval. Generated programs are executed in a sandbox built on the gVisor container runtime, with eBPF-based firewall rules blocking inbound and outbound connections except those needed for experiment control.
Training and results
Codex was produced by fine-tuning GPT models of up to 12B parameters on 159 GB of unique Python files (after filtering from 179 GB) collected in May 2020 from 54 million public GitHub repositories, training for 100 billion tokens. Starting from a pre-trained GPT did not improve final performance, which the authors attribute to the size of the fine-tuning set, but did speed convergence. Extra tokens for whitespace runs let code be represented with about 30% fewer tokens. Prompts are assembled from a header, signature and docstring, and sampling uses nucleus sampling with top p = 0.95.
With a single sample per problem, Codex-12B solves 28.8% of HumanEval problems and Codex-300M 13.2%; GPT-J-6B reaches 11.4% (11.62% in Table 1) and GPT models near 0%. Table 1 reports Codex-12B at 28.81% pass@1, 46.81% pass@10 and 72.31% pass@100; GPT-Neo-2.7B at 6.41% / 11.27% / 21.37%; GPT-J-6B at 11.62% / 15.74% / 27.74%; and TabNine's largest free model at 2.58% / 4.35% / 7.59%. GPT-Neo-2.7B is described as roughly equivalent to Codex-85M and GPT-J-6B to Codex-300M. Test loss after code fine-tuning follows a power law in non-embedding parameter count, as for language modelling. Sampling temperature must be tuned to k: for a 679M model the optimal temperature is 0.2 for pass@1 and 0.8 for pass@100, because diverse samples help only when many are drawn.
Codex-S, further fine-tuned on about 10,000 problems from competitive-programming sites and about 40,000 traced from continuous-integration test runs (filtered by keeping only problems Codex-12B could solve in 100 samples), solves 37.7% with one sample and beats Codex by an average of 6.5 percentage points on pass@1 and 15.1 points on pass@100 across model sizes. Because pass@k assumes an oracle that knows the tests, the paper also studies selecting one of many samples without tests: choosing the sample with the highest mean token log-probability lets Codex-S-12B solve 44.5% of problems, against 77.5% when the passing sample is selected by the unit tests (Figure 1, temperature 0.8). On the APPS dataset, Codex-12B evaluated one-shot performs comparably to a GPT-Neo model fine-tuned on APPS.
A reverse model, Codex-D, trained to write docstrings from code, was hand-graded on 10 samples per problem; Codex-D-12B reached 20.3% pass@1 and 46.5% pass@10 against 32.2% and 59.5% for Codex-S-12B at the same temperature.
Limitations and hazard analysis
The authors state that Codex is not sample-efficient, having seen "hundreds of millions of lines of code," and that "a strong student who completes an introductory computer science course is expected to be able to solve a larger fraction of problems than Codex-12B." On synthetic docstrings built from 13 string-manipulation building blocks, the pass rate falls by roughly a factor of two to three with each added chained operation, and the model makes errors binding operations to variables. The hazard analysis reports that Codex tends to produce worse code when the prompt contains subtle bugs, a gap that grows with model size and which the paper calls an alignment failure; that it can emit biased comments and code structure; that it may suggest insecure code; that generated code identical to training data occurred in under 0.1% of a cited study; and that the authors do not believe the models "materially lower the barrier to entry for malware development" at their current capability level.
Key claims
- HumanEval: 164 hand-written Python problems, 7.7 unit tests per problem on average, released with an evaluation harness. (high)
- pass@k is estimated without bias from n = 200 samples per problem using 1 − C(n−c, k)/C(n, k). (high)
- Codex-12B: 28.8% pass@1 (28.81% in Table 1), 46.81% pass@10, 72.31% pass@100; Codex-S-12B: 37.7% pass@1, 77.5% of problems solved within 100 samples. (high)
- GPT-3 models score near 0%; GPT-J-6B 11.4% (abstract) or 11.62% (Table 1); GPT-Neo-2.7B 6.41%. (high)
- Training data: 159 GB of Python from 54 million public GitHub repositories, collected May 2020. (high)
- Optimal sampling temperature for a 679M model: 0.2 for pass@1, 0.8 for pass@100. (high)
- Pass rate on synthetic chained docstrings falls by a factor of 2–3 per added operation. (high)
Provenance
Retrieved 22 September 2026 from the arXiv HTML rendering of version 2 (arxiv.org), whose abstract page is arxiv.org. The keywords field marks the paper for ICML. The HumanEval dataset and harness are at github.com.
Relationships
- supports: HumanEval — the primary text defining the dataset, the metric and the launch results.
- supports: AI Benchmarks and Evaluation — source of the pass@k definition and the finding that sampling budget changes scores.
- supports: Benchmark Performance Over Time — the 2021 HumanEval baseline.
- related: OpenAI — the developer of Codex and maintainer of the dataset.
- instance-of: AI Benchmarks and Evaluation