"LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code" is a paper by Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida I. Wang, Armando Solar-Lezama, Koushik Sen and Ion Stoica of UC Berkeley, MIT and Cornell, first posted to arXiv as 2403.07974 on 12 March 2024 and revised as version 2 on 6 June 2024. It introduces LiveCodeBench, a benchmark that collects new competition-programming problems over time and records each problem's release date so that models can be scored only on problems published after their training cutoff. Gu, Li, Yan and Zhang are marked as equal contributors.
Summary
Argument
The authors argue that HumanEval, MBPP and APPS "may paint a skewed or misleading picture" for two reasons: they measure only natural-language-to-code generation, and "benchmark samples are present in the training datasets," so results may reflect contamination or overfitting. LiveCodeBench is built on live updates to prevent contamination, a broader set of code-related scenarios, and problems vetted by the thousands of contest participants who solved them.
Scenarios and curation
Four scenarios are defined, chosen for usefulness in code workflows and automatic scoring. Code generation gives the model a problem statement with example input-output pairs and scores pass@1 on hidden tests. Self-repair adds one round of error feedback (an exception or a failing test) and scores the final program. Code execution, following CRUXEval, asks the model to predict a function's output on a given input. Test output prediction, a task the paper introduces, gives the problem statement and a test input and asks for the expected output, isolating test-generation reasoning without requiring the model to invent inputs.
Problems are scraped from LeetCode weekly and biweekly contests, AtCoder beginner (abc) contests, and Codeforces Division 3 and 4 contests held after April 2023, excluding problems with images or with multiple correct answers. Each problem is stored with its statement, tests, a ground-truth solution and the contest date, which allows "scrolling" evaluation over any time window. Platform tests are used where available; otherwise GPT-4-Turbo is used to build input generators from the problem specification, and some failing tests are collected from platform graders. Because competition programming is hard for models (the paper cites GPT-4's Codeforces rating of 392, in the bottom 5 percent), problems above a rating threshold are excluded and the rest are tagged Easy, Medium or Hard from platform labels: AtCoder ratings of [0, 200), [200, 400) and [400, 500], and Codeforces ratings of 800, (800, 1000] and (1000, 1300]. Codeforces problems remained harder even after filtering, so only nine were included. As of the May 2024 snapshot the code-generation set held 511 problems (182 easy, 206 medium, 123 hard; 267 from AtCoder, 235 from LeetCode) with 17.0 tests per problem on average; the post-September-2023 subset used for most comparisons held 349. The code-execution set contains 479 samples from 85 LeetCode problems, and the test-output-prediction set 442 instances from 181 LeetCode problems.
Contamination findings
Plotting performance by problem release month, the authors observe that DeepSeek-Instruct-33B's LeetCode scores fall sharply after August 2023, just before its release, and that GPT-4o's fall after November 2023, its stated cutoff; DeepSeek-Base-33B drops from about 60 pass@1 on May 2023 LeetCode problems to about 0 on September problems, which the authors take as evidence that competition problems were in DeepSeek's pretraining data. The drop appears only for LeetCode problems, not AtCoder. GPT-4-Turbo, Gemini-Pro, Mistral-Large and Claude 3 models showed no comparable variation across months. Codestral scored 36.5 on problems from May 2023 to January 2024 and 28.3 on problems from February 2024. All subsequent comparisons use problems released from September 2023.
Model comparisons
The paper evaluates 34 instruction-tuned models and 18 base models (52 in all, from 1.3B to 70B parameters), generating 10 candidates per problem with nucleus sampling at temperature 0.2 and top-p 0.95 and reporting the fraction that pass as pass@1. Appendix Table 3 gives code-generation totals on the post-September-2023 window of 41.90 for GPT-4o (2024-05-13; 88.30 easy, 33.20 medium, 4.20 hard), 41.10 for GPT-4-Turbo (2024-04-09), 36.30 for GPT-4-Turbo (1106), 33.90 for GPT-4 (0613), 33.00 for Gemini 1.5 Pro (May), 32.80 for Claude 3 Opus, 27.80 for Gemini 1.5 Flash (May), 25.00 for Claude 3 Sonnet, 22.80 for Claude 3 Haiku and 22.30 for Claude 2. Pass@1 correlations across the four scenarios exceed 0.88 for every pair (0.98 between generation and self-repair, 0.96 between test output prediction and code execution, 0.89 between generation and execution), but relative gaps shift: GPT-4-Turbo widened its lead in self-repair, rising from 24.5% to 36.9% on medium problems where Gemini-Pro rose only from 8.5% to 9.4%, and Claude 3 Opus beat GPT-4-Turbo on test output prediction. Comparing HumanEval+ with the easy subset of LiveCodeBench, the authors find a correlation of only 0.72 and two clusters of models: base and closed models score similarly on both, while many fine-tuned open models score well on HumanEval+ but poorly on LiveCodeBench. DeepSeek-Instruct-1.3B, for instance, scores 59.8% on HumanEval+ and 26.3% on LCB-Easy, and DeepSeek-Instruct-33B trails GPT-4-Turbo by 4.3 points on HumanEval+ but 16.2 points (69%) on LiveCodeBench code generation, with gaps of 96% and 134% on test output prediction and code execution with chain of thought. The authors describe HumanEval as "an easier benchmark with small and isolated programming problems and thus easier to overfit on." GPT-4-Turbo's generations contained 19.5 times more comment tokens than GPT-4's. Post-training raised Llama-3-Instruct-70B, DeepSeek-Instruct-33B and Phind-34B to 28.3, 23.6 and 21 pass@1, gains of 8.2, 7.3 and 9.5 points over their base models; only those three open models (with Mixtral) reached closed-model levels.
Key claims
- Problems are dated by contest so that evaluation can be restricted to a window after a model's cutoff; the code-generation set held 511 problems from May 2023 to May 2024 (182 easy, 206 medium, 123 hard). (high)
- DeepSeek models and GPT-4o show sharp drops on LeetCode problems released after their training dates, which the authors read as contamination; the effect does not appear on AtCoder problems. (high)
- 18 base and 34 instruction-tuned models were evaluated across four scenarios; cross-scenario pass@1 correlations exceed 0.88. (high)
- Code-generation pass@1 (10 samples, temperature 0.2, problems from September 2023): GPT-4o 41.90 (4.20 on hard), GPT-4-Turbo (April 2024) 41.10, Gemini 1.5 Pro 33.00, Claude 3 Opus 32.80, Claude 2 22.30. (high)
- HumanEval+ and LCB-Easy correlate at 0.72; fine-tuned open models cluster as strong on HumanEval+ but weak on LiveCodeBench (DeepSeek-Instruct-1.3B: 59.8% vs 26.3%). (high)
- Tests come from the platforms where available and otherwise from GPT-4-Turbo-built input generators. (high)
Provenance
Retrieved 22 September 2026 from the arXiv HTML rendering of version 2 (arxiv.org, dated 6 June 2024); the abstract page is arxiv.org. The benchmark site is livecodebench.github.io.
Relationships
- supports: LiveCodeBench — the primary text for the benchmark's design, difficulty tiers and launch results.
- supports: HumanEval — the source of the overfitting comparison against HumanEval+.
- depends-on: Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation (Liu et al., 2023) — uses HumanEval+ as the comparison set and its test-generation approach as a starting point.
- depends-on: Evaluating Large Language Models Trained on Code (Chen et al., OpenAI, 2021) — pass@1 metric.
- supports: AI Benchmarks and Evaluation — time-windowed evaluation as a contamination control.