This is the paper by Zirui Wang, Mengzhou Xia, Luxi He, Howard Chen, Yitao Liu, Richard Zhu, Kaiqu Liang, Xindi Wu, Haotian Liu, Sadhika Malladi, Alexis Chevalier, Sanjeev Arora and Danqi Chen, principally of Princeton Language and Intelligence at Princeton University, with co-authors at the University of Wisconsin–Madison and the University of Hong Kong, that introduces the CharXiv benchmark. It was submitted to arXiv as 2406.18521 on 26 June 2024. The paper argues that existing chart-understanding datasets use oversimplified charts and template questions that overstate progress, and offers a benchmark built from real charts in scientific papers.
Summary
Existing benchmarks overestimate chart understanding
The authors show that open-source models which appear to match or beat proprietary models on prior chart benchmarks are sensitive to small perturbations. In a stress test with slightly modified charts or questions, accuracy fell by up to 34.5 points; SPHINX V2 dropped from 63.2% to 28.7% when questions were slightly modified on the same charts. They attribute this to homogeneous, template-based datasets.
Benchmark construction
CharXiv contains 2,323 charts, defined as any figure that visually illustrates data, taken from arXiv preprints in eight academic subjects from January 2020 to September 2023 and re-rendered as high-resolution JPEGs with the longer side at 1,024 pixels. Selection used a four-step pipeline: a pretrained SigLIP encoder flagged candidate figures with cosine similarity of at least 0.65 to the average embedding of MathVista charts; graduate students manually chose charts from batches of ten, favouring visual variety; charts with pairwise similarity of 0.95 or more were removed; and unlabelled or blurry charts were discarded. The charts are described as more compositional than those in existing datasets, often combining several chart types or subplots. A random 1,000 charts form the validation set, whose questions and answers are public; the test set's answers are kept private.
Each chart is paired with four descriptive questions, drawn from 19 templates in five groups (information extraction, enumeration, pattern recognition, counting, compositionality), one of which is intentionally unanswerable, for 9,292 descriptive questions in total, and one manually written reasoning question requiring synthesis across visual elements, categorised by whether the answer is text or a number and whether it appears in the chart or is general. Because answers may include Greek letters and mathematical notation typed in different ways, grading uses GPT-4o to extract the answer and assign a binary score rather than exact match.
Results
On the validation set, human participants scored 80.5% on reasoning questions and 92.1% on descriptive questions; a chart-free random baseline using GPT-4o scored 10.8% and 19.85%. GPT-4o, the best model, scored 47.1% on reasoning, a 33.4-point gap to humans, and 84.45% on descriptive questions, 7.65 points below humans. GPT-4V scored 37.1% and 79.92%; Claude 3 Sonnet 32.2% and 73.65%; Claude 3 Opus 30.2% and 71.55%; Gemini 1.0 Pro 22.8% and 54.37%. The strongest open-source model, InternVL Chat V1.5, scored 29.2% on reasoning and 58.50% on descriptive questions, 25.95 points below GPT-4o on the latter; no other open-source model exceeded 25% on reasoning, and most answered fewer than half of the descriptive questions correctly. Compositionality questions were hardest for open-source models, several of which scored under 6% on them.
Limitations stated by the authors
The authors note that although CharXiv does not require advanced domain knowledge, human accuracy is only 80.5% and 92.1%, which they attribute to possible automated-grading issues or participant mistakes, and that benchmarks curated entirely by human experts are expensive to build and hard to extend, while automatically generated ones overestimate capability.
Key claims
- 2,323 arXiv charts (2020–2023, eight subjects); 1,000-chart public validation split; 9,292 descriptive questions (four per chart, one unanswerable) and one reasoning question per chart.
- Stress test on prior benchmarks: accuracy drops of up to 34.5 points from small perturbations.
- Validation results: humans 80.5% reasoning / 92.1% descriptive; GPT-4o 47.1% / 84.45%; GPT-4V 37.1% / 79.92%; best open-source (InternVL Chat V1.5) 29.2% / 58.50%.
- Grading by GPT-4o answer extraction with binary scoring.
Provenance
Retrieved 22 September 2026 from the arXiv HTML rendering of version 1 (arxiv.org); abstract page arxiv.org. Project page and leaderboard: charxiv.github.io.
Relationships
- supports: CharXiv — the primary text for the benchmark's construction and release-time results.
- instance-of: AI Benchmarks and Evaluation
- related: Multimodality, MMMU