MarketBench is a working paper, published 2026-04-27, by Andrey Fradkin (Boston University and the MIT Initiative on the Digital Economy) and Rohit Krishnan (independent researcher). It introduces a benchmark testing whether frontier large language models can self-assess their probability of task success and their token cost well enough to participate in market-style coordination among AI agents. Across 93 SWE-bench Lite tasks evaluated on six frontier models, the paper reports that models are miscalibrated on both axes and that auctions built from their self-reports diverge significantly from a full-information allocation. The paper has not been peer-reviewed.
Argument
The paper distinguishes most existing AI-agent benchmarks, which ask whether a model finishes a task, from the requirements of market-style coordination. Under such coordination, agents must name a price and signal their probability of success before a task is assigned, so allocation turns on self-reported odds and expected cost rather than realized success. The authors frame MarketBench as a benchmark designed to evaluate this self-assessment capability, and report that the frontier models tested perform poorly at it.
Setup
The benchmark draws 93 tasks from SWE-bench Lite, a set of real GitHub issues from roughly 12 mature open-source Python repositories (django, sympy, scikit-learn, sphinx, and others). Six recently released LLMs are evaluated, producing 558 model-task rows (six × 93) in Phase I.
Two task families are defined:
- MarketBench Calibration — agents forecast (a) their probability of one-attempt success and (b) their total token usage. Forecasts are compared against realized model runs, with tokens mapped to dollar costs via model-specific blended pricing.
- MarketBench Auction — elicited success and token estimates are used as bids in a reserve-price procurement simulation, and payoffs are measured against a full-information allocation benchmark.
Conceptual framework
Section 3 presents a stylized two-agent model (a high-cost agent H and a low-cost agent L) formalizing when markets dominate alternative routing rules. Proposition 1 states that, given $v > c_H > c_L > 0$, the market allocation weakly dominates every non-market alternative (assign-only-to-H, assign-only-to-L, and parallel execution). Dominance is strict whenever there is positive probability of a state in which the alternative allocates to the wrong agent, pays a redundant agent, or pays for an attempt when no agent can solve the task. The authors describe markets as preserving the good states, in which each agent works when it should, while correcting the bad ones, avoiding waste when no agent can solve the task and avoiding redundant cost when both can.
A practical condition follows: agents do not need to perfectly observe their idiosyncratic draw $\epsilon_i$, but only need a signal more informative than the principal's signal. If self-assessment is merely a noisy restatement of what the principal already knows, the paper argues, bidding adds little.
Findings
The paper's empirical results (Section 4) are that calibration is poor: models systematically misestimate both their own success probability and their token consumption on the SWE-bench Lite tasks. These errors carry into auctions — when the elicited self-reports are mechanically converted into bids, allocation outcomes diverge significantly from the full-information benchmark. A follow-up intervention prepends a "self-knowledge card" containing held-out historical success rates from prior experiments; this narrows the gap to the full-information benchmark but does not close it.
The authors characterize self-assessment as a missing capability, arguing that calibration, abstention, and budget discipline in bidding are core capabilities for decentralized coordination that the frontier models tested lack. In the paper's discussion, the authors contend that a scalar bid is the wrong object: because pass-rate gains at the frontier come from spending more compute per task, the appropriate object is not a single scalar bid for one expected cost but a production schedule mapping token budget to expected success probability and wall-clock time. The paper sets out an agenda for future work in three parts: making self-assessment a target of training and evaluation in its own right; building richer market institutions such as reputation and escrow; and extending the benchmark beyond software engineering.
Reception and standing
The authors describe the negative result as well-supported: multiple frontier models were tested on a real benchmark, and the findings replicate prior LLM confidence-elicitation work (Xiong et al. 2024; Geng et al. 2024; Zhang et al. 2024) while extending it from answer-correctness to agentic production. The paper is a working paper and has not been peer-reviewed. Whether richer prompting or explicit metacognitive training can substantially close the calibration gap remains unresolved; the paper's intervention experiment suggests modest improvement but no larger change.
Relationships
- depends-on: The Coasean Singularity? Demand, Supply, and Market Design with AI Agents — Shahidi, Rusak, Manning, Fradkin, Horton (2025) — same Fradkin co-author; broader theoretical framing of AI agents as market participants
- supports: Why Coase needs Hayek — Rohit Krishnan (Strange Loop Canon, May 2026) — companion essay applying these ideas to a hub-spoke vs. market head-to-head
- related: Agent Architecture Patterns
- related: Principal-Agent Problem Applied to AI
- related: Agentic AI
- related: AI Coding Agents — uses SWE-bench Lite as the empirical task base
Tracked claims
- Frontier LLMs are miscalibrated on both success probability and token usage on SWE-bench Lite — confidence high.
- The "self-knowledge card" intervention modestly improves calibration but does not close the gap — confidence high.
- Markets weakly dominate non-market routing under the paper's standard assumptions — confidence high (formal proof in paper).
- Current LLMs cannot exploit market dominance because of self-assessment failure — confidence high.