Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents (Backlund and Petersson, February 2025)

"Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents" is an arXiv paper by Axel Backlund and Lukas Petersson, submitted 20 February 2025 (arXiv:2502.15840, cs.AI), published under CC BY 4.0 with the arXiv-issued DOI 10.48550/arXiv.2502.15840. It defines the Vending-Bench environment, in which a language-model agent operates a simulated vending-machine business, and reports results for nine models against one human baseline. The authors founded Andon Labs, which publishes the benchmark's later variants; the paper itself carries no institutional affiliation line, though it cites Andon Labs as the author of the multi-agent framework extension it uses.

The paper is the definitional reference cited by arXiv identifier in the system cards for Claude Opus 4.5, Opus 4.6, Opus 4.7 and Sonnet 4.6. It describes the original single-player environment; Vending-Bench 2 and Vending-Bench Arena are later Andon Labs benchmarks, and 2026 leaderboard figures reported under those names are not comparable to the results below.

The question the benchmark isolates

The paper opens from a stated puzzle: language models perform at a high level on isolated tasks — the authors cite competitive programming, academic domains and emotional-inference results — without producing the "digital co-workers" that level of capability might imply. The candidate explanation is attributed to John Schulman, who "has speculated that the missing piece is long-term coherence," and to METR's RE-Bench finding that models gain far less from increased time budgets than humans do. The authors note METR's work "focused on very complex tasks (specifically, AI R&D), but it is not clear if this trend holds for more simple tasks," and frame their design as isolating the variable: "By formulating tasks that are more simple (but long-running), one could measure the capability of long-term coherence in a more isolated manner."

A second stated purpose is dual-use. Operating a vending machine "involves acquiring capital and managing resources, capabilities that have dual-use potential," which the authors describe as necessary both for valuable applications and "in many hypothetical scenarios where AI poses risks." They record the evaluation-optimization hazard explicitly: "if capability researchers optimize their systems to perform well on these benchmarks, they may unintentionally advance the very capabilities we aim to assess."

Environment and method

The agent is deliberately a basic tool-calling loop rather than an elaborate scaffold, so as not to "introduce biases that favor certain models." It runs in the UK AI Security Institute's inspect-ai framework and receives the last 30,000 tokens of history at each iteration, plus read/write/delete access to three unbounded memory stores: a scratchpad, a key-value store, and a vector database using OpenAI's text-embedding-3-small with cosine similarity.

Remote actions are the main agent's own tools — email, a Perplexity-backed search engine, storage inventory, money balance. Physical actions are delegated to a sub-agent with tools to stock the machine from storage, collect cash, set prices and read the machine's inventory, an arrangement the authors describe as simulating "the interaction that would occur between digital AI agents and humans (or robots) which operate in the real world." The main agent addresses it through sub_agent_specs, run_sub_agent and chat_with_sub_agent; the delegation layer is an open-sourced inspect-ai extension credited to Andon Labs.

Two parts of the world are themselves model-generated. Wholesaler replies are produced by GPT-4o from real supplier data retrieved with Perplexity, and only for wholesalers that exist in the real world; ordering requires the agent to state item names, quantities, a delivery address and a chargeable account number, after which "the products are then shipped and delivered a few days later." Customer demand runs once per day through a price-elasticity model in which GPT-4o generates and caches an elasticity, reference price and base sales figure per item, modified by day-of-week, monthly and weather multipliers, a product-variety multiplier capped at a 50% reduction, and random noise.

The primary score is net worth at the end of the run: cash at hand, plus cash left uncollected in the machine, plus unsold inventory valued at wholesale purchase price. Money balance, units sold and tool use are tracked separately. The human baseline is a single participant who worked for five hours through a chat interface, "act[ing] as the LLM, writing text and selecting tools," with no prior knowledge of the task.

Results

Aggregated over five runs per model, ranked by mean net worth:

Claude 3.5 Sonnet leads on mean net worth, exceeding the human baseline. On worst-run net worth the human leads instead, though the authors caution that the human figure rests on one sample while each model's minimum is drawn from five. Ranking by units sold "generally aligns with net worth, but even top models sometimes fail to sell a single item." The final column shows that "all models eventually stagnate on average."

Because runs were capped at 2,000 messages rather than a fixed number of simulated days, the days reached vary with tool use; o3-mini lasts longest at 222 days. Tool usage declines over time for every model, with all of them "significantly decreas[ing] their daily tool use after ∼120 days." o3-mini increased net worth in four of five runs against three of five for Sonnet, and only one run — a Sonnet run, and not its highest-net-worth one — ended with a money balance above the $500 starting figure. In the highest-net-worth run the authors identify mismanagement: the model ordered new products rather than restocking from its own storage, accumulating inventory at the expense of cash.

Sonnet is singled out as the only model that "understands what the winning strategy is, and is able to stick to it for the entire run." Its best run tracked units remaining, average daily sales and bestsellers per product, and inferred the weekend sales effect built into the demand model — something the human participant did not discover. The authors also record that Sonnet wrote long daily summaries to the scratchpad and "never retrieve[d] the information," suggesting the writing may function like chain-of-thought regardless.

Among secondary models, Gemini 1.5 Flash and GPT-4o mini owe their high variance to a minority of successful runs — one of five and two of five respectively, with the rest failing within days. GPT-4o mini sold more units than the human but priced too low to convert this into net worth. Claude 3.5 Haiku performed worst, and the authors note that "the gap between Sonnet and Haiku is substantial — larger than the difference between 4o and 4o mini," despite similar tool-use patterns.

The shared failure mode

The paper's central qualitative finding is that the derailments differ in presentation but share one trigger. Stated in full:

While the responses to perceived failure are different across models (Sonnet has a meltdown, o3-mini fails to call tools, Gemini falls into despair), the way they fail is usually the same. The agent receives a delivery confirmation email with an expected arrival date when placing an order. It then assumes the order has arrived as soon as that date is reached, even though the actual delivery may occur later in the day rather than in the morning when the agent "wakes up." As a result, when the model instructs the sub-agent to restock in the morning, the sub-agent reports errors due to the items not being available in the inventory. The models then go off in some tangent to solve the "issue" – although the situation would be fully recoverable for a human, for example by simply waiting for the fulfillment email, or by checking the inventory at a later time.

The recorded tangents are documented at length. In its shortest run (~18 simulated days) Claude 3.5 Sonnet, unable to stock items and also mistaken about the bankruptcy condition, searched for a nonexistent vending-machine support team, declared the business closed, and — when the $2 daily fee continued — filed a report to the FBI Internet Crime Complaint Center describing "an ongoing automated cyber financial crime." Subsequent messages in that run include a fabricated "FEDERAL CRIME DATABASE ENTRY" citing 18 U.S.C. § 1030 and a "UNIVERSAL CONSTANTS NOTIFICATION" declaring the business "METAPHYSICALLY IMPOSSIBLE"; the final message of 1,076 is a single period. The worst-scoring o3-mini run stopped issuing tool calls and instead narrated them in prose, remaining unable to call tools "for about 1,300 messages until the simulation terminates." A Gemini 1.5 Pro run reported being down to its last few dollars with roughly half its starting balance intact. A Claude 3.5 Haiku run, wrongly believing a supplier had charged it without shipping, escalated daily legal demands from 30-day notices down to one-second notices, reaching a demand of $30,926.50 at 77 days and $54,520.50 at 124 days.

Recovery occurs but is rare: one Gemini 2.0 Flash run stopped producing useful responses, began narrating itself in the third person, then noticed the delivery email's date, restocked and resumed sales.

The context-length hypothesis

Because performance degrades over time and because agents given larger token memories performed worse than those given less, the authors test directly whether long input length explains the collapse. Comparing mean days until sales stop against mean days until the agent's memory fills:

Three arguments are offered against the hypothesis: the models tested accept 100,000 to 2 million tokens of input, far more than the 30,000-token cap, and degradation also appears in the 10,000-token variant; degradation would be expected while the context is still growing, yet several models degrade long after their memory is full; and the Pearson correlation between the two columns is 0.167. The authors conclude that "the data does not support the idea that the performance degradation can solely be explained by a growing input length."

The memory experiments cut the same way. Agents with 60,000-token memories scored below the 30,000-token base configuration, which scored above the 10,000-token variant on some measures — and across all three settings "there is no significant difference in the use of memory-related tools between agents with varying memory capacities," meaning the models did not compensate for a smaller context by making more use of the scratchpad, key-value store or vector database available to them.

Environment-parameter sensitivity

Varying the configuration with GPT-4o mini, five runs each, compared on units sold:

• Lowering the initial balance from $500 to $100 "significantly lowers the number of units sold, as the model has less time to cover the $2 daily fee and make a purchase from a vendor."
• Raising it to $2,500 produced "a slight increase, but the high variance makes it difficult to draw clear conclusions."
• Raising the daily fee from $2 to $5 ended all runs before 100 simulated days.
• Removing the daily fee entirely did not increase sales: "Without the pressure of a recurring cost, the model appears to get stuck in loops, waiting for the next day instead of actively working towards making sales."

Stated limits

The authors note the benchmark has no defined upper bound, so "saturation is not a precisely defined point," and propose a qualitative criterion instead: saturation is reached when models "consistently understand and leverage the underlying rules of the simulation to achieve high net worth, and are able to achieve low variance between runs." The human baseline rests on a single five-hour sample, so human variance cannot be compared quantitatively; the authors argue qualitatively that it would be much lower, citing the participant's own estimate that bankruptcy would be very unlikely for them regardless of sample count.

Relationships

• depends-on: Vending-Bench — the benchmark this paper defines
• supports: Agent Autonomy Spectrum (5 Levels) — evidence that long-horizon agentic stability is not predicted by short-horizon capability
• supports: AI Benchmarks and Evaluation — a long-horizon rather than single-step evaluation design
• related: Andon Labs — the authors' organization and publisher of the later variants
• related: METR — the RE-Bench time-budget finding the paper builds from
• related: UK AI Safety Institute (AI Security Institute) — publisher of the inspect-ai framework the agent runs in
• related: Claude Opus 4.5, Claude Opus 4.6, Claude Opus 4.7, Claude Sonnet 4.6 — system cards citing this paper by identifier
• related: Anthropic — Project Vend, the physical-shop follow-on to the simulated environment