This is the paper by Kiran Vodrahalli (corresponding author) and 23 co-authors, 22 at Google DeepMind and two at Google Research, that introduces the Michelangelo evaluations and the Latent Structure Queries framework for long-context testing. It was submitted to arXiv as 2409.12640 on 19 September 2024; the captured text is version 2 of 20 September 2024. The paper is the published description of the MRCR task. Its title comes from a remark attributed to Michelangelo about chiselling away superfluous marble, the metaphor the authors use for a model discarding irrelevant context to reveal a latent structure.
Summary
Framework
The authors describe Michelangelo as "a minimal, synthetic, and unleaked long-context reasoning evaluation" that is easy to score automatically. Under the Latent Structure Queries (LSQ) framework, a task presents a long context in which relevant operations build up a latent structure amid irrelevant filler, then queries the model for a view of that structure. Task complexity, the number of relevant operations, is held fixed while context length grows, so any degradation reflects long-context processing rather than harder reasoning. The framework is presented as a way to build evaluations that can be extended to arbitrary length and complexity without leaking existing datasets.
Three tasks
- Latent List presents a short Python list and a sequence of operations (append, insert, pop, remove, sort, reverse) interleaved with no-op filler such as print statements, even numbers of reverses and self-cancelling blocks. The model must output a view of the resulting list (a slice, sum, min, max or length). Scoring uses exact match for printed slices and a normalised approximate error for numerical outputs.
- Multi-Round Co-reference Resolution (MRCR), previously introduced in the Gemini 1.5 report, presents a long user-model conversation in which the user requests pieces of writing on various topics and formats, with the model turns generated by prompting PaLM 2. The model must reproduce the response to one specific request, for example the second poem about penguins, where other requests share the topic, the format or both, so that keys are adversarially similar. Scoring uses an approximate edit-distance metric. The appendix estimates chance rates of 4% for a random model that outputs one of the pieces of writing uniformly and 9% for one that outputs a piece matching the topic or format.
- IDK presents an invented story followed by a large amount of random-letter filler and a four-option multiple-choice question whose correct answer may be "(D) I don't know" because the information was never given. A textual statement that the answer is not in the context also counts as correct when appropriate. Chance is 25%.
Experiments
Ten models were evaluated: Gemini 1.5 Flash and Pro (05-14 and 08-27 versions), GPT-4 Turbo (04-09), GPT-4o, Claude 3 Haiku, Sonnet and Opus, and Claude 3.5 Sonnet, up to 128K tokens, with the Gemini 1.5 models also run to 1M tokens. Results are presented as curves of cumulative average score against context length rather than as a table. Rank orderings differed across tasks: Gemini models performed best on MRCR at 128K, GPT models on Latent List, and Claude 3.5 Sonnet on IDK with Gemini 1.5 Pro close behind. Claude 3 Haiku, Claude 3 Sonnet and Claude 3.5 Sonnet were excluded from the Latent List plots because refusals made their scores very low.
The authors report a common pattern: a sharp super-linear drop in performance early in the context, often by 32K tokens, after which performance either flattens or declines roughly linearly. They contrast this with benchmarks such as RULER, where degradation appears only near 128K. From 128K to 1M tokens, Gemini performance did not decrease on any of the three tasks. Curves for different models cross over as context grows: GPT and Claude models outperformed Gemini below 8K tokens on MRCR but decayed faster, and Gemini 1.5 Flash overtook them on MRCR and IDK at long enough contexts, an advantage retained to 1M tokens. On Latent List, higher-complexity instances degraded more steeply, and at the highest complexity GPT-4o matched GPT-4 Turbo and the Gemini 1.5 models. Restricting IDK to instances whose answer is "I don't know" produced steeper fall-off, with Gemini 1.5 Pro overtaking Claude 3.5 Sonnet at around 8K. A noted failure mode is that GPT models hallucinated puzzles in the IDK task.
The authors observe that several model families have parallel curves on MRCR, which they take both as evidence that the evaluation carries signal and as a hint that those models share aspects of training. They report MRCR as particularly robust, requiring no prompt changes across models to obtain verifiable, low-variance output, and as informative for smaller and non-post-trained models in internal testing.
Conclusion
The paper concludes that current frontier models do not solve these simple beyond-retrieval tasks even at 32K tokens, that many but not all models plateau after an initial degradation, and recommends MRCR and its extensions as "a suitable default replacement" for the needle-in-a-haystack evaluation, whose needle is typically conspicuously different from the surrounding text.
Key claims
- Three LSQ tasks: Latent List (code), MRCR (natural-language conversation), IDK (question with a possibly absent answer).
- Ten models tested to 128K tokens; Gemini 1.5 Flash and Pro to 1M.
- All models degrade sharply before 32K; Gemini curves are non-decreasing from 128K to 1M.
- MRCR chance-rate estimates: 4% (uniform random passage) and 9% (topic- or format-matched passage); IDK chance 25%.
- Recommendation: MRCR as a default replacement for needle-in-a-haystack.
Provenance
Retrieved 22 September 2026 from the arXiv HTML rendering of version 2 (arxiv.org); abstract page arxiv.org.
Relationships
- supports: MRCR — the primary text describing the MRCR task and the reasoning behind it.
- depends-on: Context Length and Long Context — the capability the evaluations are designed to measure.
- instance-of: AI Benchmarks and Evaluation
- related: Google DeepMind