This paper introduces Dynabench, an open-source platform for dynamic dataset creation and model benchmarking. Its first author is Douwe Kiela of Facebook AI Research, with Max Bartolo and Pontus Stenetorp (UCL), Yixin Nie and Mohit Bansal (UNC Chapel Hill), Divyansh Kaushik (CMU), Atticus Geiger, Zhengxuan Wu and Christopher Potts (Stanford), Bertie Vidgen (Alan Turing Institute), Grusha Prasad (Johns Hopkins), Zeerak Waseem (Simon Fraser), and Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Robin Jia and Adina Williams of Facebook AI Research. It was posted to arXiv as 2104.14337 in April 2021 and published at NAACL 2021.
The paper is both a position piece and a platform description. Its argument is that models now reach estimates of human performance on new benchmarks within a few years where it once took decades, while still failing on simple challenge examples and in real use, because "our benchmark tasks are not adequate proxies for the sophisticated and wide-ranging capabilities we are targeting: they contain inadvertent and unwanted statistical and social biases that make them artificially easy and misaligned with our true goals." Its opening figure plots saturation over time for popular benchmarks, normalised so that initial performance is −1 and human performance 0.
Summary
The platform
Dynabench hosts tasks whose data is collected over successive rounds, each beginning from the then-current state of the art, with one or more target models in the loop. Annotators, either crowdworkers or expert linguists, try to produce examples the model gets wrong; examples that fool the model are validated by other humans. The resulting error rate — how often people can fool a model in live interaction — is itself the metric, and the collected examples serve both as evaluation data and as training data, which the authors hope creates a cycle of improvement.
The current iteration is set up for adversarial collection, on the reasoning that the average case measured by i.i.d. training is less interesting than the worst case a deployed system must handle. The authors note the design is not limited to the adversarial setting: examples a model is merely uncertain about, or collaborative and feedback settings, would fit the same live-in-the-loop structure.
The stack is built on PyTorch with models served through TorchServe, displays prediction probabilities, and offers an "inspect model" view of token-level integrated gradients via Captum. For each example, annotators explain what the correct label is and why they think the model was or was not fooled. Task owners can collect examples through the web interface or through Mephisto, which connects crowdworkers to the same backend; all collected data is to be open-sourced in anonymised form.
The authors expect expert linguists to matter more over time, noting that ANLI's verified model error rate for crowdworkers fell below one in ten in later rounds, and that in "Beat the AI" human performance fell while time per valid adversarial example rose as models strengthened.
Initial tasks and results
Four tasks were launched: natural language inference, seeded by the three ANLI rounds, with a fourth round using a two-architecture ensemble as adversary and collecting two kinds of natural-language explanation; extractive question answering in the SQuAD1.1 format, seeded by "Beat the AI" with BiDAF, BERT-Large and RoBERTa-Large as first-round targets; three-way sentiment analysis, where the authors compared eliciting edits to real review sentences against writing new ones and found the prompted condition produced more linguistically diverse examples; and hate-speech detection over four rounds, using trained analysts rather than crowdworkers, with granular labels for target group and statement type and minimal-edit perturbations for about half of entries in rounds 2 to 4.
Table 1 gives rounds, examples and validated model error rate (vMER, human-validated model errors divided by total examples) per task: NLI 4 rounds, 170,294 examples, 33.24%; QA 2 rounds, 36,406, 33.74%; sentiment 3 rounds, 19,975, 35.00%; hate speech 4 rounds, 41,255, 43.90%. The authors caution that rates are not comparable across tasks because interfaces and in-the-loop models differ, and read the high rates as showing that the field is "far from solved." On hate speech they report that content became more adversarial and models improved across rounds, with the final round's target model reaching 94% on the HateCheck diagnostic suite.
Objections the authors address
The paper answers seven. On unnatural distributions: crowdsourced text is already artificial and adversarial collection can worsen it, though prior work shows models can learn both distributions when trained on combined data, so adversarial and naturally collected data should be mixed. On annotators overfitting to a model: the risk is cyclical progress where later models forget earlier rounds, and the response is to evaluate on all rounds plus static test sets rather than only the newest, and to use ensembles of architectures in the loop. On future models not in the loop: the authors concede they cannot account for these and treat an ensemble as an approximation. On comparability as the benchmark changes: they draw an analogy to successive SemEval and WMT datasets, where scores across years are already understood as non-comparable. On generative tasks: Dynabench covers classification and span extraction, where being wrong is easy to establish, and extending to generation would require discretising outputs or asking annotators to judge them. On whether models are needed in the loop: evidence of gains is mixed, and they recommend combining adversarial with non-adversarial data. On cost: they cite roughly $0.50 per validated SNLI example, $0.64 for MultiNLI and closer to $1.00 for ANLI, acknowledge that costs rise through rounds as models harden, and argue that longer-lived benchmarks may justify the expense.
The conclusion lists near-term goals: letting anyone run a task given a target model, contexts and annotators; extending beyond English text to multilingual and multimodal settings; and live model evaluation through a standard upload interface that would score compute use, inference time and fairness alongside accuracy, enabling dynamic leaderboards and putting new state-of-the-art models into the loop automatically.
Key claims
- Benchmarks now saturate within a few years rather than decades, while models still fail on simple challenge examples.
- Dynabench measures how often humans can fool a model in live interaction, over rounds that restart from the current state of the art.
- Four initial tasks; vMER 33.24% (NLI), 33.74% (QA), 35.00% (sentiment), 43.90% (hate speech); rates not comparable across tasks.
- Adversarial data collection costs roughly double static collection per validated example and rises across rounds.
- Coverage is limited to English text and to classification and span-extraction tasks.
Provenance
Retrieved 22 September 2026 from the arXiv HTML rendering (arxiv.org); abstract page arxiv.org.
Relationships
- supports: AI Benchmarks and Evaluation — the saturation argument and the dynamic-benchmarking response.
- supports: Benchmark Performance Over Time — the source of the shortening-saturation-interval claim.
- related: AI and the Everything in the Whole Wide World Benchmark (Raji et al., 2021) — a contemporaneous critique of benchmark construct validity.