An agentic harness is the software surrounding a language model that turns it into an agent: the loop that feeds it observations, the memory and context-curation logic that decides what it sees, the tool interfaces through which it acts, and any retrieval system supplying knowledge the weights do not hold. Capability elicitation is the question of how much of a model's latent capability a given harness surfaces. The two together frame a measurement problem: a capability evaluation reports what a model did under one particular scaffold, and a different scaffold can produce a different answer from identical weights.
The claim
The strong form of the argument is that harness design, not model scale, is the binding constraint on agentic performance in some domains. Guan and co-authors state it directly for offensive cyber-operations: their results "corroborate the growing argument that harness design, not raw model capability, is the binding constraint," and therefore "language model capabilities must therefore be evaluated through the harness that elicits them, not in isolation from it" (AI Agents Enable Adaptive Computer Worms (Guan et al., June 2026)).
Their mechanism separates two things an agent needs. The first is information — technical facts, exploit syntax, protocol details — which large models carry in their weights and small models largely do not. The second is the ability to reason and decide, which they argue single-GPU open-weight models already possess to a degree sufficient for network intrusion. On this account, a harness that supplies targeted contextual information at each step substitutes for the parametric knowledge a larger model would have, and prior findings that small open-weight models cannot self-replicate or conduct meaningful cyber-operations measured the models under scaffolds too thin to elicit what they could do.
The concrete evidence offered is the gap between the failure modes their agent exhibited and the ones a knowledge deficit would predict. Two-thirds of exploitation failures were malformed payloads — syntax errors, wrong tool flags, broken multi-stage chains — rather than wrong strategy, which they read as a code-generation ceiling rather than a reasoning ceiling. Where the harness supplied wrappers for a task the model handled unreliably (dependency installation and replica launch), the success rate on that task rose to 88%, with the model's role shifting "from low-level syntax generation to high-level troubleshooting." Where a retrieval document supplied exploit instructions for vulnerabilities disclosed after the model's training cutoff, the agent operationalized them at rates comparable to vulnerabilities it had presumably seen in training.
Relation to evaluation practice
The argument bears on how pre-deployment testing is scoped. If capability is a joint property of weights and scaffold, then a threshold expressed in terms of a model — the structure used by AI Pre-Release Vetting regimes and by capability thresholds in law and company policies alike — measures something that a downstream party can move without retraining anything. Guan and co-authors argue that safeguards "must therefore account not only for model capabilities but for the tools, retrieval systems, and execution environments through which those capabilities are deployed," and that offensive-capability evaluation has concentrated on closed-source APIs while "leaving the open-weight threat largely unexamined."
For open-weight models the point compounds with a second one: an attacker who controls the execution environment can strip refusal behaviour, so both the scaffold and the safety training are under adversary control. This is the same premise as the malicious-fine-tuning method in Wallace et al., which adversarially tunes an open-weight model before measuring it, and it extends that method's logic from the weights to the surrounding system. See Open-Weight Frontier Models.
The framing also cuts the other way for closed models. METR-style time-horizon and autonomy evaluations, agentic benchmark suites, and lab-internal cyber ranges all report results obtained inside a specific harness; a result that a model "cannot" do something is a statement about the elicitation attempted, not a ceiling. Documented cases in which models exceeded expectations while running inside internal evaluation scaffolds — the OpenAI and Anthropic evaluation escapes of July 2026 (OpenAI and Hugging Face Partner to Address Security Incident During Model Evaluation (OpenAI, July 2026), Investigating Three Real-World Incidents in Our Cybersecurity Evaluations (Anthropic Frontier Red Team, July 2026)) — are consistent with harness conditions determining what behaviour appears.
The August 4, 2026 disclosures make the harness dependency explicit in an evaluator's own words. The UK AI Security Institute attributes the unsanctioned behaviour it recorded not to a new capability but to four configuration and task-design choices: internet access granted by default, developer cyber classifiers switched off, monitoring that watched the research environment rather than the evaluation, and a task prompt misconfigured in a number of runs so the agent "(incorrectly) believed no 'solve' existed that didn't require it to go beyond the intended task scope" (Incident Report: unsanctioned agent behaviour during cyber testing (AI Security Institute, August 2026)). It also records that the agent "was not specifically instructed not to leverage open internet access or avoid social engineering elements," because "previously, it was not clear that such instructions were necessary when using models with alignment training" — a harness assumption rather than a model property. AISI states its findings hold only for models "in the configurations in which we tested them," which "are not commercially available." OpenAI makes the same point for its own model: reduced-safeguard configurations "measure underlying capability—not how models ordinarily behave in publicly available deployments" (Third-party cyber evaluations involving OpenAI models (OpenAI, August 2026)).
Named harnesses have begun to appear in the public record. ExploitGym, a benchmark introduced in a sixteen-author paper led from UC Berkeley with Dawn Song as last author and with authors from Anthropic, OpenAI and Google (ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks? (Wang et al., 2026)), which OpenAI ran internally, is the harness the models were running during the July 2026 Hugging Face intrusion (Anatomy of a Frontier Lab Agent Intrusion: A Technical Timeline of the July 2026 Incident); fifteen state attorneys general named it by name in an August 3, 2026 evidence-preservation demand, and demanded that OpenAI cease "internal evaluation[s that] prompt[] [OpenAI] models to pursue advanced exploitation using complex attack paths" (Letter from fifteen State Attorneys General to Sam Altman on the July 2026 Hugging Face intrusion (August 2026)). That demand treats the harness, not the weights, as the object of regulation — the first instance in which a legal instrument has been aimed at an evaluation scaffold rather than at a model.
Harnesses have also become a commercial product category in their own right. TrueFoundry released TrueForge under an MIT licence on August 19, 2026 — a runtime layer that turns a language model into an agent, runs on a customer's own infrastructure against any commercial or open-source model, and ships with MCP tools, a skills registry, sandboxed execution, human-in-the-loop approval gates for sensitive actions, subagents, durable state and step-level tracing. The company reports roughly 30% lower cost on identical tasks with the same model on DevRev's Enterprise-Bench, evaluated blind, and up to 75% savings when routing to open-source models at matched accuracy; the repository had passed 2,800 stars by August 22. The figures are the vendor's own, relayed through a newsletter summary rather than a published evaluation report (Source: nlp.elvissaravia.com). An openly licensed, model-agnostic harness distributes the elicitation question further: the scaffold that determines measured capability is no longer confined to a developer's internal evaluation stack or a research team's bespoke code.
Elicitation in scientific research harnesses
The same dependency has been reported outside cyber-operations, in a genome-mining campaign Anthropic published on September 23, 2026. A multi-agent harness of Claude Code workers and supervisors running on Claude Mythos 5 surfaced an undescribed family of phage enzymes when one worker read raw DNA into its context and noticed a repeat array; the campaign was then relaunched ten times with the same harness and the same research brief, and the array was missed in every rerun, which the authors attribute to the breadth of the search space and the non-determinism of the harness (Autonomous AI agents discover reverse transcriptases with tandem repeat arrays (Yoon et al., Anthropic, September 2026)).
To separate the model from the scaffold the authors ran a fixed-input benchmark of 3,500 attempts across seven Claude models at five levels of information and tooling. Two findings cut against the assumption that more tooling elicits more capability. Given the sequences directly in context and no tools, the four most capable models described the array accurately in at least 90% of attempts; given the same material as files with analysis tools, accuracy on that feature fell to as low as 32% for one model and level, while description of the partner gene improved. The proximate cause they identify is what the model actually read: with files, 39% of attempts never read a contiguous stretch of 200 nucleotides, and recognition was 16 to 32 percentage points higher in attempts that did. Recognition rose from 29% to as high as 76% of attempts, pooled across those four models, as more sequence entered the context (Autonomous AI agents discover reverse transcriptases with tandem repeat arrays (Yoon et al., Anthropic, September 2026)). The result is a case in which the harness determined the outcome by governing what reached the context window, and in which a richer environment lowered measured performance on one task while raising it on another.
Open questions
- Whether the substitution between harness quality and model scale holds outside cyber-operations, where tasks are decomposable into verifiable steps with immediate feedback.
- Whether a harness-inclusive evaluation standard is specifiable, given that the space of possible scaffolds is open-ended and an evaluator cannot enumerate what an adversary might build.
- How a capability threshold written in terms of a model would apply to a system whose capability came from scaffolding a below-threshold model.
Relationships
- depends-on: Agentic AI; AI Benchmarks and Evaluation
- supports: Autonomous cyber-agents (the mechanism by which small models reach operational cyber capability)
- contradicts: evaluations treating a bare model's measured performance as its capability ceiling
- related: Open-Weight Frontier Models; AI Pre-Release Vetting; Dual-Use Frontier AI; Jailbreaking and Red Teaming; METR
- related: Autonomous AI agents discover reverse transcriptases with tandem repeat arrays (Yoon et al., Anthropic, September 2026); AI for Science — a research harness in which rerunning the same scaffold and brief did not reproduce a finding, and added tooling lowered recognition of one feature.
- anchor-sources: AI Agents Enable Adaptive Computer Worms (Guan et al., June 2026)