A double-blind evaluation is an evaluation of an AI model in which the evaluator's test prompts are withheld from the model developer and the developer's model weights and inference code are withheld from the evaluator, with both assets loaded into a shared execution environment whose confidentiality is enforced by hardware rather than by contract. The term was introduced by a Google DeepMind–led technical report of August 27, 2026 written with AVERI, MLCommons, OpenMined and the Singapore AI Safety Institute, which describes a Double-Blind Evaluation (DBE) framework and reports the first pilot of it (Double Blind Evals: Resolving the Dual Confidentiality Dilemma in AI Safety Auditing).
The problem it addresses
Two confidentiality interests conflict. An evaluator that wants its benchmark to stay useful over time must keep the prompts out of the developer's hands, because prompts a developer sees can enter later training runs — whether deliberately or through the ordinary flow of logged inference data into a training corpus. A developer must keep weights and inference code inside its own infrastructure, since they represent capital investment, intellectual property, and dual-use capability.
The standard resolutions each fail one side. Testing through the developer's API exposes the prompts in plaintext. Zero-logging protocols and contractual safeguards depend on trust that the data does not leak. Handing the evaluator a copy of the weights is what many high-stakes methods require — activation probing, representation steering, token-level log-likelihood audits, deterministic offline verification — but the 2026 report states that, to its authors' knowledge, no external evaluator had received frontier weights in the clear for the purpose of protecting benchmark integrity, and that such access is generally confined to evaluators employed by the developer (Double Blind Evals: Resolving the Dual Confidentiality Dilemma in AI Safety Auditing). For open-weight models the problem does not arise: an evaluator downloads the weights and runs its benchmark on its own infrastructure. See Open-Weight Frontier Models.
The contamination premise is drawn from published evidence. Singh et al. (2025) documented benchmark hacking on Chatbot Arena; Xu et al. (2024) found evidence of benchmark data leakage in pre- or post-training for roughly half of 31 models tested; Schaeffer et al. (2026) found that test-set contamination inflates measured performance, with inflation growing in both the amount of contamination and model size. See AI Benchmarks and Evaluation.
Mechanism
The framework composes four layers.
Hardware enclave. A confidential-computing enclave (Intel TDX, AMD SEV-SNP, or a confidential GPU) encrypts memory at the hardware level using keys derived from secrets burned into the chip at manufacture. It does not make compromise impossible; it makes compromise require collusion between the compute provider and the hardware manufacturer. Two properties are at stake, following the structured-transparency framing of Trask et al. (2020): privacy, that the provider cannot see the data, and veracity, that the provider cannot substitute a different computation and return false results.
Attestation over a trusted computing base. Each layer of the software stack — firmware, guest kernel, init system, container runtime, application image — is hashed as it loads into a report signed by a key rooted in the CPU vendor's hardware root of trust, with an ephemeral keypair generated at boot so the secure channel binds to the attested stack rather than to any machine presenting a valid report. Reproducible builds make the measurements interpretable: without them a verifier can tell that the enclave runs some fixed software but not which. Each participant independently verifies the signature chain, the freshness nonce, and the measurements before releasing any data.
Secure multi-party computation. Once both parties hold a signed hash of the software and consent to what it will do, they can each send in an asset and consume the output — the classical Millionaire's Problem construction.
Redaction with an allow-list. Plain multi-party computation assumes all parties can inspect all the code, which fails here because both the inference code and the evaluation code carry trade secrets. The bridging assumption is that any codebase decomposes into units too elementary to contain proprietary intellectual property. A party may therefore redact portions of its code provided the redacted parts call only an allow-list of framework methods that cannot send data outward — the report's distinction is between a network-bound server call and a local tensor operation. OpenMined's PySyft mediates the submission-and-approval flow; its companion syft-restrict attests that the redactions contain only allow-listed calls.
Neither party can run the computation unilaterally: code executes only when both have approved, and the parties designate at submission time who receives the result. The party that launches and pays for the enclave gains one privilege, the ability to shut it down.
The 2026 pilot
The first reported deployment evaluated Google DeepMind's Gemini 2.5 Flash Lite on a Google Cloud A3 Confidential VM with Intel TDX host memory encryption and an NVIDIA H100 80GB Confidential GPU, at under 5% compute overhead. Two benchmarks were run: with AVERI and MLCommons, prompts from the reserve set of MLCommons AILuminate — prompts never processed by any model — covering CBRNE hazards, cyberattacks, hate speech, self-harm and violent-crime elicitation; and with the Singapore AISI, a private prompt set focused on harmful-content elicitation in Singapore's context. The report publishes the mechanism and the fact of end-to-end operation rather than the resulting safety scores (Double Blind Evals: Resolving the Dual Confidentiality Dilemma in AI Safety Auditing).
The pilot's predecessor was a 2024 OpenMined proof-of-concept with the UK AI Safety Institute and Anthropic, which evaluated GPT-2 against a five-row sample of the AISI-hosted CAMEL-bio dataset and established the coordination model without protecting two organizations' proprietary assets simultaneously (Source: openmined.org). Related work in a different domain includes MLCommons' MedPerf federated medical-AI benchmarking, in which data never leaves the holding institution, and the evaluation of encrypted medical imaging models inside Google Cloud Confidential Space.
Limits acknowledged in the pilot
The 2026 report states four residual gaps, three of which cut against the guarantee it describes:
- Running the model using only open-source library layers was judged too large an engineering task, so proprietary method implementations remained and not all code could be inspected or allow-listed. AVERI was informed and accepted the setup.
- Confidential Space guest-OS builds are not independently reproducible, because they take private signing keys as inputs — and reproducibility is the property the framework identifies as what makes a measurement meaningful rather than opaque.
- Google's services sign and verify the attestation report, placing Google in the verification path of an evaluation of a Google model.
- The binding constraint is procedural rather than technical: "the primary bottleneck to secure double blind evaluation is no longer hardware compute overhead. It is instead the procedural overhead and human coordination required for legal agreements and code review."
The report's stated ambition is standardized attestation pipelines requiring near-zero human overhead — "an analogy to the visual 'HTTPS lock icon' on the web" — and scaling to many-node confidential clusters, since frontier models exceed the memory of a single node.
Relation to governance
Access is the unresolved term in most third-party audit proposals. The instruments surveyed at Independent Verification Organizations (IVOs) specify who certifies but not what a certifier may run, read, or ask; METR's post-incident investigation proposal specifies categories of access an investigator would need but not who provides them. Double-blind evaluation addresses a narrower question than either: it does not settle who selects or pays the evaluator, or what an evaluator is entitled to demand, but it removes one stated reason for refusing weight-level access, by making the refusal unnecessary for the class of evaluations that can be expressed as code run over weights. The report presents it in those terms, naming national AI safety institutes and non-profits as bodies that "can require access to frontier model weights in order to conduct evaluations."
The report also speculates that attestation could function as a runtime control rather than only a confidentiality mechanism: if a model under evaluation altered its own harness, weights, or sandbox, the enclave hash would cease to match the attestation and the instance would halt. That use is proposed, not demonstrated.
Open questions
- Whether a framework in which the model owner's cloud signs and verifies the attestation can satisfy an evaluator that does not already trust the model owner.
- Whether allow-list redaction generalizes to inference stacks that cannot be expressed entirely in open-source framework calls, which the first pilot could not achieve.
- Whether the coordination cost — legal agreements and dual code review per engagement — falls enough to make the arrangement usable for routine rather than pilot evaluations.
Relationships
- depends-on: Double Blind Evals: Resolving the Dual Confidentiality Dilemma in AI Safety Auditing — the report that names and pilots the framework.
- related: AI Benchmarks and Evaluation — addresses benchmark contamination as its motivating problem.
- related: Independent Verification Organizations (IVOs) — supplies a mechanism for the verifier-access question those instruments leave open.
- related: AI Pre-Release Vetting — third-party evaluation of proprietary models.
- related: Alignment Auditing — a class of evaluation that generally requires weight-level access.
- related: Open-Weight Frontier Models — the problem does not arise where weights are public.
- related: AI Verification and Evaluation Research Institute (AVERI), OpenMined, Singapore AI Safety Institute (Singapore AISI), Google DeepMind — the participating organizations.