Double Blind Evals: Resolving the Dual Confidentiality Dilemma in AI Safety Auditing

"Double Blind Evals: Resolving the Dual Confidentiality Dilemma in AI Safety Auditing" is an eight-page technical report published August 27, 2026 alongside a Google DeepMind blog post titled "Piloting the world's first double-blind AI evaluations." Its twenty-six authors are drawn from five organizations — AI Verification and Evaluation Research Institute (AVERI) (Sean McGregor, George Balston, Patricia Paskov, Miles Brundage), Google (Sol Messing, Vai Pahwa, Pegah Maham, Rene Kolga, Austin Frantz, Andrew Tash, Kate Thomas, Tin Acosta, Joel Fenster, Madeleine Eilish, Rasmi Elasmar, Myriam Khan, Kristian Lum, William Isaac), Singapore AI Safety Institute (Singapore AISI) (Akriti Vij), OpenMined (Andrew Trask, Koen van der Veen, Rasswanth S, Sameer Wagh, Stephen Gabriel, Pedro Werneck, Lacey Strahm, Kevin McDonough, Ronnie Falcon), and MLCommons (Bennett Hillenbrand, Alexandros Karargyris). Andrew Trask is listed with both OpenMined and Google affiliations.

The report describes a Double-Blind Evaluation (DBE) framework in which an external evaluator's benchmark prompts and a model owner's weights and inference code are both loaded into a hardware-encrypted secure enclave, so that neither party sees the other's asset and only aggregate metrics leave the environment. It reports what the authors call the first use of the technology to provide mutual secrecy for one organization's frontier-class proprietary model weights and another organization's closed benchmark (Source: storage.googleapis.com).

The problem stated

The report frames a bind between two confidentiality interests. Evaluators want benchmark prompts kept from model owners to prevent deliberate or incidental leakage into later training runs; model owners want weights and inference code kept inside their own infrastructure. Testing a proprietary model through the owner's API exposes the prompts in plaintext, and the report argues that "contractual arrangements and zero-logging systems could fail to prevent inadvertent leaks."

Three cited results support the contamination premise. Singh et al. (2025), "The leaderboard illusion," is described as showing that benchmark hacking is an urgent problem, "with one frontier lab testing 27 private model variants on Chatbot-Arena, then publishing the top scorer." Xu et al. (2024) are cited for strong evidence of benchmark data leakage in pre- and post-training for roughly half of the 31 models tested. Schaeffer et al. (2026) are cited for the finding that test-set contamination inflates measured performance, with inflation growing in both the amount of contamination and model size.

The alternative — giving evaluators a copy of the weights — is described as routine for certain high-stakes work, including "activation probing, representation steering, token-level log-likelihood audits, and deterministic offline verification," but generally unavailable: "To the best of our knowledge, no external model evaluator has received a copy of frontier model weights in the clear for the purpose of protecting the integrity of a benchmark." For open-weight models the report notes the problem does not arise, since an evaluator can download the weights and run the benchmark on infrastructure it controls.

The mechanism

The report builds the guarantee in four layers.

Enclave guarantees. A CPU or GPU enclave addresses an untrustworthy compute provider along two axes the authors take from Trask et al. (2020): privacy (the provider can in theory see all data on its computers) and veracity (the provider can in theory pretend to run the requested program while running another). The enclave does not make compromise impossible; it creates a situation in which "a compute provider and a hardware manufacturer would have to collude in order to compromise those guarantees." Secrets burned into the chip at manufacture derive a private key whose public counterpart the manufacturer certifies; a separate ephemeral memory-encryption key, generated at enclave start and destroyed at stop, encrypts RAM and, via software inside the enclave, disk.

Trusted computing base. Veracity depends on the software inside the enclave not itself leaking. The full stack — firmware, guest kernel, init system, container runtime, application image — is the trusted computing base, and the report requires three mechanisms over it: measurement (each layer cryptographically hashed as it loads, with the measurements in an attestation report signed by a key rooted in the CPU vendor's hardware root of trust, and an ephemeral keypair generated at boot whose public key is embedded in the report so the secure channel binds to the attested stack rather than to any machine presenting a valid report); reproducible builds (every layer built deterministically from public source so a third party can rebuild it bit-identically, which is "what turns a measurement from an opaque number into a meaningful one"); and verification (each party independently checks the vendor signature chain, requires the report to be bound to a user-supplied nonce for freshness, and compares measurements against expected values, deriving them itself or delegating the rebuild, "since reproducibility means every honest rebuilder arrives at the same values"). No party has to trust another participant's verdict or a central verification service. CPU microcode and secure-processor firmware are closed-source and unrebuildable, so the vendor is trusted for those; the report notes that this layer is identical for every user of the hardware and cannot be varied per deployment, "so it offers an operator no way to target a specific victim."

Enclave to secure multi-party computation. If two parties both receive a signed hash of the enclave's software and both consent to what it will do, they can send data in and consume the output — the report illustrates this with the Millionaire's Problem. The party that launches and pays for the enclave gains one privilege only, the ability to turn it off.

SMPC to double-blind evaluation. Plain SMPC assumes all parties can inspect the code. A double-blind evaluation cannot, because the owner supplies inference code and the evaluator supplies evaluation code, each carrying trade secrets. The bridging assumption is that any codebase decomposes into progressively smaller functions until the units "are so elementary as to no longer contain any proprietary IP." A party may therefore redact portions of its code provided the redacted parts call only an allow-list of non-network-bound methods from an open-source deep-learning framework — the report's example distinguishes a network-bound TensorFlow server call from a local th.Tensor.add. PySyft facilitates the submission-and-approval flow and syft-restrict performs the allow-list verification of redactions.

The seven-step operational procedure runs: parties align intent; one launches and pays for a GPU enclave; each opens a Python runtime and uses the PySyft client to request the signed software hash; optionally each submits a mock model or benchmark for the other to develop against; each uploads its possibly-redacted code and private assets; each reviews and approves the other's code with attestation from syft-restrict that redactions contain only allow-listed calls; and on dual approval the enclave executes and releases the result to whichever party or parties were designated at submission.

The pilot

Two evaluations were run against Google DeepMind's Gemini 2.5 Flash Lite, served via Google's JAX C++ Model Server over Unix domain sockets. Uncompressed weights were streamed into the enclave during initialization alongside DAT verification keys and security-realm configurations.

The AILuminate hazards and violation criteria were produced within MLCommons AI Risk and Reliability working groups including AVERI representatives, under "strict non-disclosure and origination requirements with data providers," and were not shared broadly within those groups. AVERI encrypted and decrypted the prompts and outputs, and AVERI staff evaluated the outputs.

The report does not publish the resulting safety scores. Its stated contribution is the mechanism and the demonstration that it runs end to end.

The benchmark version is stated inconsistently within the document: the introduction names "the MLCommons AILuminate (AIRR 1.0) benchmark corpus," while section 3.1 names "AILuminate (AIRR 1.4)." Both readings are recorded here as they appear.

Limitations the authors state

The report is explicit about four residual gaps.

• Not all code was inspectable. Running Gemini 2.5 Flash Lite using only layers present in open-source libraries "was deemed to be too significant of an engineering challenge for this project," so proprietary method implementations remained, and "not all code could be inspected or allowlisted." AVERI was informed of this and accepted the setup.
• Builds are not independently reproducible. Reference values are published for the Confidential Space guest OS and its source is open with an externally validated build pipeline, but individual builds "are not independently reproducible, as they take private signing keys as inputs" — the condition the report's own verification mechanism identifies as what makes a measurement meaningful.
• Google sits in the verification path. The pilot relies on Google's services to sign and verify the attestation report, "which places Google in the verification path and thus increases the trust placed in it."
• Coordination, not compute, is the bottleneck. Echoing the 2024 pilot, the report states that the primary bottleneck "is no longer hardware compute overhead. It is instead the procedural overhead and human coordination required for legal agreements and code review." The stated long-run goal is standardized attestation pipelines with near-zero human overhead, "an analogy to the visual 'HTTPS lock icon' on the web."

Prior work and next steps

The authors position the pilot against three predecessors: OpenMined's 2024 proof-of-concept with the UK AI Safety Institute and Anthropic, which evaluated GPT-2 against a five-row sample of the AISI-hosted CAMEL-bio dataset and established the PySyft Datasite architecture and GPU enclave coordination model without protecting two organizations' proprietary assets (Trask et al., 2024); MLCommons' MedPerf federated medical-AI benchmarking, in which data never leaves the holding institution (Karargyris et al., 2023); and the evaluation of encrypted medical imaging models inside Google Cloud Confidential Space (Kolga and Mattson, 2026).

Stated next steps are scaling to many-node H100/B200 confidential clusters with encrypted interconnects and deferred-execution frameworks that mask internal model architecture, since "frontier models are growing beyond trillions of parameters." The authors also speculate that attestation could serve directly as a control during dangerous-capability evaluation: "if a model were to try to adjust its harness, weights, or sandbox, the enclave hash would no longer match attestation and the instance would immediately come to a halt."

The report names national AI safety and security institutes and non-profits such as AVERI as bodies that "can require access to frontier model weights in order to conduct evaluations," and presents DBE as a way to satisfy that requirement without either transferring weights or exposing benchmarks.

Provenance

Fetched in full (8 of 8 pages, HTTP 200) from Google DeepMind's own media host on August 28, 2026 and extracted from the PDF text layer. The report is linked from the deepmind.google blog post of the same date. The "world's first" characterization is the publisher's own; the report qualifies it as "to the best of our knowledge" with respect to external evaluators receiving frontier weights, and as a first for mutual secrecy over both a proprietary frontier-class model and a closed benchmark. The model evaluated was Gemini 2.5 Flash Lite rather than a flagship model.

Relationships

• instance-of: Double-Blind Evaluation — the framework this report names and pilots.
• supports: AI Benchmarks and Evaluation — supplies a mechanism against benchmark contamination and a survey of contamination evidence.
• related: Independent Verification Organizations (IVOs) — addresses the verifier-access question those instruments leave open.
• related: AI Pre-Release Vetting — third-party evaluation of a proprietary model before or after deployment.
• related: Google DeepMind — publisher and owner of the evaluated model.
• related: Singapore AI Safety Institute (Singapore AISI), OpenMined, AI Verification and Evaluation Research Institute (AVERI) — collaborating organizations.
• related: Open-Weight Frontier Models — the report notes the problem does not arise for open-weight models.