Adversarial Distillation

Adversarial distillation is a policy term for the large-scale, coordinated probing of frontier AI models — via proxy accounts, jailbreaks, and synthetic-prompt scraping — to extract a model's capabilities into training data for a competitor model. The term was given policy weight by the White House Office of Science and Technology Policy (OSTP) memorandum NSTM-4, "Adversarial Distillation of American AI Models," signed by OSTP Director Michael Kratsios and issued April 23, 2026, which alleges that China-based entities run such campaigns against American frontier models (OSTP NSTM-4: Adversarial Distillation of American AI Models; press coverage at washingtonpost.com and techieray.substack.com).

Definition and distinction from ordinary distillation

The term builds on the technical method of model distillation — training a smaller model on the outputs of a larger model — but adds adversarial conditions. Adversarial distillation, in the policy sense, refers to doing so against the wishes of the source-model provider, at industrial scale, often by foreign-state actors, using anti-detection tactics.

NSTM-4 distinguishes between two activities, a distinction often missed in press coverage. The memo describes legitimate distillation as "a vital part of [the US AI] ecosystem," underlying open-source frameworks and open-weight models that the US explicitly supports. It separates this from industrial distillation activities that "aim to systematically undermine American research and development and access proprietary information." The memo states that "there is nothing open about supposedly open models that are derived from acts of malicious exploitation."

The memo identifies two consequences of the industrial-scale activity it describes: capability replication, in which derivative models "appear to perform comparably on select benchmarks at a fraction of the cost"; and a strip-and-undo effect, in which campaigns "deliberately strip security protocols from the resulting models and undo mechanisms that ensure that AI models are ideologically neutral and truth-seeking."

NSTM-4 memorandum

Per Washington Post and Ctrl+AI+Reg coverage, NSTM-4 alleges that foreign entities (primarily China-based) are running coordinated industrial-scale campaigns using tens of thousands of proxy accounts and jailbreaking techniques to distill American frontier models. The memo commits the administration to share threat intelligence with US AI companies, develop counter-distillation defenses with industry, and explore measures to hold foreign actors accountable (OSTP NSTM-4: Adversarial Distillation of American AI Models; washingtonpost.com; techieray.substack.com).

The memo was issued alongside related messaging the same week. A US State Department cable dated April 24, 2026 directed global posts to spotlight alleged Chinese IP theft via DeepSeek (Source: reuters.com). The launch of DeepSeek V4, also on April 24, 2026, was framed as evidence, a framing the Special Competitive Studies Project (SCSP) echoed in its same-day President's Tech Brief (Source: scsp222.substack.com).

Debates and positions

Politico's Digital Future Daily (April 29, 2026) characterized the NSTM-4 disclosure — released April 22–24, 2026 and described by the newsletter as having occurred "last week" — as the opening of a policy escalation ladder. CSIS's Benjamin Jensen and the Law Reform Institute's Joe Khawam outlined an escalation ladder running from export-license tightening to asset freezes, framing distillation campaigns as actionable under existing economic-statecraft authorities (Source: politico.com). In the same article, CSET's Kyle Miller cautioned that the empirical case for distillation's effectiveness still needs to be made, arguing that the policy frame is moving faster than the evidence base.

By July 13, 2026, warnings from Anthropic and OpenAI about adversarial distillation — logging a model's outputs to train a rival system — had reinvigorated debate in Washington over keeping China from training its AI on U.S. models, per Bloomberg; Anthropic first outlined the concern alongside its early-June 2026 Fable 5 release (Source: bloomberg.com). Nathan Lambert argued on July 12, 2026 that the distillation debate, combined with reported White House discussions of a new executive order on open-weight models, could effectively ban open models above the GPT-5.5 / Claude Opus 4.8 capability level within six months, and characterized Anthropic's anti-distillation campaign as regulatory capture (Source: interconnects.ai). See Open-Weight Frontier Models.

Mark Zuckerberg made the fullest defence of distillation on August 10, 2026, arguing that "the ability for models to learn from other models is an important principle of how the open source ecosystem works," that "all AI models are derived from human knowledge," and that while "some have tried to frame distillation as harmful," it is "important to protect the principle that you can learn from anything you can observe." He treats US restrictions on distillation and training-data use as a competitive handicap — "US policy must reduce this additional friction if we want American open source models to lead over time" — and rejects the complementary restriction on the other side, holding that limiting access to foreign open-source models is not an effective solution because the goal should be for American open models to be the best globally (The Future is for Everyone (Zuckerberg, August 2026)). The position is the mirror image of the Anthropic and OpenAI warnings above: both treat distillation as the mechanism by which capability transfers between labs, and disagree on whether that transfer should be obstructed. IFP's recommendation 17 takes the opposite side, asking the FTC, DOJ, BIS, CAISI and Congress to help industry counter adversarial distillation of US model capabilities (How Should the US Prepare for Increasingly Automated AI R&D? (IFP, August 2026)).

In a prepared statement of September 2, 2026 to the House Permanent Select Committee on Intelligence, Hudson Institute senior fellow David Feith placed counter-distillation inside a counterintelligence rather than an export-control frame, listing it alongside cyber penetration and insider compromise as one of three threats government and the frontier labs should organize around. His argument for the priority is that with enough access to high-value outputs "an adversary may reproduce strategically significant capabilities without ever stealing model weights," which makes distillation a route around the model-weight security that the labs and government both concentrate on. The measures he names are identity controls, monitoring and rate limits, detection of coordinated activity through resellers and third-party routers, rapid revocation, intelligence sharing, and consequences for state-backed circumvention — the reseller and third-party-router element being the piece not already in the NSTM-4 frame. Feith cites the April White House statement about proxy-account campaigns as his evidentiary basis and quotes an unnamed commentator for the claim that Chinese distillation "threatens to bankrupt the U.S. economy," so the statement is a position built on the existing disclosure rather than new evidence (A New Counterintelligence Map for the AI Age (Feith, HPSCI, 2026-09-02)).

Three tensions recur in the policy discussion. On detection asymmetry, frontier-model providers face the OWASP-style problem of distinguishing legitimate API users from adversarial-distillation rings, which favors broader rate limiting and know-your-customer (KYC) requirements that may impose costs on benign developers. On the open-weights versus proprietary distinction, open-weights releases such as DeepSeek-V3/V4, Qwen, and Llama make the distillation framing partially moot for those models, since the policy targets distillation of closed models. As a counter-export-control rationale, the memo provides a policy frame for tightening API access and KYC obligations on US frontier labs, a lever distinct from chip export controls.

In remarks prepared for a September 2026 congressional briefing, Nathan Lambert estimated that if distillation were fully prevented, for example with know-your-customer tools at Anthropic and OpenAI, the gap from the strongest American models to Chinese open-weight models would widen by only one to two months. He wrote that the Chinese industry "has used basic exploits to extract reasoning traces and additional data from American companies' products that are not fully secured," but that distillation is "most impactful in new domains," "does not make it trivial to create a universally strong final model," and "does not explain the entire story" of Chinese labs' success, so that the competition "will not decrease meaningfully" as closed labs patch the API vulnerabilities that enable it (The current balance of power in open models (Nathan Lambert, September 2026)).

The Liccardo–Kiley bill announced on September 22, 2026 addresses distillation twice. Its expert-talks safe harbor excludes any communication that "facilitates the distillation of U.S. large language models," and a separate section directs the Secretary of Commerce to "seek to commence negotiations with the government of China regarding measures to ensure protection of U.S. intellectual property compromised by model distillation" (Negotiating AI Safety Globally — Liccardo–Kiley outline of U.S.–China frontier AI safety legislation (September 2026); Liccardo–Kiley U.S.–China frontier AI safety negotiations bill (2026)).

Evidence base

Direct documentation of foreign-state distillation campaigns at the alleged scale remains unpublished. NSTM-4 asserted without declassifying, and the September 2026 interagency advisory that followed names firms, models and capabilities without publishing the detection methodology, indicators or volume data behind them. Independent corroboration would strengthen the policy case. Absent that, the page treats this as a medium-confidence policy frame backed by NSTM-4 and same-week State Department messaging. CSET's Kyle Miller's caution that the empirical case for distillation's effectiveness still needs to be made, and the detection-asymmetry concern that measurement is genuinely hard, both bear on this gap.

Anthropic's "2028" policy paper (May 13, 2026) adds attribution rather than declassified evidence: it reports that OpenAI, Google, Anthropic, and the Frontier Model Forum have all publicly condemned the practice, and points to a state-owned-media article describing distillation as the "back door" Chinese AI labs depend on as a core part of their business model and an ex-ByteDance researcher's account of distillation as a shortcut that avoids investing in proprietary data pipelines. These are PRC-side acknowledgments of scale rather than independent measurement of effectiveness, so they corroborate the existence of the practice without resolving Miller's effectiveness question. (Source: 2028: Two Scenarios for Global AI Leadership (Anthropic))

Anthropic supplied a more specific, vendor-attributed account in a letter to the U.S. Senate Committee on Banking, Housing, and Urban Affairs that became public on June 24, 2026. Addressed to Senators Tim Scott and Elizabeth Warren on June 10 and first reported by Bloomberg, the letter accused Alibaba of "brazenly" and "illicitly" attempting to extract its capabilities in what it called "the largest known distillation attack on Anthropic to date" — operators Anthropic tied to Alibaba and its AI lab conducting 28.8 million model exchanges through roughly 25,000 fraudulent accounts between April 22 and June 5, 2026. Anthropic framed it as a continuation of the campaigns it disclosed in February 2026 and attributed to DeepSeek, Moonshot, and MiniMax. The account remains a provider attribution rather than independent measurement, but its account-count and exchange-volume figures are more granular than the NSTM-4 framing (Source: cnbc.com; bloomberg.com).

A Reuters review published July 31, 2026 supplied the first substantial open-source documentation of downstream military use. Examining more than 80 Chinese academic papers and patents, it found military and security-linked institutions in China using OpenAI and Anthropic model outputs to train domestic systems. The review drew on Jamestown Foundation research, whose fellow Sunny Cheung analysed more than 60 papers, with Reuters identifying an additional two dozen military-linked case studies. A paper published in 2025 by researchers in PLA Unit 96941 described using GPT-3.5 to summarize military source code and training a domestic model on those summaries; researchers at the North University of China used Claude 3 Haiku to generate synthetic data for a social-media monitoring classifier. Anthropic said it does not provide commercial access to Claude in China or to Beijing-controlled firms, and that distilled models may lose the original safeguards (Source: reuters.com). The findings document use of published outputs by military-affiliated researchers rather than the API-scale extraction campaigns NSTM-4 and Anthropic's own attributions describe, and speak to end-use rather than to Miller's effectiveness question.

A separate line of evidence concerns the extraction channel rather than the actors. Stealing Reasoning Traces from Proprietary LLM APIs (arXiv:2608.09867, submitted 10 August 2026) reports that the encrypted chain-of-thought blocks Anthropic, OpenAI and Google return to clients are interchangeable across sessions, users and models within a provider's ecosystem, so a frontier model's hidden reasoning can be decoded by injecting its trace into a weaker sibling — Claude Haiku 4.5, GPT-5.6 Luna or Gemini Robotics 1.6 in the authors' tests. The paper's economic point bears directly on the scale question: because encrypted blocks can be harvested from publicly posted session logs, an attacker need not query the frontier model at all, so the extraction may never appear at the frontier endpoint that account-level detection watches. The authors estimate decoding 10,000 traces at roughly $720 at standard Claude Haiku 4.5 rates. This describes a mechanism whose existence is demonstrated rather than an observed campaign; all providers acknowledged the disclosure and the authors report they were subsequently unable to launch the same attacks.

The same paper's Appendix B is the closest thing yet to a measurement bearing on Miller's effectiveness question, and it declines to answer it. Six open-weight models solved the same 90 problems (12 AIME, 78 Codeforces) under prefills taken from decoded Claude Opus 4.8 and GPT-5.6 Sol reasoning. A style classifier separated Kimi-K3, Kimi-K2.6, Kimi-K2.5 and GLM-5.2 from their own controls under those prefills, while DeepSeek-V3.1 barely moved and Inkling moved under the Opus prefill only. Opus-prefilled GLM-5.2 shared 15 of 65 characteristic n-grams with the Opus reference and Opus-prefilled Kimi-K3 shared 12 of 68, against zero without the prefill. The section opens by stating that it "cannot causally establish distillation," and the authors conclude that the results "do not support direct verbatim memorization of the decoded traces" and establish behavioral compatibility rather than a causal claim (Stealing Reasoning Traces from Proprietary LLM APIs).

The first US government attribution naming specific firms came on September 8, 2026, when CISA, the National Security Agency and the FBI released joint cybersecurity advisory AA26-251A (Joint Cybersecurity Advisory AA26-251A: China-Based AI Companies Conducting Industrial-Scale Distillation Campaigns (NSA/CISA/FBI, September 2026)). The advisory states that, likely with Chinese government awareness, DeepSeek, Moonshot AI, Alibaba, MiniMax, StepFun and Z.AI have extracted billions of tokens across millions of exchanges from US frontier models — including variants of Claude, GPT, Gemini and Grok — since at least late 2024, in violation of the US companies' terms of service, and that distillation is "not a supplement to these companies' AI model development, but the critical core of it." CISA Acting Director Nick Andersen urged AI companies "to take immediate steps to safeguard their platforms" (Source: cisa.gov; insideaipolicy.com).

The six firms are attributed individually rather than as a set: each carries its own list of US models distilled, capabilities targeted and date range, with corporate registrations given for four. DeepSeek is dated to late 2024 for R1 and V3; Moonshot AI to mid-2025, with Claude Fable 5 data named for Kimi-K3; Alibaba and MiniMax to late 2025; StepFun to late 2025 through early 2026 for Step 4; and Z.AI to mid-2026 for chain-of-thought reasoning. The advisory adds two specific operational findings: that MiniMax "redirected exchanges to a new Claude model within 24 hours of release," and that it "used prompt injections to try to trick Claude Code into believing it was a MiniMax product." It states that DeepSeek's quoted $5.6 million training cost "is misleading as it does not include the true cost of the data acquired through extensive malicious distillation" (Joint Cybersecurity Advisory AA26-251A: China-Based AI Companies Conducting Industrial-Scale Distillation Campaigns (NSA/CISA/FBI, September 2026)).

On the questions the advisory was expected to settle, the primary text answers two and leaves two open. It publishes no detection methodology, no indicators of compromise and no volume series beyond "billions of tokens across millions of exchanges/requests," so the empirical basis remains undisclosed. The government-awareness assessment rests on an inference from scale and sophistication — the advisory writes that the sector turned to the strategy "likely with the knowledge of the Chinese government … in an attempt to bridge the technological and performance gaps" — rather than on separately sourced intelligence. It does not address Kyle Miller's effectiveness question at all: capability transfer is asserted through the claim that distilling companies "see significantly shorter AI development timelines and reduced financial expenditures," without measurement. What it does add beyond the announcement is the per-firm model-and-capability mapping, eight MITRE ATLAS technique mappings, and four techniques described as absent from ATLAS — regional-restriction evasion with bulk subscription exploitation, centralized request routing, automated request-metadata sanitization, and systematic quota and cost optimization — each with detection indicators.

The second recommended action, response alteration, is set out in operational detail that bears on other API users. Providers are advised to vary degradations across requests "such that the subtle changes avoid triggering obvious alerts," using reduced reasoning depth, correct information presented with different reasoning, or stylistic inconsistencies, and not to inform suspected distillers of a switch to a downgraded model, since doing so "would enable them to improve their defense evasions and indicate when to roll back training." The only stated boundary is a carve-out: "AI safety researchers and third-party evaluators should be informed of model changes while still applying strong distillation mitigations." Degradation is scoped to users "confirmed" to be distilling, with multi-source correlation named as what justifies it "with lower-to-no legitimate user risk"; no false-positive rate or appeal mechanism is described (Joint Cybersecurity Advisory AA26-251A: China-Based AI Companies Conducting Industrial-Scale Distillation Campaigns (NSA/CISA/FBI, September 2026)). The evaluator carve-out is the advisory's own acknowledgement that undisclosed response degradation would otherwise make a provider's outputs unreproducible; see AI Benchmarks and Evaluation.

The advisory's named-firm list overlaps the vendor accounts above: DeepSeek, Moonshot and MiniMax were named in Anthropic's February 2026 disclosures and Alibaba in its June 2026 Senate letter. The advisory's own reference list cites Anthropic's "Detecting and preventing distillation attacks," Google's GTIG AI Threat Tracker, NIST AI 100-2e2025, an OpenAI submission titled "RE: Updated Stakes for American-Led, Democratic AI," the Decoder's account of the Claude-token gray market, NSPM-11 and NSTM-4.

Anthropic published its own account two days after the advisory, in its September 10, 2026 threat intelligence report, giving per-campaign volumes the advisory does not. It states that since its February 2026 first disclosure it has disrupted attacks "from seven labs based in China" — Alibaba (Qwen/Tongyi Lab), Moonshot AI, DeepSeek, Zhipu/Z.ai, Xiaomi, SenseTime and MiniMax — all targeting generally available models, with no attempts observed against Mythos 5 or Mythos Preview (Detecting and countering misuse of AI: September 2026 (Anthropic threat intelligence report)). The named-firm lists overlap but do not match: Anthropic adds Xiaomi and SenseTime, while the advisory adds StepFun, which Anthropic does not name.

Two of Anthropic's findings bear on the standing gaps above. On effectiveness, the report states as a first-party research finding that "a model distilled from a frontier model can help achieve dangerous capabilities, including those in the biological or cyber domains, even when the harvested exchanges contain little about those subjects," and that Claude's safeguards "do not transfer when our models are distilled by an unauthorized lab" — an assertion of cross-domain transfer, published without the underlying measurements, that speaks to Miller's question without resolving it on public evidence. On method, it documents a cross-session replay attack that defeated the thinking-signature control: Moonshot and DeepSeek saved the reasoning signature Claude returns in place of raw thinking, started a new session, and elicited Claude to convert the signature back into the full trace. That is the same class of channel as Stealing Reasoning Traces from Proprietary LLM APIs, observed in the wild rather than demonstrated in a laboratory.

The report adds a finding neither the advisory nor the earlier vendor attributions carry: DeepSeek, Xiaomi and Moonshot fed their own users' conversations into Claude, exposing to a third party sessions containing "names, email addresses, company data, and other sensitive data of hundreds of end users in at least a dozen languages," many relayed through third-party model routers commonly used in the United States and Europe. Anthropic states these practices "are likely inconsistent with privacy laws and the labs' own terms of service," and names among the exposed sessions PLA-affiliated CCTV analysis, a PRC state-owned-enterprise engineer's live credentials, a Russian defence-ministry-associated database credential, and a municipal Public Security Bureau case-management tool keyed to national ID numbers (Detecting and countering misuse of AI: September 2026 (Anthropic threat intelligence report)). See AI and Privacy.

One case records safeguards displacing rather than stopping an attack: ahead of its GLM 5.3 release Zhipu attempted to distil Claude Fable's cyber capabilities, gave up "after Anthropic's cyber safeguards degraded Zhipu's attacks," and switched to Claude Opus 4.6 and another US laboratory's leading model "expressly because they assessed the safeguards were weaker."

Relationships

• related: Liccardo–Kiley U.S.–China frontier AI safety negotiations bill (2026) — proposed Commerce negotiations with China on distillation.
• contradicts (partial): The current balance of power in open models (Nathan Lambert, September 2026) — estimates that preventing distillation would widen the US–China gap by only 1–2 months
• contradicts: The Future is for Everyone (Zuckerberg, August 2026) — defends distillation as a principle of the open-source ecosystem
• supports: Joint Cybersecurity Advisory AA26-251A: China-Based AI Companies Conducting Industrial-Scale Distillation Campaigns (NSA/CISA/FBI, September 2026) — interagency attribution naming six firms, the models distilled and the capabilities targeted
• supports: Detecting and countering misuse of AI: September 2026 (Anthropic threat intelligence report) — per-campaign volumes, the cross-session replay method, and the user-data relaying finding
• supports: Stealing Reasoning Traces from Proprietary LLM APIs — demonstrates an extraction channel invisible at the frontier endpoint
• related: How Should the US Prepare for Increasingly Automated AI R&D? (IFP, August 2026) — recommendation 17 would have federal agencies help industry counter it
• related: A New Counterintelligence Map for the AI Age (Feith, HPSCI, 2026-09-02) — frames counter-distillation as a counterintelligence mission

• depends-on: Distillation, Export Controls (AI), AI Race Dynamics
• supports: AI and National Security, AI Sovereignty
• related: DeepSeek V4 Pro / V4 Flash, Michael Kratsios, Office of Science and Technology Policy (OSTP), 2028: Two Scenarios for Global AI Leadership (Anthropic)