Author: Nita Farahany Source: https://nitafarahany.substack.com/p/the-ai-control-paradox-youre-the Published: September 7, 2025
The fourth class (of 27) in Nita Farahany's introductory AI Law & Policy course, published as a Substack essay on September 7, 2025. The class addresses governance of open-weight foundation models. It is structured as a simulated Presidential briefing in which the student is asked to defend, in turn, each of three positions drawn from the U.S. National Telecommunications and Information Administration (NTIA) debate over dual-use foundation model weights, and then to confront the structural defect Farahany identifies in each.
Summary
The class opens with Farahany holding up a paperclip and asking whether it is a weapon or an office supply, a framing she uses to introduce the dual-use problem in which a single technology serves both benign and harmful ends. From there the student is placed in the role of an adviser who must argue each of the three NTIA positions on whether and how to control the weights of dual-use foundation models.
In the restrict (pro-restriction) position, model weights are treated like nuclear materials. The argument cites an MIT bioweapon study, the irreversibility of a public weights release, and existing precedent in the control of nuclear materials and cryptography. Farahany's stated counter is that the analogy breaks down because uranium does not improve when copied while AI does, so restriction tends to convert American AI into training data for competitors.
In the monitor (pro-monitoring) position, the student advances Stanford's Daniel Ho's evidence-based caution: the marginal risk of foundation models is unproven and the evidence is weak; regulatory overreach can kill research; and the better path is adverse-event reporting modeled on the U.S. Food and Drug Administration. Farahany's stated counter is a timing problem. A model released in January could be deployed for harm by June, and a roughly six-month detection lag would mean mass casualties had already occurred before monitoring caught the harm.
In the open (pro-openness) position, the arguments are innovation acceleration, democratization away from the Big Three developers, the strategic advantage China would gain if the United States restricts, research and safety benefits, and economic competitiveness. Farahany's stated counter is that openness carries documented harms, including child sexual abuse material (CSAM), harassment, and deepfakes.
The class closes with what it presents as a reframing attributed to Stanford HAI: the debate may be fixed on the wrong control point. Rather than controlling the weights themselves (described as the water), governance could instead target the training data (the pipes), compute (the dams), or downstream applications (the distribution).
Key claims
- Each of the three NTIA positions on dual-use weights carries a structural defect: restriction turns American models into competitors' training data because copying improves AI rather than degrading it; monitoring fails on timing, given the example of a January release, June deployment for harm, and an approximately six-month detection lag; and openness carries visceral documented costs including CSAM, harassment, and deepfakes.
- Daniel Ho's evidence-based caution holds that the marginal risk of foundation models is unproven and the supporting evidence weak, that regulatory overreach can kill research, and that adverse-event reporting in the manner of the FDA is preferable to restriction. Farahany notes that the MIT bioweapon-design study found students locating information already available on Wikipedia, and points to the collapse of Stanford's election research following a compliance review as an instance of regulatory overreach backfiring. She raises the possibility that some calls for regulation protect business models more than public safety.
- The chokepoint reframe attributed to Stanford HAI relocates the policy question from controlling weights to controlling training data, compute, or downstream applications.
Provenance
Single-source essay by Nita Farahany, published on her Substack on September 7, 2025, as Class 4 of a 27-class introductory AI Law & Policy series. It documents a teaching exercise and presents positions as arguments to be advanced and rebutted rather than as settled findings.
Relationships
- part-of: Nita Farahany intro course series (Class 4 of 27)
- related: Open-Source AI / Open-Weight Models, Dual-Use Frontier AI, Daniel Ho (planned), Three-Lane Standards-Based AI Governance (related framing)
- previous: Inside My AI Law & Policy Class 3: Open vs. Closed AI Models (Farahany, September 2025) next: Inside My AI Law & Policy Class 5: The $1.5 Billion Question (Farahany, September 2025)