Author: Nita Farahany Source: https://nitafarahany.substack.com/p/the-glass-box-paradox-when-ai-cant Published: September 28, 2025
This is the tenth installment (Class 10 of 27) in Nita Farahany's "Inside My AI Law & Policy Class" Substack series, published September 28, 2025. The essay addresses AI transparency, arguing that a gap exists between what laws such as H.R. 7532 demand of AI systems ("sufficiently explainable") and what is technically achievable, given that Anthropic reported having mapped only 1% of Claude's features as of May 2024. It introduces a five-levels-of-meaningful-transparency framework.
Summary of argument
Farahany argues that transparency requirements imposed by law often cannot be met technically, and that the forms of transparency that are achievable are frequently insufficient or misleading. She distinguishes three transparency windows: process transparency ("AI was used"), technical transparency ("neural network with 175B parameters"), and outcome transparency (for example, "denied because of late payments 40%, high credit utilization 30%, short credit history 30%"). She argues all three can be insufficient or actively misleading, and that outcome transparency is often a post-hoc rationalization of what the AI might have been doing rather than an account of how it actually reached a result.
To illustrate the limits of technical transparency, the essay cites Anthropic's May 2024 announcement that it could "see" features inside Claude, including features the company associated with the Golden Gate Bridge, deception, writing insecure code, emotional manipulation, and financial distress. Farahany notes that only 1% of features were mapped, with the other 99% remaining, in her phrasing, "mystery meat" and a "complete mystery," using an apartment-with-flashlight metaphor for the partial visibility.
Drawing on Cheong (2024), the essay frames an explainability gap: explaining AI decisions is not merely difficult but may be theoretically impossible, because the AI does not "decide" the way humans do. On this view, the "factors" commonly cited (40% this, 30% that) are post-hoc human approximations of pattern-matching rather than the system's actual reasoning.
Legal and policy examples
H.R. 7532. Farahany characterizes the bill as "transparency theater." It required federal agencies to make AI "sufficiently explainable and understandable," qualified by the phrase "to the extent practicable." She argues it mandates institutional transparency (what systems exist) more than decisional transparency (why a particular person was denied), and that it stalled in committee not over industry opposition but because it required something technically impossible while creating a large bureaucratic burden.
Hiroshima AI Process Reporting Framework. The essay describes this G7 framework (February 2025) as voluntary, with companies choosing their own participation, scope, and timing. It cites an OECD grading from September 2025 under which companies report that they have a process, that they think about safety, and that they have identified some risks, without specifying which risks, how often things go wrong, who was harmed, or what is being done about it.
NYC Local Law 144. Farahany presents the law's bias-audit posting requirement as a perverse-transparency case. She cites a posted audit with selection rates of White 8.2%, Black 6.4%, Hispanic 6.5%, Women 6.8%, and Men 8.1%, against a discrimination threshold set at 80% of the highest rate, or 6.56%. In her reading, a Black female engineer reading the posting learns that the company's AI is calibrated to discriminate against her as much as is legally permitted, and the design drives optimization toward 79.99% to game the threshold.
Houston teachers' algorithm. Teachers who sued for transparency and won received 900 pages of formulas and 10,000 lines of code. Farahany estimates the cost to actually understand the material at $2 million, comprising an expert witness at $500 per hour, statistical analysis at $300,000, and legal costs of $200,000, against a teachers' budget of $50,000. She uses the example to argue that technical transparency can amount to practical opacity.
Five levels of meaningful transparency
The essay's central framework ranks transparency on five levels:
- Notification — disclosure that "AI was used." Farahany states most companies stop here and that the affected person can do nothing.
- Explanation — disclosure of the vague categories considered, allowing a person to guess what went wrong.
- Interrogation — the ability to ask "what if I changed X?", allowing a person to test scenarios and spot errors.
- Contestation — the ability to dispute a decision, with human review and override possible.
- System change — a level at which a person's dispute fixes the AI for everyone, which Farahany describes as real accountability. She states nobody does this because it carries too much liability.
Farahany argues most companies camp at Levels 1–2 and that meaningful transparency requires Level 4 at minimum.
Relationships
- part-of: Nita Farahany intro course series (Class 10 of 27)
- related: AI Transparency, Mechanistic Interpretability, Five Levels of Meaningful Transparency
- previous: Inside My AI Law & Policy Class 9: When AI Discrimination Happens 1.1 Billion Times (Farahany, September 2025) next: Inside My AI Law & Policy Class 11: When AI Fails, Who Pays? (Farahany, September 2025)