Author: Nita Farahany Source: https://nitafarahany.substack.com/p/how-ai-actually-works Published: August 26, 2025
This is the second installment (Class 2 of 27) in Nita Farahany's "Inside My AI Law & Policy Class" Substack series, published August 26, 2025. The class covers the technical foundations of how large language models function and builds the core vocabulary the rest of the course relies on. It uses Steven Schwartz's sanctions case as the recurring example for why AI systems produce confident, precise-sounding fabrications.
Summary
The class is anchored on the Mata v. Avianca sanctions case (S.D.N.Y., May 2023), in which attorney Steven Schwartz was penalized $5,000 for citing six fake AI-generated cases produced by ChatGPT. Farahany uses the episode to argue that the fabricated citations were not a malfunction but the system operating as designed, and walks through next-token prediction, parameters and weights, embeddings, the attention mechanism, transformer architecture, the AI supply chain, and reinforcement learning from human feedback (RLHF) to explain why.
Key claims
The class presents next-token prediction as the core mechanism. Given a prompt such as "When my dog was ___," a model selects the statistically most likely next word from a vocabulary of 50,000+ candidates; on this account Schwartz's fake citations followed directly from the system working exactly as designed.
Parameters and weights are presented as "dials" set during training and frozen during inference, with GPT-4's roughly 1.8 trillion parameters as the example. Farahany uses a judge-sentencing analogy in which prior convictions carry HIGH weight, defendant age MEDIUM, and day of week ZERO.
Embeddings are described as coordinates in a meaning-space, with 768 to 1,536 numbers per word, placing "judge" and "court" mathematically near each other and "banana" far away. The class flags a legal danger: motion-to-dismiss, motion-for-summary-judgment, motion-in-limine, and motion-to-compel may be near-neighbors in embedding space despite governing radically different legal standards.
The attention mechanism is illustrated with the sentence "The witness, who was clearly nervous and had contradicted herself twice, stated that she saw the defendant." In predicting the next word, "saw" receives HIGH weight (0.8), "witness" HIGH (0.7), "stated" MEDIUM, "nervous" LOW, and "and" NEAR ZERO.
Reasoning models, the class argues, still hallucinate: they generate "reasoning tokens" that look like thinking but remain token prediction, producing the same hallucinations with more elaborate explanations.
The class introduces a 7-layer AI supply chain — Data Providers, Compute Providers, Foundation Model Developer, Model Host, Application Developer, Application Deployer, End User — and argues that distributed control across these layers breaks traditional negligence analysis, pointing toward a contributory negligence framework.
RLHF is presented as the training stage where human preferences become the model's "truth." Farahany illustrates this with a class poll on legal-jargon versus accessible-language preferences, noting that whoever does the ranking sets the bar. The class describes KL divergence as a safety belt and recounts an incident in which researchers reversed the math by a single minus sign, turning GPT-2 into what was described as "maximally lewd."
Relationships
- part-of: Nita Farahany intro course series (Class 2 of 27)
- related: Ai Supply Chain Liability, RLHF (Reinforcement Learning from Human Feedback), Sycophancy and Hallucination
- previous: Inside My AI Law & Policy Class 1: What is AI? (Farahany, August 2025) next: Inside My AI Law & Policy Class 3: Open vs. Closed AI Models (Farahany, September 2025)