Author: Nita Farahany Source: https://nitafarahany.substack.com/p/when-ai-discrimination-happens-11 Published: September 23, 2025
The ninth installment of Nita Farahany's AI law and policy course covers algorithmic discrimination. It is anchored on two cases: Mobley v. Workday, in which Workday's screening tools rejected 1.1 billion applications during the relevant period and Judge Rita Lin ruled in July 2024 that Workday "is an agent of its client-employers" for Title VII, ADEA, and ADA purposes; and FTC v. Rite Aid, in which facial-recognition systems produced thousands of false positives concentrated in non-white-majority areas. Around these cases the class develops how bias enters automated systems, why standard de-biasing fails, the limits of existing law, and recent state and federal shifts in liability and enforcement.
How bias enters automated systems
Farahany uses Amazon's 2014–2015 resume tool as the central example of bias originating in data rather than code. The tool was trained on ten years of resumes from successful employees, who were predominantly male, and learned to penalize phrases such as "women's chess club captain," graduates of all-women colleges, and a preference for masculine-coded language. No engineer wrote discriminatory code; the pattern was present in the training data.
The class cites a 2023 Joint Statement from the CFPB, the DOJ Civil Rights Division, the EEOC, and the FTC, which identified three sources of automated discrimination: data and datasets, model opacity, and design and use.
The proxy trap describes why removing a protected characteristic such as gender does not eliminate discrimination. Zip codes predict race through housing segregation, first names predict gender and age, hobbies predict gender, and college names predict race and class. Other available variables become proxies for protected characteristics.
The mathematical impossibility of perfect fairness
The class presents the position that a system cannot simultaneously satisfy all three of the following: equal approval rates across groups, equal accuracy across groups, and equal meaning of scores across groups. Farahany describes this trade-off as mathematically proven. She connects it to NYC Local Law 144, which deliberately does not define an acceptable "impact ratio" on the rationale that there is no single correct answer.
Existing legal framework and its limits
Disparate impact doctrine supplies a three-step framework: the plaintiff shows a statistical disparity; the defendant proves a business justification; the plaintiff shows an alternative method that achieves the goal without the discriminatory effect. Farahany argues that without disparate impact doctrine, AI discrimination would be effectively legal, because intent is unprovable for black-box systems. Trump's Executive Order 14281 (April 23, 2025) directed federal agencies to stop enforcing the disparate impact standard.
Coverage of existing anti-discrimination statutes is uneven. Title VII covers employment, the FHA covers housing, and ECOA covers credit. Not covered are healthcare AI denying treatment, criminal justice AI setting bail, education AI screening admissions, and public-accommodations AI in stores. Bains (Brookings) proposes a single comprehensive federal AI anti-discrimination statute to close these gaps.
NYC Local Law 144, the first-in-nation AI-hiring bias-audit law, took effect in January 2023 and serves as the class's example of weak enforcement. Six months in, only 18 of 391 large NYC employers had posted bias audits, fewer than 5% compliance, and zero complaints had been filed. PepsiCo posted an audit and then removed it, claiming the tool did not qualify as an automated employment decision tool (AEDT).
Shifting liability: California FEHA and Rite Aid
California FEHA regulations effective October 1, 2025 make a three-part change: employers must retain all data for four years; discrimination law applies to AI deployments, so "the AI did it" is not a defense; and vendors can be held liable as agents of employers. Farahany notes that vendors such as Workday and iCIMS therefore acquire direct exposure.
FTC v. Rite Aid is presented as consumer-protection statutes reaching civil-rights harms, using the FTC's Section 5 unfair-practices authority where civil rights statutes do not reach. In the "Bronx Case," a single enrollment image generated 1,000 false-positive alerts in two months, with 99% occurring in Los Angeles, on the opposite coast. While 80% of Rite Aid stores were in majority-white areas, 60% of the stores using facial recognition were in plurality non-white areas.
Provenance
This page summarizes a Substack essay by Nita Farahany published September 23, 2025, the ninth class in her 27-part AI law and policy course (Source: https://nitafarahany.substack.com/p/when-ai-discrimination-happens-11). It is one author's course material; the legal characterizations and the "mathematically proven" framing reflect Farahany's presentation.
Relationships
- part-of: Nita Farahany intro course series (Class 9 of 27)
- related: Mobley v. Workday, Inc., AI Bias and Discrimination, Mathematical Impossibility of Perfect Fairness, NYC Local Law 144 (Automated Employment Decision Tools), California Feha Ai Regulations (planned)
- previous: Inside My AI Law & Policy Class 8: Your Electricity Bill / Compute Governance (Farahany, September 2025) next: Inside My AI Law & Policy Class 10: The Glass Box Paradox (Farahany, September 2025)