Catastrophic risk, systemic risk, and existential risk are three distinct AI-risk magnitude terms that are frequently used interchangeably but denote different combinations of severity, reversibility, and scope. They are not synonyms, and the choice among them carries analytical and regulatory consequences. The three categories can be invoked about the same event depending on which framing is primary: a successful rogue-AI takeover is catastrophic in the short run (mass casualties), systemic in its diffusion across institutions, and existential in its endpoint (permanent loss of human control). Disambiguation concerns which framing is primary for a given analytical claim, not mutually exclusive phenomena.
The three primary categories
| Term | Severity | Reversibility | Scope | Example | |
|---|---|---|---|---|---|
| Catastrophic | Very severe (mass casualties, major economic or societal damage) | Recoverable in principle | May be local or global | Biosecurity event with CBRN uplift; large-scale cyberattack on critical infrastructure | |
| Systemic | Severe at the level of a society, sector, or economy | Possibly recoverable but through structural rather than individual fixes | Population- or economy-scale, cascading across systems | [[taxonomy-systemic-risks-gpai | Uuk et al. 2024]] 13-category taxonomy; EU AI Act systemic-risk threshold |
| Existential | Terminal for humanity or civilisation | Irreversible | Species-level, permanent | Loss-of-control to misaligned superintelligence; [[cais-statement-on-ai-risk | CAIS extinction statement]] |
Catastrophic risk
A catastrophic risk is a harm of very large magnitude — typically mass casualties, critical infrastructure failure, or economic losses in the hundreds of billions or more — that is in principle recoverable: humanity continues, institutions rebuild, and the loss is bounded in time.
The term predominates in frontier-AI safety frameworks. The OpenAI Preparedness Framework (OpenAI Preparedness Framework V.2) operationalises "catastrophic" through CBRN, cybersecurity, and autonomy categories at defined capability thresholds. Anthropic's Responsible Scaling Policy uses similar tiering via AI Safety Levels (ASL), with catastrophic-risk framing at ASL-3 and above. Surveys of frontier-AI safety frameworks (AI Safety Cases and Frameworks) generally use "catastrophic" as the main threshold noun, with "systemic" and "existential" appearing less often. The Adolescence of Technology (Amodei) and the autonomy-risk discourse both centre catastrophic rather than strictly existential framings, arguing about recoverable harms severe enough to warrant extraordinary precaution.
Catastrophic functions as the working term in policy because it does not commit the speaker to species-level claims and is compatible with bounded regulatory frameworks built on thresholds, evaluations, and tiered response.
Systemic risk
A systemic risk is one that propagates through or degrades the structural capacities of a society, sector, or economy, where the harm is not the individual incident but the cascading effect across systems. Its distinctive feature is being cascading and distributed: the Economy category is not "AI replaces one job" but "AI reshapes labour markets in ways that destabilise earnings distributions"; the Information category is not "a deepfake misleads voters" but "the epistemic environment degrades to the point trust collapses." Systemic risk can be catastrophic (if cascades are severe enough) and can approach existential (if cascades are irreversible at civilisation scale), but the emphasis is on the mechanism of structural propagation rather than the endpoint magnitude.
The term has two distinct lineages. In the EU regulatory lineage, the EU AI Act adopts "systemic risk" as the threshold category for GPAI: the 10^25 FLOP training-compute threshold serves as the legal proxy for GPAI "with systemic risk," triggering Article 55 obligations (model evaluation, adversarial testing, serious-incident reporting, and cybersecurity protection). This is the only major regulatory regime in which "systemic" is the threshold term. In the academic lineage, Uuk et al. (2024) provide a 13-category taxonomy — Control, Democracy, Discrimination, Economy, Environment, Fundamental rights, Governance, Harms to non-humans, Information, Irreversible change, Power, Security, and Warfare — designed to inform EU systemic-risk compliance guidance. The GPAI Code of Practice operationalises EU systemic-risk compliance, and systemic-risk channels appear in AI Labor Disruption (labour-market cascade) and AI and Authoritarianism (power-concentration cascade).
Existential risk (x-risk)
An existential risk is a risk of human extinction or of permanent and drastic curtailment of humanity's long-term potential. The term is due to Nick Bostrom (2002); subsequent refinements in Toby Ord's The Precipice (2020) distinguish three forms:
- Extinction — literal end of the human species.
- Unrecoverable collapse — civilisational loss so severe that recovery is impossible.
- Unrecoverable dystopia (value lock-in) — a permanent undesirable state, not necessarily accompanied by population loss.
In AI policy discourse, "x-risk" typically refers to a specific failure mode: a sufficiently capable, misaligned AI system that humans cannot correct or shut down. This is the failure mode assumed by AI 2027, the autonomy-risk discourse, the FLI pause letter, and the CAIS statement ("mitigating the risk of extinction from AI should be a global priority"). Related framings include Compressed 21st Century (Amodei on the outer edge of the existential framing).
Where the wiki uses each term
The catastrophic framing (the working policy term) appears in OpenAI Preparedness Framework V.2 (CBRN, cyber, and autonomy as catastrophic-risk categories), Anthropic's Responsible Scaling Policy (Version 3.1) (ASL thresholds tied to catastrophic risk, with catastrophic-risk framing at ASL-3 and above), AI Safety Cases and Frameworks (general survey in which catastrophic is the dominant term), AI Biosecurity (uplift-as-catastrophic framing), and The Adolescence of Technology (Amodei's core argument about catastrophic-not-necessarily-existential harms).
The systemic framing (EU regulatory plus academic) appears in EU AI Act (Regulation 2024/1689) (10^25 FLOP threshold, Article 55 obligations), A Taxonomy of Systemic Risks from General-Purpose AI (13-category taxonomy), EU General-Purpose AI Code of Practice (Final Version, 2025) (operationalises EU systemic-risk compliance), AI Labor Disruption (labour-market cascade as a systemic-risk channel), and AI and Authoritarianism (power-concentration cascade).
The existential framing (the x-risk school) appears in Statement on AI Risk (CAIS) (the canonical statement), FLI — Pause Giant AI Experiments: An Open Letter (precursor framing), AI 2027 (narrative scenario of loss-of-control), Compressed 21st Century (Amodei on the outer edge of the existential framing), and AI Autonomy Risk (the failure mode through which x-risk materialises).
Related x-risk terms
- Global catastrophic risk (GCR) — a broader umbrella from the Global Catastrophic Risks volume (Bostrom and Ćirković, 2008) that includes pandemics, nuclear war, asteroid impact, and similar events. X-risk is the subset that is terminal.
- Lock-in — irreversible concentration of power (economic, political, or value-system) enabled by AI that forecloses humanity's future options without necessarily causing mass casualties. Emphasised by the authoritarianism literature and by AI 2027.
- S-risk (suffering risk) — long-term future states involving large-scale suffering. Marginal in policy discourse but active in longtermist research.
Debates and positions
The AI safety vs. AI ethics divide
The three terms map partially onto the split between the AI safety and AI ethics discourses. The safety / x-risk school foregrounds existential and catastrophic framings and is associated with FLI, CAIS, MIRI, and x-risk longtermism. The ethics / systemic school foregrounds systemic and structural framings and is associated with AI Now, the Algorithmic Justice League, and critical-AI scholarship. The split is less a disagreement about facts than a disagreement about which magnitude is the right policy object: a regulator writing the EU AI Act systemic-risk regime and a researcher writing a pause letter are asking different questions, even when both are nominally about "AI risk."
Substitution errors
Three recurring substitution errors arise when the terms are conflated. Calling every catastrophic harm "existential" inflates the threshold and makes the term do no work, since most CBRN uplift scenarios are catastrophic but not extinction-level. Calling every systemic harm "catastrophic" reads cascading structural risks as acute shocks and misses the slow-diffusion failure modes (lock-in, epistemic collapse, deskilling). Calling existential harm "systemic" understates its irreversibility and the distinctive policy question it raises (human continuation versus bounded loss).
The EU AI Act's choice of "systemic"
The EU chose "systemic risk" as its legal threshold term rather than "catastrophic" (described as too acute) or "existential" (described as too contested). "Systemic" is broad enough to cover the catastrophic and x-risk cases while being grounded enough in observable societal impact to be justiciable. This choice has since influenced G7 and OECD vocabulary.
Relationships
- depends-on: A Taxonomy of Systemic Risks from General-Purpose AI — academic taxonomy of systemic risks
- depends-on: Statement on AI Risk (CAIS) — existential-risk framing
- related: The Adolescence of Technology — catastrophic-not-existential framing
- related: AI 2027 — existential-endpoint scenario
- related: AI Autonomy Risk — mechanism through which x-risk materialises
- related: EU AI Act (Regulation 2024/1689) — systemic-risk regulatory threshold
- related: AI Safety Cases and Frameworks, OpenAI Preparedness Framework V.2, Anthropic's Responsible Scaling Policy (Version 3.1) — catastrophic-risk operationalisations
- related: AI Biosecurity — catastrophic-risk vertical
- related: Compressed 21st Century — edge case straddling catastrophic and existential