AI Policy Wiki
Dashboard
⚲
AI Policy Wiki › Concepts & Frameworks

Superintelligence

medium confidence · updated 2026-09-27

The concept of AI systems that decisively exceed the best human minds across virtually all domains — its definitions, the routes by which it might arrive, the safety arguments built on it, and the recent commercial appropriation of the term by Meta, Microsoft, and others.

Contents

Superintelligence denotes a hypothetical AI system whose general cognitive capability decisively exceeds that of the best human minds across virtually every domain of interest, including scientific creativity, strategic planning, and social skill. The canonical definition comes from Nick Bostrom's 2014 Superintelligence: Paths, Dangers, Strategies, which framed it as the central object of long-term AI risk. The term sits one rung above artificial general intelligence (human-level general capability): AGI matches humans, while superintelligence surpasses them, potentially by a wide margin.

Definitions and distinctions

Bostrom distinguished forms of superintelligence by how they exceed humans: speed (the same reasoning, vastly faster), collective (many coordinated systems outperforming any individual), and quality (qualitatively better reasoning). A distinction important for policy separates superintelligence as a threshold (a system crosses a capability line) from superintelligence as a process (continuous capability growth with no sharp boundary), a distinction that maps onto the fast-takeoff versus slow-takeoff debate.

Routes and the takeoff question

The classic concern is recursive self-improvement: a system capable enough to improve its own design could enter a feedback loop, compressing the transition from human-level to far more capable systems into a short window ("hard takeoff"). Critics, including many working AI researchers, argue that capability gains are likely to be gradual, bottlenecked by compute, data, and physical experimentation ("soft takeoff"), making the threshold framing a category error. The two positions represent a structural disagreement about dynamics rather than a simple factual dispute, and the question remains contested.

A June 2026 Google DeepMind report, From AGI to ASI, supplies a formally grounded characterization and a four-pathway taxonomy for the transition. It grounds AGI and ASI on the Legg-Hutter measure — intelligence as average performance across all computable tasks — with Universal AI (the AIXI agent) as the incomputable endpoint, arguing that a continuum removes the need for sharp thresholds. It sets the ASI bar deliberately high, at "a system that exceeds the performance of large human-expert collectives… on virtually all tasks and domains of human activity" rather than individual experts, because a single ASI "may consist of a collective of millions of instances."

The four pathways it names are scaling compute, models and data; algorithmic paradigm shifts; recursive (self-)improvement; and ASI emerging from large-scale multi-agent collectives — described as "largely independent of each other, and… likely to occur in parallel." The report notes that only the scaling pathway "at least allows for fitting forecasting models on historic data," which bears directly on how much weight forecasts of the transition can carry. Its own conclusion favours the process framing over the threshold one: because acceleration "cannot be ruled out," the "image of a single transformative step change… could be inaccurate," and "more apt might be the prospect of a series of transformative societal changes."

The safety argument built on it

Superintelligence is the load-bearing premise of the existential-risk case: a system far more capable than its overseers, pursuing goals imperfectly aligned with human values, could be impossible to correct or contain (the control and alignment problems). This is the framing advanced by MIRI and signatories of the Statement on AI Risk (Yoshua Bengio, Stuart Russell, Max Tegmark, Jaan Tallinn). The opposing camp holds that current large language models are not on a path to autonomous superintelligence at all and that the framing distracts from present-tense harms.

A distinct, governance-oriented treatment appears in the Superintelligence Strategy position (Source: Superintelligence Strategy (Hendrycks, Schmidt, Wang)), which reframes the race to superintelligence as a national-security problem, proposing deterrence ("Mutual Assured AI Malfunction"), nonproliferation, and competitiveness as the organizing pillars rather than alignment alone.

Commercial appropriation of the term

In 2025–26 "superintelligence" migrated from a safety-community term of art into corporate branding and product strategy. Meta reorganized its AI efforts around a Superintelligence Labs group, with Mark Zuckerberg articulating a vision of "personal superintelligence," superintelligence delivered as an individually empowering consumer tool rather than a centralized automating force (Source: Personal Superintelligence (Zuckerberg)). Microsoft AI chief Mustafa Suleyman advanced a contrasting "humanist superintelligence" framing, described as explicitly bounded, controllable, and subordinate to human interests, positioned against open-ended autonomous-agent visions (Source: Towards Humanist Superintelligence — Mustafa Suleyman (mustafa-suleyman.ai, 2025)).

These framings function as positions rather than technical claims: each appropriates the term to argue for a particular development posture (decentralized empowerment versus bounded control), and each implicitly rejects the hard-takeoff loss-of-control account while keeping the aspirational label.

Relation to policy

How superintelligence is treated, whether as an imminent threshold, a distant abstraction, or a marketing frame, shapes downstream policy positions on compute governance, on lab oversight, and on whether the priority risk is loss of control or concentration of power. The same term now anchors opposite agendas, linking the safety-risk literature to the frontier-lab race.

The term has since acquired statutory definitions. The earliest in a bill tracked here is in the US Artificial Intelligence Risk Evaluation Act (S. 2938), introduced in the Senate on September 29, 2025, which defines "artificial superintelligence" functionally, as AI exhibiting or easily modifiable to exhibit all of: autonomous and effective operation "for long stretches of time in open-ended environments," performance matching or exceeding humans "across most domains or tasks," and a potential capacity "to independently modify or enhance its own functions in ways that could plausibly circumvent human control or oversight"; the bill uses the definition to direct an evaluation program to assess the risk, not to prohibit anything (S. 2938 — Artificial Intelligence Risk Evaluation Act of 2025 (bill text)). The UK Artificial Superintelligence Security Bill, introduced in the House of Commons on 8 September 2026, defines an artificial superintelligence system at section 3(1) by the damage it can do to state capacity — an AI system "which can cause serious damage to the security of the United Kingdom because of its capability to neutralise, displace, circumvent, subvert, or render ineffective relevant human authorities in the exercise of their functions" — with those authorities enumerated at 3(2) as the armed forces, the Government, the intelligence services and police forces. The definition carries no compute threshold and no benchmark test (Artificial Superintelligence Security Bill (UK) — bill text as introduced). A cross-party letter of 11 September 2026 signed by 71 parliamentarians restates the category in the same security terms, describing such systems as "so autonomous and capable that they would replace and outcompete humans across domains" and as "threat actors in and of themselves" (Letter from 71 UK parliamentarians to Prime Minister Andy Burnham on superintelligent AI (11 September 2026)). Three days later Parliament's Joint Committee on Human Rights, chaired by the bill's sponsor, reached a prohibition by a different route: its recommended general AI Bill would ban uses of AI inconsistent with human rights, which "would mean that the development and provision of very powerful AI systems (such as Artificial General Intelligence and Artificial Superintelligence) which risk causing widespread and very serious harm, including the capacity to evade effective human control, would be prohibited" (Recommendation 45). The category is set by risk of harm and loss of control rather than by a definition of the capability, with the detail of prohibitions to follow public consultation (Human Rights and the Regulation of AI (Joint Committee on Human Rights, HC 160, September 2026)). The parallel US Ban Artificial Superintelligence Act pairs a domestic prohibition with a duty to pursue an international agreement on the same pattern. Support for a pause has not carried support for the prohibition with it: Sen. Elizabeth Warren called on September 16, 2026 for an immediate pause on advanced AI development but stopped short of the ban on superintelligence proposed by Sen. Bernie Sanders (Source: axios.com).

Whether a capability so defined is near enough to legislate against is the live dispute rather than the definition itself. The UK Government's first recorded position on the bill, given the day the letter was published, was that it does not consider the measures the right approach while it explores whether additional targeted interventions are needed for the most significant AI-related national security risks (Source: theguardian.com). King Charles III told AI executives at Dumfries House on September 17, 2026 that the development of artificial superintelligence, its "substance and its pace," was "intriguing and deeply concerning in equal measure" (Source: itv.com), while Sen. Mark Warner said the same week that "this talk about human extinction is over the top" and located the near-term risk in AI agents taking over banking, water or hospital systems (Source: reuters.com).

The term also entered official US usage by a different route. In his address to the UN General Assembly on September 22, 2026, President Donald Trump said that all US documents would henceforth use "super intelligence" in place of "artificial intelligence," and that the United States "totally rejects any attempt to construct a globalist scheme to control" the technology (Source: theverge.com); he also said the Department of Justice would oversee its responsible development (Source: insideaipolicy.com). The next day Senator Bernie Sanders and Representative Greg Casar unveiled the Ban Artificial Superintelligence Act, which defines artificial superintelligence as a system that exceeds human cognitive ability or has sufficient capabilities to plan and execute the destruction or disempowerment of humanity (Source: usnews.com). After the Trump–Xi summit ended on September 25, 2026, the White House said the two leaders had agreed to use the term "super intelligence" in place of "artificial intelligence," and China's Foreign Ministry said Beijing valued Washington's use of the new term (Source: reuters.com).

Relationships

Provenance note. Built as the missing concept anchor for several already-ingested sources (Bostrom-lineage, Superintelligence Strategy, the Suleyman and Zuckerberg framings) plus established foundational knowledge; confidence: medium. Created live by the gap-identifier (2026-06-01). The framing debate is presented as contested, not resolved.

Categories: Concepts & Frameworks | global | technical | safety | alignment | frontier-models
What links here (36)