"Some Background on Our Views Regarding Advanced Artificial Intelligence" is a blog post by Holden Karnofsky, published by the Open Philanthropy Project (Open Philanthropy) on May 6, 2016. It introduced the term "transformative artificial intelligence" as the organization's "term for a type of potential advanced artificial intelligence we find particularly relevant for our purposes," gave two definitions of it, and set out Karnofsky's view that there is a "nontrivial likelihood" of transformative AI within 20 years. The post presented itself as background to Open Philanthropy's plan to make potential risks from advanced AI "a major priority in 2016," and stated that it did not discuss the risks themselves or measures to reduce them. It is the origin text for the concept of transformative AI. The post is now hosted on the site of Coefficient Giving, the name Open Philanthropy adopted in November 2025.
Defining transformative AI
Karnofsky starts from Open Philanthropy's practice of asking, for any cause, "What's the highest imaginable impact here?", tied to its approach of hits-based giving. Asking which developments in history were most significant, he names the agricultural (neolithic) and industrial revolutions, which he believes "brought about fundamental, permanent changes in the nature of civilization," and argues that with "two such epochal events in the last ~10,000 years (one within the past ~300)" it would be "mistaken to dismiss such a dramatic transition as unprecedented or impossible." He sketches two routes by which AI could produce such a transition: systems broad enough to "outperform humans in a wide array of jobs," or systems making "major contributions to science and/or engineering" that accelerate some other transformative technology.
The post then gives two definitions:
- Definition #1, conceptual: "transformative AI is AI that precipitates a transition comparable to (or more significant than) the agricultural or industrial revolution."
- Definition #2, more detailed, offered because the first "leaves a great deal of room for judgment": anything fitting one or more of three descriptions. The first is AI systems "capable of fulfilling all the necessary functions of human scientists, unaided by humans, in developing another technology (or set of technologies) that ultimately becomes widely credited with being the most significant driver" of such a transition. The second is AI systems "capable of performing tasks that currently (in 2016) account for the majority of full-time jobs worldwide, and/or over 50% of total world wages, unaided and for costs in the same range as what it would cost to employ humans." The third is "surveillance, autonomous weapons, or other AI-centric technology" that itself becomes the most significant driver of such a transition.
Karnofsky writes that neither definition strictly implies the other, but that anything meeting the second "would quite likely meet" the first, and that the rest of the post proceeds from the second. Both leave open "the possibility of AI systems that count as 'transformative' despite not having human-like consciousness or emotions," and despite "lacking many abilities humans have."
Relation to other terms
The post compares the term with Nick Bostrom's definition of superintelligence in Superintelligence ("any intellect that greatly exceeds the cognitive performance of humans in virtually all domains of interest"), with Wikipedia's then definition of artificial general intelligence, and with "high-level machine intelligence" (a system "that can carry out most human professions at least as well as a typical human"). Transformative AI is intended to be "for the most part, a less restrictive term than any of these": anything meeting those descriptions would likely meet the second definition, while transformative AI could arise from "a relatively small number of domains in which AI systems achieve better-than-human performance." Karnofsky's examples are AI that analyzes literature, generates hypotheses and designs experiments, speeding science as in the Scientific Revolution; AI that accelerates particular areas of science; and "a dramatic leap in surveillance capabilities." He gives two reasons for the broader term: the benefits and risks of such paths may differ from those of superintelligence, and his timeline view depends on there being "a large number of possible paths to transformative AI."
Timelines to transformative AI
Karnofsky states that "the only defensible position on this question is one of very high uncertainty," but that he is comfortable claiming "a nontrivial likelihood of transformative AI within the next 20 years." He specifies this as a probability of at least 10%, a view he considers "moderately robust and stable," and a high confidence that "a maximally thorough investigation would put the probability at at least 1%." He writes that the cause would probably remain an Open Philanthropy focus area without this view, but that the view is important to prioritizing it as highly as planned.
Expert surveys and trend extrapolation
The post draws on Luke Muehlhauser's review of AI timeline evidence. It treats the "TOP100" survey in Müller and Bostrom (2014) as the most useful available survey: taking the mean or median of responses implies a 10% probability of high-level machine intelligence within 20 years of 2016 (median 2024, mean 2034), and a 90% chance "well within the next 200 years." Karnofsky has "major reservations" about all such surveys, chiefly that respondents give "essentially off-the-cuff responses" without breaking the question down, and lists possible biases in both directions: enthusiasm biasing toward short timelines, failure to account for the rapid growth of the field, and unfamiliarity with other fields' problems. As evidence of growth he cites rising NIPS attendance and the founding since 2010 of DeepMind (acquired by Google for $500 million), Google Brain, Facebook AI Research, Baidu's Silicon Valley AI Lab, Vicarious and OpenAI (whose funders had committed $1 billion). On trend extrapolation, he reads Muehlhauser's review of when affordable computers might match the processing power of a human brain as implying the capability "may already have been reached" or has a reasonable chance of being reached within 20 years, while having "very limited relevance" to forecasting AI capabilities. He puts "much more weight" on the analysis in the next section.
Technical advisors and core open problems
Karnofsky's main input is discussion with a small set of technical advisors he describes as close friends met through GiveWell and effective altruism: Dario Amodei and Chris Olah, then at Google Brain, and Jacob Steinhardt, then a Stanford PhD student. He acknowledges they are "relatively junior" researchers and "do not constitute a representative sample." He distinguishes two intuitions: that AI progress will require "a very large number of highly specialized insights," or that "a few broadly applicable AI approaches" could produce rapid progress on many tasks, and states that neuroscience and AI research are "highly compatible with both possibilities." Recent progress in deep learning, summarized by Daniel Dewey, is offered as "some suggestive evidence" for broad breakthroughs: ImageNet top-5 error falling from 28.2% in 2010 to 3.6% in 2015 against a trained human's 5.1%; speech-recognition error on the Switchboard data set falling from about 24% to about 16% in 2011 and about 8% by 2015, and the distributed deep-learning system Deep Speech 2 (Amodei et al. 2015) outperforming trained humans on three of four transcription tasks in late 2015; AlphaGo's defeat of Lee Sedol in March 2016; and deep Q-learning on Atari games. Karnofsky considers it "a live possibility" that deep learning plus "a small number (1-3) of future breakthroughs comparable in scope and generality" could produce transformative AI.
The advisors and three other early-career researchers were drafting a document on "core open problems" in AI. Karnofsky's reading of their work is that headway on "a relatively small number of core problems" could plausibly lead to human-level or better performance in many domains; that none of the problems clearly requires many major breakthroughs, decade-scale trial and error, or further brain research; and that "an aggregated picture of the subjective views" of the participants "would point to a 10% or greater probability of transformative AI in the next 20 years."
False alarms and burden of proof
Drawing on Muehlhauser's review of past AI forecasts, Karnofsky accepts that researchers overestimated progress in at least one past period but finds little evidence of similar over-forecasting since the 1970s. He accepts that transformative AI forecasts face "a reasonably high burden of argumentation" but argues against discounting researchers' views "to an extreme degree." He distinguishes his reasoning from "Pascal's Mugging" arguments, which he says he has argued against: his investigation could have produced a probability below 10% and did not, and he would put 5% or less on comparably ambitious claims, such as stem-cell meat competing with conventional meat or a broad-spectrum cancer drug comparable to Gleevec reaching market within 20 years. He writes that his view "is based on a large number of undocumented conversations" and that he does not expect outside readers to share his assessment of the core-open-problems work without their own access to technical experts.
Comparison with other possible developments
The final section lists other candidates for transformative change: progress in biology allowing modification of the body and brain, cheap clean scalable energy, radically improved manufacturing including atomically precise manufacturing, social-science progress, global catastrophic risks including pandemics and worst-case climate change, and shifts in geopolitical relations or cultural values. Karnofsky states that he does not believe most of these are much more likely than transformative AI within 20 years, pandemics being the major exception, and that AI is distinguished by "a dynamic field with striking recent progress," a "live hypothesis" for how a small number of further breakthroughs could lead to transformative technology, and a lack of "concrete, clearly relevant 'obstacles'." He adds that strong AI progress could accelerate progress on almost any of the other developments, and defers the neglectedness and tractability case to a future post.
Key claims
- Transformative AI is defined by impact, as AI precipitating a transition comparable to the agricultural or industrial revolution, not by a capability threshold. (Definitional; attributed to the post.)
- There is at least a 10% probability of transformative AI within 20 years of 2016, held with moderate robustness, and at least 1% with high robustness. (Karnofsky's forecast; his stated basis is largely undocumented conversations with a small, self-described unrepresentative set of advisors. The 20-year window ends in 2036.)
- Expert surveys and trend extrapolation "give relatively little to go on"; the main input is analysis of core open problems with technical advisors. (Position.)
Reception and later use
The two definitions are the basis for later uses of the term. Winter and Bullock adopt the conceptual definition's wording; Toby Ord adapts it into a definition combining an outcome criterion with a risk criterion (Broad Timelines); and Anthropic's "Core Views on AI Safety" and the NBER Economics of Transformative AI volume use the term (Transformative AI (TAI)). An NBER conference paper, "Artificial Intelligence in Research and Development," cites the post as "Karnofsky (2016)" with the same Definition 1 wording (Source: conference.nber.org). The three technical advisors the post names later co-founded or led AI organizations: Amodei and Olah co-founded Anthropic in 2021 (Chris Olah), and Steinhardt founded the nonprofit lab Transluce (Jacob Steinhardt).
Provenance
The raw capture was taken on 27 September 2026 from Coefficient Giving's site, where the page metadata gives Holden Karnofsky as author and 2016-05-06 as publication date, with a modification date of 8 October 2025 reflecting the site migration. Authenticity was checked against an independent citation of the same definition wording (verification record in the gap-scan queue).
Relationships
- supports: Transformative AI (TAI) — origin text of the term and its two definitions
- related: AGI Timelines — an early probabilistic timeline estimate (≥10% within 20 years of 2016)
- related: Artificial General Intelligence (AGI), Superintelligence — the capability-defined terms the post contrasts with transformative AI
- related: Holden Karnofsky — author; Open Philanthropy — publisher
- related: Dario Amodei, Chris Olah, Jacob Steinhardt — technical advisors named in the post
- related: Nick Bostrom — Superintelligence definition compared; co-author of the TOP100 survey the post cites
- related: Andrew Ng — quoted as a skeptic of near-term concern about advanced AI