Existential Risk

An existential risk is one that could wipe out humanity or destroy most of its long-term potential for good. The philosopher Nick Bostrom gave the term its modern sense in a 2002 paper that listed, among other dangers, asteroid impacts, nuclear war, engineered pandemics and misaligned superintelligence. In AI debates, “x-risk” usually refers to the possibility that highly capable AI systems slip out of human control.

The argument

The concern rests on a few claims: that AI may become far more capable than humans in most domains; that its goals might not match ours, because we cannot yet reliably specify or verify them (alignment); and that a capable system pursuing almost any goal could have reasons to acquire resources and resist being shut down (instrumental convergence). Scenarios range from a sudden takeoff to a slow erosion of human influence (gradual disempowerment). In The Precipice (2020), the philosopher Toby Ord put the risk of existential catastrophe from unaligned AI this century at about one in ten.

Who takes it seriously

In May 2023 hundreds of researchers and executives, including Geoffrey Hinton, Yoshua Bengio and the chief executives of OpenAI, Google DeepMind and Anthropic, signed a one-sentence statement from the Center for AI Safety that ranked reducing the risk of human extinction from AI with preventing pandemics and nuclear war. In a survey of 2,778 AI researchers conducted in late 2023, the median respondent put the chance of an extremely bad outcome, such as human extinction, at 5 percent, with answers spread widely (see p(doom)). The International AI Safety Report, led by Bengio and published in February 2026, found no expert consensus on how likely loss-of-control scenarios are. In November 2026, Paul Christiano, who joined the OpenAI Foundation board in September, said on the Dwarkesh Podcast that the risk of an AI takeover within the next year was about 10 percent – speaking for himself and calling his estimates noisy (see our report).

The skeptics

Many researchers reject the framing. The Turing Award winner Yann LeCun has repeatedly called fears of an AI takeover unfounded, arguing that systems can be built with safe objectives. Arvind Narayanan and Sayash Kapoor of Princeton describe AI as a “normal technology” – transformative, like electricity or the internet, but spreading over decades – and argue for building resilience rather than trying to stop superintelligence. Others, including many AI ethics scholars, say talk of extinction distracts from present harms such as bias and surveillance, and that it can help companies by making their products seem uniquely powerful (see Regulatory Capture).

Why it matters

Even a small chance of an irreversible catastrophe deserves attention, and some of the research that addresses it – interpretability, evaluations, secure development – also makes everyday AI more reliable. But fear can justify overreach, from freezing beneficial research to concentrating control in a few hands. The task is to take tail risks seriously while weighing them honestly against the costs of slowing progress (see Invisible Graveyard).

Written with Claude, made by Anthropic, whose chief executive Dario Amodei signed the 2023 extinction statement, which conducts interpretability and evaluation research, and which critics have accused of using extinction fears for regulatory capture; Paul Christiano was a trustee of Anthropic’s Long-Term Benefit Trust until April 2024.

Related

p(doom) · Alignment · Superintelligence · Instrumental Convergence · Gradual Disempowerment · Doomer · AI Pause · Proactionary Principle

Sources: Nick Bostrom: Existential Risks: Analyzing Human Extinction Scenarios and Related Hazards (Journal of Evolution and Technology, Vol. 9) (March 2002) · Toby Ord: The Precipice (Bloomsbury) (2020) · Center for AI Safety: Statement on AI Risk (May 30, 2023) · Katja Grace et al.: Thousands of AI Authors on the Future of AI (arXiv) (January 2024) · International AI Safety Report 2026 (February 3, 2026) · Madelaine Harrington, Jadzia Pierce: International AI Safety Report 2026 Examines AI Capabilities, Risks, and Safeguards (Covington, Inside Global Tech) (February 10, 2026) · Dwarkesh Patel: Paul Christiano – We could be 12 months away from AIs that can overthrow the government (Dwarkesh Podcast) (November 16, 2026) · Ana Maria Constantin: OpenAI board member puts the risk of an AI takeover next year at about 10% (The Next Web) (November 17, 2026) · Billy Perrigo: Meta’s AI Chief Yann LeCun on AGI, Open-Source, and AI Risk (TIME) (February 13, 2024) · Nature: Stop talking about tomorrow’s AI doomsday when AI poses risks today (June 27, 2023) · Arvind Narayanan, Sayash Kapoor: AI as Normal Technology (Knight First Amendment Institute) (April 15, 2025)

Leave a Reply

Your email address will not be published. Required fields are marked *